# MVGAL Section 3 — Research and Analysis Phase

**Date:** 2026-05-30  
**Scope:** Sections 3.1 (Existing Technologies), 3.2 (Codebase Examination), 3.3 (Technology Stack Selection)  
**Status:** Complete — ready to proceed to Section 5 (Implementation Plan)

Upstream clones used for this pass (shallow, `/tmp/mvgal-upstream/`):

| Repository | Commit | Date |
|------------|--------|------|
| nvidia/open-gpu-kernel-modules | `57130a27` | 2026-05-26 |
| GPUOpen-Drivers/AMDVLK (manifest) | `0f7b1d76` | 2025-09-15 |
| GPUOpen-Drivers/xgl (ICD) | `e9782eb3` | 2025-04-29 |
| dixyes/mtgpu-drv | `96475126` | 2026-01-16 |
| intel/media-driver | `7fb7e47f` | 2026-05-29 |

---

## 3.1 Existing Technologies Study

### 3.1.1 Vendor Multi-GPU Frameworks (SLI, CrossFire, NVLink, NCCL, MPI)

| Technology | Scope | Workload model | Relevance to MVGAL |
|------------|-------|----------------|-------------------|
| **NVIDIA SLI / NVLink** | Same-vendor, driver-coordinated | AFR, SFR, SLIAA; NVLink enables P2P + unified memory on supported pairs | Patterns (AFR/SFR tiling) apply; NVLink P2P does **not** extend cross-vendor |
| **AMD CrossFire / XGMI** | Same-vendor | AFR, SFR; XGMI = high-bandwidth fabric like NVLink | AMD XGL exposes `EnumeratePhysicalDeviceGroups` for PAL-managed groups only |
| **CUDA multi-GPU / NCCL** | NVIDIA-only | Explicit contexts per device; NCCL for collective ops | MVGAL CUDA shim must fan out explicitly; no NCCL equivalent across vendors |
| **MPI + CUDA / OpenCL** | HPC clusters | Domain decomposition + halo exchange | Model for compute split; latency budget much higher than frame rendering |
| **PyTorch / TF Distributed** | Framework-level | Data/model parallelism, all-reduce over network | Useful for AI offload strategy, not transparent rendering |
| **FluidX3D / HPC sims** | Multi-GPU CFD | Regular grid decomposition, ghost cells | Validates compute-decomposition + sync patterns for `compute_offload` strategy |

**Key conclusion:** All production multi-GPU stacks require either (a) same vendor + driver support, or (b) explicit application awareness. MVGAL's transparency goal is achievable only at the **API interception layer**, not by emulating SLI/CrossFire inside a kernel meta-driver.

### 3.1.2 GPU Virtualisation (DRM, virtio-gpu, SR-IOV)

| Approach | Mechanism | MVGAL fit |
|----------|-----------|-----------|
| **Linux DRM/KMS** | Render nodes, GEM, dma-buf, sync_file | Primary interop surface; MVGAL should coordinate, not replace vendor DRM |
| **virtio-gpu** | Paravirtual GPU for VMs | Useful reference for command queue + resource virtualization, not direct reuse |
| **SR-IOV / MIG** | Hardware partitioning | Opposite goal (split one GPU) but teaches fence/isolation patterns |
| **PRIME / dma-buf** | Cross-driver buffer sharing | **Critical path** for cross-vendor memory |

MVGAL kernel Phase 1 correctly avoids registering as a competing DRM driver on PCI devices. Future Phase 2 may add a **virtual DRM node** for compositor integration, but command submission stays in user space through vendor ICDs.

### 3.1.3 API Interception Surfaces

| API | Interception point | Multi-device primitive |
|-----|-------------------|------------------------|
| **Vulkan** | Layer (`VK_LAYER_MVGAL`) or ICD (`mvgal_vulkan_icd.so`) | `VK_KHR_device_group`, external memory/semaphore fd |
| **OpenGL** | LD_PRELOAD (glX/EGL swap) or Zink (GL→VK) | Limited; frame pacing telemetry only today |
| **OpenCL** | ICD / LD_PRELOAD | `cl_khr_gl_sharing`, P2P extensions vendor-specific |
| **CUDA** | LD_PRELOAD shim | `cudaDeviceCanAccessPeer`, multi-context manual |
| **D3D (Wine)** | DXVK / VKD3D → Vulkan layer | Inherits Vulkan aggregation path |
| **SYCL / oneAPI** | Level Zero / OpenCL backend | Intel-focused; heterogeneous via explicit queues |

**Vulkan layering spec:** Layers may intercept instance/device commands but cannot replace the ICD. A **virtual ICD** (`src/userspace/vulkan_icd/`) is the correct place to expose one `VkPhysicalDevice`; the layer handles submit-time scheduling (AFR/SFR).

### 3.1.4 Wine/Proton Synchronisation (Esync, Fsync, NTSYNC)

| Mechanism | Implementation | Performance | MVGAL role |
|-----------|----------------|-------------|------------|
| **Esync** | Eventfd-based | Good; fd limits on heavy games | Works today with aggregated GPU |
| **Fsync** | futex-based | Better for many waits | Default on modern Proton |
| **NTSYNC** | Windows NT sync object semantics in kernel | Best thread scaling; Wine 9.0+ staging | MVGAL scaffold: `kernel/mvgal_ntsync.c` — userspace misc device, not yet wired to Wine tree |
| **WoW64** | 32-on-64 struct repacking | Required for 32-bit Windows games | Scaffold: `src/userspace/win32/wow64_thunk.c` — handle table + partial Vulkan thunks |

NTSYNC integration should **not** fork Wine wholesale initially; ship a kernel module + patch set Proton can consume via `WINE_NTSYNC=1` when MVGAL's implementation is production-ready.

---

## 3.2 Codebase Examination

### 3.2.1 Repository Reality vs Documentation

| Document | Claim | Verified reality |
|----------|-------|------------------|
| `README.md` / `STATUS.md` | "Production-ready MVP", 100% tests | Large scaffold with passing unit/integration tests; **cross-vendor transparent aggregation not proven on real multi-GPU hardware** |
| `docs/RESEARCH.md` (prior) | Prototype baseline | **Accurate** — kernel was rebuilt as PCI topology probe |
| `docs/ENGINEERING_DESIGN.md` | Full architecture design | Design-complete; several components are Phase 1 stubs |

**Honest maturity estimate:** ~70% infrastructure, ~30% production aggregation (SPIR-V routing Phase 1, ICD scaffold, scheduler logic present but heterogeneous submit path incomplete).

### 3.2.2 MVGAL Component Inventory

#### Kernel (`kernel/`)

| File | ~LOC | Actual behaviour |
|------|------|------------------|
| `mvgal_main.c` | 881 | PCI display-class scan, `/dev/mvgal0`, 8 implemented ioctls + 4 future ioctls (`-EOPNOTSUPP` unless `MVGAL_FULL_STACK`) |
| `mvgal_core.c` | 719 | Character device, GPU slot management |
| `mvgal_memory.c` | 1078 | DMA-BUF scaffolding; cross-vendor alloc returns `-EOPNOTSUPP` in skeleton builds |
| `mvgal_scheduler.c` | 353 | Priority queue skeleton |
| `mvgal_sync.c` | 342 | Fence/timeline scaffolding |
| `mvgal_ntsync.c` | ~400 | Misc device `mvgal_ntsync` for Wine sync primitives |
| `vendors/*.c` | 443–518 each | Per-vendor ops tables; init via sysfs/module probes; BAR-based VRAM estimates |

**UAPI** (`include/mvgal/mvgal_uapi.h`): Version 0.1.0 feature flags explicitly mark `READ_ONLY_SKELETON`, `FUTURE_DMABUF`, `FUTURE_SUBMISSION`.

**Design win:** No `pci_register_driver` — MVGAL observes GPUs without stealing them from vendor drivers.

#### User-space interception

| Component | Path | Status |
|-----------|------|--------|
| Vulkan layer | `intercept/vulkan/vk_layer.c` (~1419 LOC) | Loader-compliant; registers physical devices; AFR injects `VkDeviceGroupSubmitInfo` when `is_aggregate` — **does not create synthetic physical device** |
| Vulkan ICD | `vulkan_icd/` (~3500 LOC) | Virtual `VkPhysicalDevice`, memory aggregation, device group emulator |
| SPIR-V routing | `mvgal_shader_translate.c` | Phase 1: FNV-1a cache + capture; Phase 2 backends not wired |
| OpenCL / CUDA / D3D / Metal / WebGPU | `intercept/*` | Wrappers compile; distribution strategies delegate to scheduler |
| OpenGL | `opengl/mvgal_gl_preload.c` | Swap-chain telemetry; rendering via Zink |

#### Runtime

| Component | Language | Role |
|-----------|----------|------|
| `runtime/daemon/` | C++20 | Scheduler, memory, power, IPC (21 message types), D-Bus |
| `safe/*` | Rust | fence_manager, memory_safety, capability_model (12 tests) |
| `src/userspace/scheduler/` | C | 7 strategies (AFR, SFR, hybrid, etc.) |

### 3.2.3 Upstream Vendor Repository Analysis

#### NVIDIA open-gpu-kernel-modules (`57130a27`, release 610.43.03)

**Anchors:** `README.md`, `nv-ioctl-numbers.h`, `nvidia-drm-gem-dma-buf.c`

Findings:

1. Kernel modules require matching user-mode driver + GSP firmware (same release train).
2. Private RM ioctls: `NV_ESC_REGISTER_FD`, `NV_ESC_IOCTL_XFER_CMD`, `NV_ESC_EXPORT_TO_DMABUF_FD`.
3. DMA-BUF via `nvidia-drm` PRIME helpers — suitable for **explicit** buffer sharing, not generic meta-driver submit.

**MVGAL integration strategy:** User-space bridge to `/dev/nvidia*` + Vulkan layer/ICD; kernel shim reads sysfs only.

#### AMD XGL / AMDVLK manifest

**Anchors:** `xgl/icd/api/vk_instance.cpp`, `vk_memory.cpp`, `vk_device.cpp`

Findings:

1. AMDVLK repo is a manifest; ICD lives in `xgl`.
2. `Instance::EnumeratePhysicalDeviceGroups` groups PAL devices on same stack — **homogeneous only**.
3. External memory: `VK_STRUCTURE_TYPE_IMPORT_MEMORY_FD_INFO_KHR`, DMA_BUF + OPAQUE_FD in `vk_memory.cpp`.
4. `VK_IMAGE_CREATE_SPLIT_INSTANCE_BIND_REGIONS_BIT` rejected — peer page tables not implemented for split bindings.
5. XGMI unlocks peer transfer features in `GetMultiGpuCompatibility`.

**MVGAL integration strategy:** Sit above RADV/AMDVLK; use external semaphore/memory fd for cross-vendor sync.

#### Intel media-driver (`7fb7e47f`, release 26.2.x)

**Anchors:** `media_libva.cpp` → `DdiMedia_ExportSurfaceHandle`

Findings:

1. VA-API media driver — **not** primary graphics/compute path.
2. Exports surfaces via `mos_bo_export_to_prime` as DRM PRIME 2/3 — valuable for **video** leg of professional workloads.
3. Graphics/compute must target `i915`/`xe` + ANV (Mesa) or Intel compute runtime.

#### Moore Threads mtgpu-drv (`96475126`)

**Anchors:** `mtgpu_drm.h`, `mtgpu_drm_gem.c`, README (unmaintained)

Findings:

1. Public mirror explicitly **unmaintained** since 2026-01.
2. Rich DRM ioctl surface: job submit, timeline, fence/semaphore-to-fd, DMA transfer.
3. PCI vendor ID `0x1ED5` (MVGAL UAPI corrected from erroneous `0x1EAC`).
4. dma-buf via standard DRM GEM — similar integration model to AMD/Intel.

### 3.2.4 Common Abstraction Targets

| Interface | AMD | NVIDIA | Intel | MTT |
|-----------|-----|--------|-------|-----|
| Topology query | PCI + amdgpu sysfs | PCI + procfs/nvml | PCI + i915/xe sysfs | PCI + mtgpu DRM |
| Buffer export | dma-buf (GEM) | NV_ESC_EXPORT_TO_DMABUF_FD / nvidia-drm | dma-buf (GEM) | dma-buf (GEM) |
| Fence export | sync_file / timeline | sync_file (limited) | sync_file | MTGPU_FENCE_TO_FD |
| Command submit | DRM_CS / PAL | RM ioctl / proprietary | execbuf2 / xe_exec | DRM_MTGPU_JOB_SUBMIT |
| Power control | sysfs DPM | NVML / power limit | RPS sysfs | vendor-specific |

**Lowest common denominator for cross-vendor:** dma-buf + sync_file/semaphore fd + host staging fallback.

### 3.2.5 Gap Analysis vs Reference Technologies

| Feature | NVLink/SLI | CrossFire | MVGAL today | MVGAL target |
|---------|------------|-----------|-------------|--------------|
| Transparent single device | Driver-internal | Driver-internal | ICD scaffold | ICD + layer submit rewrite |
| AFR | Hardware + driver | Driver | Layer inject (partial) | Full multi-ICD submit |
| SFR / tiling | Driver | Driver | Scheduler code exists | Needs compositing blit |
| P2P copy | NVLink/PCIe | PCIe/XGMI | P2P detect scaffold | Topology-aware routing |
| Unified VRAM | NVLink unified memory | HSA-style (APU) | Not implemented | Demand migration (hard) |
| Compute split | CUDA/NCCL | ROCm | CUDA shim strategies | SPIR-V grid split |

---

## 3.3 Technology Stack Selection

### 3.3.1 Languages (chosen mix)

| Layer | Language | Justification |
|-------|----------|---------------|
| Kernel module | **C** (GNU11, kernel style) | Only viable choice for out-of-tree Linux module; GCC/Clang per distro |
| Runtime daemon | **C++20** | Complex subsystems (scheduler, IPC, D-Bus); existing codebase |
| Safety-critical | **Rust** (2021 edition) | Fence/memory/capability models; FFI to C |
| User-space core / intercept | **C** | Vulkan layer/ICD ABI, LD_PRELOAD shims, minimal overhead |
| SPIR-V tooling | **C++** (optional) | SPIRV-Tools + SPIRV-Cross when `MVGAL_ENABLE_SPIRV_OPT=ON` |
| CLI tools | **C** | Small footprint, no runtime deps |
| REST API | **Go** | HTTP server for dashboard backend |
| Qt dashboard | **C++ / Qt5/6** | Optional control panel |
| Bindings | Python (ctypes), Go, Java, C#, etc. | Tooling only |

**Not selected for core path:** Zig (optional build), D/Nim/V/Crystal/Haxe (bindings only), CUDA C++ (runs on NVIDIA via shim), SYCL (Intel-specific backend later).

### 3.3.2 Build systems

| Tool | Role |
|------|------|
| **CMake** | Primary — kernel option, runtime, tests, packaging hooks |
| **Meson** | Alternative documented in README |
| **Kbuild** | Kernel module (`kernel/Kbuild`, DKMS) |
| **Cargo** | Rust safety crates |
| **Ninja** | Preferred generator |

**Compiler policy:** GCC for maximum distro compatibility on kernel module; Clang acceptable for userspace sanitizers. Intel **oneAPI** reserved for future Intel GPU compute fast-path, not main build.

### 3.3.3 Graphics/compute API strategy

| API | MVGAL approach |
|-----|----------------|
| Vulkan | **ICD** (virtual device) + **layer** (scheduling, telemetry) |
| OpenGL | LD_PRELOAD + Zink fallback |
| OpenCL | ICD wrapper / LD_PRELOAD |
| CUDA | LD_PRELOAD shim with explicit multi-device fan-out |
| D3D | DXVK/VKD3D → Vulkan path |
| VA-API | Side integration via Intel media-driver dma-buf export |

### 3.3.4 Privilege model

All root operations via **`pkexec`** + PolicyKit action `org.freedesktop.policykit.mvgal` — no direct `sudo` in scripts (verified in `config/`, `scripts/install_dependencies.sh`).

### 3.3.5 Target platform matrix

| Target | Minimum | Verified |
|--------|---------|----------|
| Kernel | 5.15+ | 7.0.9 (Fedora 44) |
| Distros | Ubuntu, Fedora, Arch, Debian, openSUSE | COPR packages for Fedora/RHEL family |
| Architectures | x86_64 primary | aarch64 possible for kernel module |

---

## Cross-Cutting Recommendations (Implementation Priorities)

Ordered for Section 5 execution:

1. **Harden kernel topology UAPI** — already PCI-backed; add userspace probe tool output tests.
2. **Vulkan ICD as default aggregation surface** — layer alone cannot hide multiple physical devices.
3. **Prototype one cross-vendor memory pair** — AMD↔Intel dma-buf round-trip (tests exist as scaffold).
4. **SPIR-V Phase 2** — wire SPIRV-Cross/NVIDIA PTX path for compute offload only (narrow scope).
5. **SFR compositing** — blit tiles via primary GPU before present.
6. **NTSYNC** — align `mvgal_ntsync.c` with upstream Wine ioctl layout before Proton fork.
7. **WoW64** — complete Vulkan struct thunks in `wow64_thunk.c`.
8. **Replace optimistic status docs** with milestone gates tied to hardware validation matrix.

---

## Risks

1. **NVIDIA closed RM path** — open kernel modules insufficient alone; user-mode coupling mandatory.
2. **MTT driver unmaintained** — treat as best-effort; prefer mainline if MTT merges.
3. **Extension intersection** — virtual device must advertise honest capability intersection, not sum-of-all.
4. **Anti-cheat** — Vulkan layers blocked by some titles; ICD path may fare better.
5. **Unified VRAM** — highest risk item; defer until dma-buf + migration proven.

---

## References

- NVIDIA open-gpu-kernel-modules: https://github.com/nvidia/open-gpu-kernel-modules
- AMDVLK / XGL: https://github.com/GPUOpen-Drivers/AMDVLK , https://github.com/GPUOpen-Drivers/xgl
- Intel media-driver: https://github.com/intel/media-driver
- MTT mtgpu-drv: https://github.com/dixyes/mtgpu-drv
- Linux DRM/dma-buf: https://docs.kernel.org/gpu/drm-uapi.html
- Vulkan Layer specification: https://github.com/KhronosGroup/Vulkan-Loader/blob/main/docs/LoaderLayerInterface.md
- Wine NTSYNC: Wine 9.0+ / wine-staging patches
