# MVGAL Complete Architecture Analysis

**Version:** 1.0  
**Date:** 2026-06-02  
**Status:** Research Phase Complete Analysis  

---

## 1. Executive Summary

MVGAL (Multi-Vendor GPU Aggregation Layer for Linux) is a sophisticated multi-layered system designed to aggregate heterogeneous GPUs from different vendors (AMD, NVIDIA, Intel, Moore Threads) into a unified logical device. The current implementation (v0.2.3) provides a solid foundation with:

- **Kernel Module Layer**: Character device `/dev/mvgal0` exposing GPU topology via ioctls
- **Safety Subsystems**: Rust crates for fence management, memory safety, and capability modeling
- **Runtime Daemon**: C++20 daemon (`mvgald`) with scheduler, memory manager, and power manager
- **API Interception**: Vulkan layer, OpenCL ICD, CUDA shim for transparent workload distribution
- **Tooling**: CLI tools, Qt dashboard, REST API for monitoring and control

**Key Finding**: MVGAL correctly positions itself as an **above-driver coordination layer**, not a replacement for vendor DRM drivers. This architectural decision avoids the hard problem of cross-vendor command buffer submission while enabling practical multi-GPU aggregation for compatible workloads.

---

## 2. Existing Architecture

### 2.1 Six-Layer Stack

```
┌─────────────────────────────────────────────────────────────┐
│ Layer 6 — Tooling: mvgal-info, mvgal-status, Qt dashboard,    │
│                  REST API                                   │
├─────────────────────────────────────────────────────────────┤
│ Layer 5 — API Interception: VK_LAYER_MVGAL, OpenCL ICD,    │
│           CUDA shim, OpenGL preload                         │
├─────────────────────────────────────────────────────────────┤
│ Layer 4 — Execution Engine: Frame sessions, migration plans,  │
│           Steam profiles                                    │
├─────────────────────────────────────────────────────────────┤
│ Layer 3 — Runtime Daemon (mvgald): Scheduler, MemoryMgr,     │
│           PowerMgr, MetricsCollector, IPC                   │
├─────────────────────────────────────────────────────────────┤
│ Layer 2 — Rust Safety Subsystems: fence_manager, memory_safety,│
│           capability_model                                  │
├─────────────────────────────────────────────────────────────┤
│ Layer 1 — Kernel Module (mvgal.ko): DRM meta-driver,        │
│           /dev/mvgal0, vendor ops                           │
└─────────────────────────────────────────────────────────────┘
```

### 2.2 Component Summary

| Component | Location | Language | Status |
|-----------|----------|----------|--------|
| Kernel module | `kernel/` | C (GPL-2.0) | ✅ Builds, ioctls implemented |
| Vulkan layer | `src/userspace/intercept/vulkan/` | C | ✅ 18 functions intercepted |
| OpenCL ICD | `src/userspace/intercept/opencl/` | C | ✅ LD_PRELOAD complete |
| CUDA shim | `src/userspace/intercept/cuda/` | C | ✅ 40+ functions intercepted |
| Scheduler | `src/userspace/scheduler/`, `runtime/daemon/scheduler.cpp` | C++, C | ✅ 7 strategies implemented |
| Memory manager | `src/userspace/memory/`, `runtime/daemon/memory_manager.cpp` | C++, C | ✅ DMA-BUF, P2P stubs |
| Rust crates | `safe/` | Rust | ✅ 12/12 tests pass |
| CLI tools | `tools/` | C | ✅ All tools functional |
| Steam integration | `steam/` | C | ✅ Frame pacer, AFR |
| UI dashboard | `ui/` | C++, Go | ✅ Qt5/Qt6 + REST API |

---

## 3. Linux DRM Integration

### 3.1 Direct Rendering Manager (DRM)

MVGAL integrates with Linux DRM through:

- **PCI Enumeration**: Scans display-class PCI devices via `pci_get_device()` in `mvgal_main.c`
- **DRM Node Discovery**: Reads `/sys/class/drm/cardN/device/` for GPU properties
- **Render Node Access**: Uses `/dev/dri/renderD*` for command submission (vendor-native)
- **DMA-BUF Integration**: Uses `DRM_IOCTL_PRIME_HANDLE_TO_FD` and `DRM_IOCTL_PRIME_FD_TO_HANDLE`

### 3.2 GEM (Graphics Execution Manager)

- AMD/Intel: Full GEM support for buffer management
- NVIDIA: GEM support in open kernel modules (525+), NV_dma_buf for proprietary
- MTT: GEM-based (per documentation)

### 3.3 TTM (Translation Table Maps)

- AMD uses TTM for VRAM management; MVGAL abstracts via sysfs readouts

### 3.4 KMS (Kernel Mode Setting)

MVGAL does not perform mode setting; relies on vendor drivers for display management.

---

## 4. Linux GPU Subsystems

### 4.1 DMA-BUF

**Status**: Primary cross-GPU transfer path

- AMD: Full export/import support
- Intel: Full support
- NVIDIA: Open modules (525+) + `nvidia-drm.modeset=1` for proprietary
- MTT: Documented support, needs hardware validation

### 4.2 PRIME

- Built on DMA-BUF
- MVGAL uses for shared buffer allocation

### 4.3 IOMMU

- Not directly used by MVGAL (vendor drivers handle IOMMU mapping)

### 4.4 VFIO

- Not currently integrated (potential future: GPU passthrough for VMs)

### 4.5 HMM (Heterogeneous Memory Management)

- AMD: Uses HMM for unified memory
- Intel: Uses HMM on newer architectures
- MVGAL plans to leverage via capability probes

### 4.6 NUMA

- MVGAL reads NUMA node from `/sys/bus/pci/devices/<slot>/numa_node`
- NUMA-aware allocation for staging buffers

---

## 5. GPU Vendor Analysis

### 5.1 NVIDIA

**Repository**: https://github.com/NVIDIA/open-gpu-kernel-modules

**Key Features**:
- `nvidia.ko` + `nvidia-drm` character device registration
- CUDA, Vulkan, OpenGL support
- NVLink (high-speed GPU-GPU interconnect)
- NCCL for collective operations
- GPUDirect for direct storage/IO access

**Integration Points**:
- LD_PRELOAD CUDA shim intercepts kernel launches
- Vulkan layer routes via VkDeviceGroupSubmitInfo
- `nvidia-smi` for metrics (proprietary driver)

**Limitations**:
- Anti-cheat (EAC, BattlEye) blocks Vulkan layers
- Cross-vendor P2P requires nvidia-drm.modeset=1
- NVLink not currently supported in MVGAL

### 5.2 AMD

**Repository**: https://github.com/GPUOpen-Drivers/AMDVLK (discontinued per README)

**Key Features**:
- `amdgpu.ko` DRM driver (active)
- ROCm/HIP for compute
- Mesa RADV (open-source Vulkan)
- Infinity Fabric interconnect (same-socket multi-GPU)

**Integration Points**:
- Sysfs metrics: `gpu_busy_percent`, `mem_info_vram_*`
- DRM render nodes for command submission
- DMA-BUF P2P within same root complex

**Limitations**:
- AMDVLK discontinued; RADV/Mesa is primary Vulkan path
- No cross-vendor command stream format

### 5.3 Intel

**Repository**: https://github.com/intel/media-driver

**Key Features**:
- `i915` (Gen 9-12) and `xe` (Xe/ARC) DRM drivers
- Level Zero API
- oneAPI DPC++/SYCL support
- OpenCL, Vulkan ANV

**Integration Points**:
- DRM render nodes
- Intel GPU tools (`intel_gpu_top`) for metrics
- DMA-BUF sharing

**Limitations**:
- Media-driver is VA-API focused (not full 3D)
- Cross-vendor limited to DMA-BUF staging

### 5.4 Moore Threads (MTT)

**Repository**: https://github.com/dixyes/mtgpu-drv

**Status**: Experimental - source provenance needs audit

**Integration Points**:
- `mtgpu.ko` DRM driver
- MUSA (CUDA-like API)
- MTML for machine learning

**Limitations**:
- Source audit pending
- Limited OpenCL/Vulkan extension support

---

## 6. Workload Scheduling Analysis

### 6.1 Implemented Strategies

| Strategy | Implementation | Use Case |
|----------|----------------|----------|
| Round-robin | `scheduler/strategy/round_robin.c` | Even distribution, general compute |
| AFR | `scheduler/strategy/afr.c` | Gaming - odd/even frames on different GPUs |
| SFR | `scheduler/strategy/sfr.c` | Gaming - horizontal/vertical tile split |
| Task-based | `scheduler/strategy/task.c` | Mixed graphics + compute |
| Compute offload | `scheduler/strategy/compute_offload.c` | Route compute to best GPU |
| Hybrid | `scheduler/strategy/hybrid.c` | Adaptive selection |
| Single GPU | - | Fallback / compatibility |

### 6.2 Scheduler Architecture

```
Workload Submission
       │
       ▼
PriorityQueue (16 priority levels)
       │
       ▼
Scheduling Algorithm (Static/Dynamic/Profile/AI)
       │
       ▼
Work-Stealing Check
       │
       ▼
GPU Assignment + Dispatch
```

### 6.3 AI-Driven Scheduling

- ONNX model integration planned (`scripts/mvgal-vulkan-env.sh` references)
- Features: GPU utilization, VRAM pressure, temperature
- Prediction: Recommended GPU for workload

---

## 7. Memory Virtualization Architecture

### 7.1 Three-Tier Transfer Strategy

```
┌─────────────────────────────────────────────────────────┐
│                    Application                           │
└─────────────────────┬───────────────────────────────────┘
                      │ Buffer allocation request
                      ▼
┌─────────────────────────────────────────────────────────┐
│           MVGAL Memory Manager Decision Tree             │
├─────────────────────────────────────────────────────────┤
│ 1. DMA-BUF viable?       │ Zero-copy, kernel supported    │
│ 2. PCIe P2P viable?      │ Direct GPU-GPU transfer        │
│ 3. Host-RAM staging      │ Always works, highest latency  │
└─────────────────────────────────────────────────────────┘
```

### 7.2 Memory Placement Policies

| Policy | Allocation Location | Trigger |
|--------|-------------------|---------|
| Small (< 64MB) | GPU with most free VRAM | Size threshold |
| Render target | GPU that writes first | Workload history |
| Large buffer | GPU with access pattern | Historical access |
| Shared buffer | Primary GPU + DMA-BUF to others | gpu_mask > 1 |

---

## 8. Data Transfer Layer

### 8.1 Transfer Methods

| Method | Bandwidth | Latency | Requirements |
|--------|-----------|---------|--------------|
| GPU-local VRAM | 400-900 GB/s | Very low | N/A |
| DMA-BUF zero-copy | 20-30 GB/s | Low | Same/compatible vendor |
| PCIe P2P | 12-16 GB/s | Medium | Same root complex, kernel 5.10+ |
| Host-RAM staging | 6-12 GB/s | High | Always available |

### 8.2 Synchronization

- DMA-BUF fences via `sync_file`
- Vulkan timeline semaphores
- MVGAL's Rust `fence_manager` crate

---

## 9. Kernel Module Analysis

### 9.1 Current Implementation (`mvgal_main.c`)

**Features implemented**:
- PCI device enumeration (`pci_get_device`)
- Character device `/dev/mvgal0` registration
- Sysfs interface under `/sys/class/mvgal/`
- IOCTL interface (10 commands)
- Hotplug monitoring via `bus_register_notifier`

**Features stubbed (require `MVGAL_FULL_STACK`)**:
- DMA-BUF export/import (`MVGAL_IOC_EXPORT_DMABUF`, `MVGAL_IOC_IMPORT_DMABUF`)
- Cross-vendor allocation (`MVGAL_IOC_ALLOC_CROSS_VENDOR`, `MVGAL_IOC_FREE_CROSS_VENDOR`)

### 9.2 IOCTL Interface

| IOCTL | Direction | Description |
|-------|-----------|-------------|
| `MVGAL_IOC_QUERY_VERSION` | Read | UAPI version |
| `MVGAL_IOC_GET_GPU_COUNT` | Read | Number of GPUs |
| `MVGAL_IOC_GET_GPU_INFO` | Read/Write | GPU descriptor |
| `MVGAL_IOC_ENABLE` | Write | Enable MVGAL |
| `MVGAL_IOC_DISABLE` | Write | Disable MVGAL |
| `MVGAL_IOC_GET_STATS` | Read | Usage statistics |
| `MVGAL_IOC_GET_CAPS` | Read | Aggregate capabilities |
| `MVGAL_IOC_RESCAN` | Write | Rescan PCI devices |
| `MVGAL_IOC_EXPORT_DMABUF` | Write | Export buffer as DMA-BUF |
| `MVGAL_IOC_IMPORT_DMABUF` | Write | Import DMA-BUF |
| `MVGAL_IOC_ALLOC_CROSS_VENDOR` | Read/Write | Unified allocation |
| `MVGAL_IOC_FREE_CROSS_VENDOR` | Write | Free unified allocation |

---

## 10. Security Architecture

### 10.1 Privilege Escalation

All privileged operations use `pkexec`, not `sudo`:
- Kernel module loading (`pkexec systemctl start mvgald`)
- GPU power state changes
- Configuration modification

### 10.2 IPC Authentication

- Unix socket `/run/mvgal/mvgal.sock`
- `SCM_CREDENTIALS` for process authentication
- Only `video` group members or root can submit workloads

### 10.3 PolicyKit Integration

- Policy file: `config/org.freedesktop.policykit.mvgal.policy`
- Least-privilege design

---

## 11. Packaging and Distribution

### 11.1 Build Systems

| System | Status | Notes |
|--------|--------|-------|
| CMake | Primary | Full build matrix |
| Meson | Alternative | `meson.build` present |
| Zig | Alternative | `build.zig` present |

### 11.2 Package Formats

| Format | Location | Status |
|--------|----------|--------|
| RPM | `packaging/rpm/mvgal.spec` | ✅ COPR builds |
| DEB | `packaging/deb/` | ✅ `build_deb.sh` |
| PKGBUILD | `packaging/arch/` | ✅ `makepkg -si` |

### 11.3 Distribution Support

Fedora 40-44/Rawhide, RHEL 9-10/AlmaLinux/Rocky, CentOS Stream 9-10, Ubuntu 22.04/24.04, Debian 12, Arch Linux, openSUSE Tumbleweed, Amazon Linux 2023.

---

## 12. Test Coverage

### 12.1 Test Suites

| Suite | Pass | Total | Notes |
|-------|------|-------|-------|
| C unit tests | 6 | 6 | Core, GPU, memory, scheduler, execution, config |
| C integration tests | 21 | 21 | Multi-GPU, UAPI probe, Vulkan layer, etc. |
| Rust unit tests | 12 | 12 | Fence, memory, capability model |
| **Total** | **88** | **88** | **100%** |

### 12.2 Benchmark Results (Reported)

| Benchmark | Single GPU | Dual GPU | Speedup |
|-----------|------------|----------|---------|
| Memory copy | baseline | 1.7× | ✅ |
| Compute (DAXPY) | baseline | 1.8× | ✅ |
| Scheduling latency avg | - | 1.81 µs | - |

---

## 13. Technical Limitations

### 13.1 Hard Incompatibilities

1. **No cross-vendor coherent VRAM**: DMA-BUF/PRIME provides zero-copy but requires driver support
2. **No universal command stream format**: Each vendor uses proprietary formats
3. **Anti-cheat blocks Vulkan layers**: EAC, BattlEye block custom layers
4. **MTT source provenance**: Community driver needs audit before production claims

### 13.2 Design Constraints

- MVGAL does not claim PCI devices from vendor drivers
- Character device (not full DRM) in Phase 1
- Transparent scaling is per-API, not universal
- Pass-through default for unsupported paths