# MVGAL Driver Architecture Design

**Version:** 1.0  
**Date:** May 2026  
**Status:** Implementation Phase

---

## 1. Executive Summary

This document defines the modular driver architecture for MVGAL (Multi-Vendor GPU Aggregation Layer for Linux). The architecture separates concerns between:

- **Kernel Module Layer**: PCI discovery, topology management, UAPI
- **Vendor Backend Interface**: Standardized operations structure
- **Vendor-Specific Shims**: Thin wrappers around vendor driver interfaces
- **Userspace Runtime**: Scheduling, memory management, API interception

---

## 2. Architectural Layers

### 2.1 Layer Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                        Applications                             │
│          (Games, Blender, PyTorch, OpenCL programs)             │
└────────────┬──────────────┬──────────────┬──────────────────────┘
             │ Vulkan        │ OpenCL        │ CUDA / OpenGL
             ▼              ▼              ▼
┌─────────────────────────────────────────────────────────────────┐
│                  Layer 5: API Interception                       │
│  VK_LAYER_MVGAL │ libmvgal_opencl.so │ libmvgal_cuda.so        │
│  libmvgal_gl.so (OpenGL LD_PRELOAD)                             │
└────────────────────────────┬────────────────────────────────────┘
                             │
┌────────────────────────────▼────────────────────────────────────┐
│                  Layer 4: Execution Engine                       │
│  Frame sessions · Migration plans · Steam/Proton profiles        │
│  (src/userspace/execution/execution.c)                          │
└────────────────────────────┬────────────────────────────────────┘
                             │ IPC (Unix socket /run/mvgal/mvgal.sock)
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│               Layer 3: User-Space Runtime Daemon                 │
│                         mvgald (C++20)                           │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────────────┐ │
│  │Scheduler │ │MemoryMgr │ │PowerMgr  │ │  MetricsCollector  │ │
│  └──────────┘ └──────────┘ └──────────┘ └────────────────────┘ │
└────────────────────────────┬────────────────────────────────────┘
                             │
┌────────────────────────────▼────────────────────────────────────┐
│               Layer 2: Rust Safety Subsystems                    │
│  fence_manager · memory_safety · capability_model               │
│  (safe/ — Rust, edition 2021, MSRV 1.75)                        │
└────────────────────────────┬────────────────────────────────────┘
                             │ ioctl / sysfs
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│               Layer 1: Kernel Abstraction Driver                 │
│                         mvgal.ko (GPL-2.0)                       │
│              /dev/mvgal0  (character device)                     │
└──────┬──────────────┬──────────────┬──────────────┬─────────────┘
       │              │              │              │
       ▼              ▼              ▼              ▼
  amdgpu.ko     nvidia.ko       i915/xe.ko    mtgpu-drv.ko
  (AMD GPU)    (NVIDIA GPU)    (Intel GPU)   (MTT GPU)
```

---

## 3. Layer 1: Kernel Module (`mvgal.ko`)

### 3.1 Responsibilities

| Responsibility | Description |
|----------------|-------------|
| PCI Discovery | Enumerate all display-class GPUs via PCI subsystem |
| UAPI Exposure | Character device `/dev/mvgal0` with ioctl interface |
| Topology Info | Export GPU hierarchy, PCIe links, NUMA nodes |
| DMA-BUF Coordination | Cross-GPU memory sharing support |
| Fence Primitives | Cross-vendor synchronization |
| Sysfs Interface | `/sys/class/mvgal/` for runtime control |

### 3.2 Module Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `enable_debug` | bool | `false` | Verbose kernel logging |
| `max_gpus` | uint | `32` | Maximum GPUs to enumerate |

### 3.3 Character Device Interface

**Device Node:** `/dev/mvgal0`

**Major/Minor:** Dynamic allocation via `alloc_chrdev_region`

**File Operations:**
- `open()` / `release()` - Reference counting
- `unlocked_ioctl()` - Dispatch to handler functions
- `compat_ioctl()` - 32-bit compatibility

### 3.4 IOCTL Interface

Defined in `include/mvgal/mvgal_uapi.h`:

| IOCTL | Direction | Handler | Description |
|-------|-----------|---------|-------------|
| `MVGAL_IOC_QUERY_VERSION` | Read | `mvgal_ioctl_query_version` | UAPI version check |
| `MVGAL_IOC_GET_GPU_COUNT` | Read | `mvgal_ioctl_get_gpu_count` | Number of detected GPUs |
| `MVGAL_IOCTL_GET_GPU_INFO` | Read | `mvgal_ioctl_get_gpu_info` | Per-GPU information |
| `MVGAL_IOCTL_ENABLE` | Write | `mvgal_ioctl_enable` | Enable aggregation |
| `MVGAL_IOCTL_DISABLE` | Write | `mvgal_ioctl_disable` | Disable aggregation |
| `MVGAL_IOCTL_GET_STATS` | Read | `mvgal_ioctl_get_stats` | Runtime statistics |
| `MVGAL_IOCTL_GET_CAPS` | Read | `mvgal_ioctl_get_caps` | Unified capability profile |
| `MVGAL_IOCTL_RESCAN` | Write | `mvgal_ioctl_rescan` | Trigger GPU rescan |
| `MVGAL_IOCTL_EXPORT_DMABUF` | Read/Write | `mvgal_ioctl_export_dmabuf` | DMA-BUF export |
| `MVGAL_IOCTL_IMPORT_DMABUF` | Read/Write | `mvgal_ioctl_import_dmabuf` | DMA-BUF import |
| `MVGAL_IOCTL_WAIT_FENCE` | Write | `mvgal_ioctl_wait_fence` | Fence synchronization |
| `MVGAL_IOCTL_SIGNAL_FENCE` | Write | `mvgal_ioctl_signal_fence` | Fence signaling |

---

## 4. Layer 2: Rust Safety Subsystems

### 4.1 `fence_manager`

**Purpose:** Cross-device fence lifecycle management

**State Machine:**
```
Pending → Submitted → Signalled → Reset
```

**C FFI Interface:**
```c
uint64_t mvgal_fence_create(uint32_t gpu_index);
void     mvgal_fence_submit(uint64_t handle);
void     mvgal_fence_signal(uint64_t handle);
uint32_t mvgal_fence_state(uint64_t handle);   // 0=Pending, 1=Submitted, 2=Signalled, 3=Reset
void     mvgal_fence_reset(uint64_t handle);
void     mvgal_fence_destroy(uint64_t handle);
```

### 4.2 `memory_safety`

**Purpose:** Safe allocation tracking with reference counting

**Placements:**
- `SystemRam` - Host memory
- `GpuVram` - GPU video memory
- `Mirrored` - Both CPU and GPU accessible

**C FFI Interface:**
```c
uint64_t mvgal_mem_track(uint64_t size, uint32_t placement);
void     mvgal_mem_retain(uint64_t handle);
void     mvgal_mem_release(uint64_t handle);
void     mvgal_mem_set_dmabuf(uint64_t handle, int32_t fd);
uint64_t mvgal_mem_size(uint64_t handle);
uint32_t mvgal_mem_placement(uint64_t handle);
```

### 4.3 `capability_model`

**Purpose:** GPU capability normalization and JSON export

**Tiers:**
- `Full` - All GPUs support same API set
- `ComputeOnly` - Graphics aggregation not possible
- `Mixed` - Mixed API support

**C FFI Interface:**
```c
uint64_t    mvgal_cap_compute(const GpuCapability *caps, uint32_t count);
void        mvgal_cap_free(uint64_t handle);
uint64_t    mvgal_cap_total_vram(uint64_t handle);
uint32_t    mvgal_cap_tier(uint64_t handle);
const char *mvgal_cap_to_json(uint64_t handle);
```

---

## 5. Layer 3: Runtime Daemon (`mvgald`)

### 5.1 Subsystems

| Subsystem | File | Purpose |
|-----------|------|---------|
| DeviceRegistry | `device_registry.cpp` | GPU enumeration, capability detection |
| Scheduler | `scheduler.cpp` | Workload distribution (7 strategies) |
| MemoryManager | `memory_manager.cpp` | Cross-GPU memory allocation |
| PowerManager | `power_manager.cpp` | Idle detection, DVFS, thermal |
| MetricsCollector | `metrics_collector.cpp` | Sysfs polling, telemetry |
| IpcServer | `ipc_server.cpp` | Unix socket communication |
| DbusService | `dbus_service.cpp` | D-Bus API integration |

### 5.2 IPC Protocol

**Unix Domain Socket:** `/run/mvgal/mvgal.sock`

**Message Format:**
```
┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│  magic   │ version  │  type    │ reqId    │ payloadSz│  flags   │ reserved │
│ 'MVGL'   │    1     │ uint32   │ uint32   │ uint32   │ uint32   │ uint32   │
│ 4 bytes  │ 4 bytes  │ 4 bytes  │ 4 bytes  │ 4 bytes  │ 4 bytes  │ 4 bytes  │
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
```

**Authentication:** `SCM_CREDENTIALS` - Only `video` group members or root

---

## 6. Layer 4: Execution Engine

### 6.1 Frame Session Lifecycle

```c
mvgal_execution_begin_frame()   → allocates frame_id, records API/strategy/app
mvgal_execution_submit()        → creates execution plan, selects GPUs, chooses migration path
mvgal_execution_present()       → finalizes frame, triggers frame pacer
mvgal_execution_get_frame_stats() → returns timing, GPU list, bytes migrated
```

### 6.2 Migration Path Selection

| Path | Flag | Description |
|------|------|-------------|
| Stream | `MVGAL_EXECUTION_MIGRATION_STREAM` | DMA-BUF zero-copy |
| P2P | `MVGAL_EXECUTION_MIGRATION_STREAM` + P2P | PCIe direct |
| Evict | `MVGAL_EXECUTION_MIGRATION_EVICT` | Host staging |
| Mirror | `MVGAL_EXECUTION_MIGRATION_MIRROR` | Replicate data |

---

## 7. Layer 5: API Interception

### 7.1 Vulkan Layer

**Library:** `libVK_LAYER_MVGAL.so`

**Registration:** `/usr/share/vulkan/implicit_layer.d/VK_LAYER_MVGAL.json`

**Intercepted Functions:**
- Instance: `vkCreateInstance`, `vkDestroyInstance`, `vkEnumeratePhysicalDevices`
- Device: `vkCreateDevice`, `vkDestroyDevice`, `vkGetDeviceQueue`, `vkGetDeviceQueue2`
- Queue: `vkQueueSubmit`, `vkQueueSubmit2`, `vkQueueSubmit2KHR`
- Physical Device: `vkGetPhysicalDeviceProperties`, `vkGetPhysicalDeviceFeatures`, `vkGetPhysicalDeviceMemoryProperties`

### 7.2 OpenCL Layer

**Library:** `libmvgal_opencl.so`

**Registration:** `/etc/OpenCL/vendors/mvgal.icd`

**Features:**
- Single platform presenting all GPUs
- NDRange kernel partitioning
- DMA-BUF buffer sharing

### 7.3 CUDA Shim

**Library:** `libmvgal_cuda.so` (LD_PRELOAD)

**Intercepted Functions:**
- Memory: `cuMemAlloc`, `cuMemFree`, `cuMemcpy*`
- Launch: `cuLaunchKernel`, `cudaLaunchKernel`
- Context: `cuCtxCreate`, `cuCtxPushDynamic75`

---

## 8. Scheduling Strategies

| Strategy | Description | Best For |
|----------|-------------|----------|
| `round_robin` | Even distribution across GPUs | General compute |
| `afr` | Alternate frames (odd/even split) | Gaming |
| `sfr` | Screen tiles (horizontal/vertical) | Gaming |
| `task` | Route by workload type | Mixed workloads |
| `compute_offload` | Route compute to best GPU | AI/HPC |
| `hybrid` | Adaptive strategy selection | General use |
| `single` | Use only primary GPU | Fallback/compatibility |

---

## 9. Memory Management

### 9.1 Three-Tier Transfer Strategy

```
1. DMA-BUF zero-copy   (preferred — kernel-supported)
2. PCIe P2P transfer   (fallback — same root complex)
3. Host-RAM staging    (last resort — always works)
```

### 9.2 Allocation Policy

| Condition | Policy |
|-----------|--------|
| Size < 64 MB | GPU with most free VRAM |
| Render target | GPU that will write first |
| Large buffer | GPU likely to use it (access pattern) |

### 9.3 Memory Flags

| Flag | Value | Description |
|------|-------|-------------|
| `HOST_VALID` | 1<<0 | CPU can access |
| `GPU_VALID` | 1<<1 | GPUs can access |
| `SHARED` | 1<<4 | Shared across GPUs |
| `DMA_BUF` | 1<<5 | Use DMA-BUF |
| `P2P` | 1<<6 | Enable P2P transfers |
| `REPLICATED` | 1<<7 | Mirror across GPUs |

---

## 10. Vendor Backend Interface

### 10.1 `struct mvgal_vendor_ops`

Each vendor implements this interface:

```c
struct mvgal_vendor_ops {
    int  (*init)(struct mvgal_gpu_device *dev);
    void (*fini)(struct mvgal_gpu_device *dev);
    int  (*submit_cs)(struct mvgal_gpu_device *dev, struct mvgal_workload *wl);
    int  (*alloc_vram)(struct mvgal_gpu_device *dev, size_t size, uint64_t *addr);
    void (*free_vram)(struct mvgal_gpu_device *dev, uint64_t addr);
    int  (*wait_idle)(struct mvgal_gpu_device *dev, uint32_t timeout_ms);
    int  (*set_power_state)(struct mvgal_gpu_device *dev, enum mvgal_power_state);
    struct dma_buf *(*export_dmabuf)(struct mvgal_gpu_device *dev, uint64_t addr, size_t size);
    int  (*import_dmabuf)(struct mvgal_gpu_device *dev, struct dma_buf *buf, uint64_t *addr);
    int  (*query_utilization)(struct mvgal_gpu_device *dev, uint32_t *util_percent);
    int  (*query_vram)(struct mvgal_gpu_device *dev, uint64_t *used, uint64_t *total);
    int  (*query_temperature)(struct mvgal_gpu_device *dev, int32_t *temp_c);
    int  (*query_power)(struct mvgal_gpu_device *dev, uint32_t *power_w);
};
```

### 10.2 Integration Notes by Vendor

#### AMD (amdgpu)
- Can access DRM device via `pci_get_drvdata()`
- Use `drm_gem_prime_export/import` for DMA-BUF
- Read utilization from `/sys/class/drm/cardX/device/gpu_busy_percent`

#### NVIDIA
- Cannot access proprietary driver internals from kernel
- Must use user-space daemon for actual submission
- Kernel module provides topology and coordination only

#### Intel (i915/xe)
- Full open-source driver access
- Use `drm_i915_gem_execbuffer2` (i915) or `xe_exec` (xe)
- DMA-BUF via `drm_gem_prime_export/import`

---

## 11. Build System

### 11.1 CMake Options

```cmake
DMVGAL_BUILD_KERNEL      - Build kernel module
DMVGAL_BUILD_RUNTIME     - Build daemon
DMVGAL_BUILD_API         - Build userspace library
DMVGAL_BUILD_TOOLS       - Build CLI tools
DMVGAL_BUILD_TESTS       - Build tests
DMVGAL_ENABLE_RUST       - Enable Rust crates
DMVGAL_BUILD_FULL_STACK  - Enable full aggregation (requires MVGAL_FULL_STACK)
```

### 11.2 Rust Features

```toml
[features]
default = []
full-stack = []
```

---

## 12. Testing Strategy

### 12.1 Unit Tests

| Component | Language | Framework |
|-----------|----------|-----------|
| Kernel module | C | KUnit |
| Runtime daemon | C++ | GoogleTest |
| Rust crates | Rust | Built-in |
| Userspace API | C | Custom |

### 12.2 Integration Tests

| Test | Description |
|------|-------------|
| `test_gpu_detection` | Enumerate all available GPUs |
| `test_dmabuf_integration` | Cross-GPU memory transfer |
| `test_vulkan_layer_submit` | Intercept and submit workloads |
| `test_opencl_intercept` | NDRange partitioning |
| `test_unified_heap` | Memory allocation across GPUs |

---

## 13. References

- Linux DRM UAPI: https://dri.freedesktop.org/docs/drm/
- Vulkan Device Groups: https://registry.khronos.org/vulkan/specs/latest/html/chap12.html
- DMA-BUF: https://www.kernel.org/doc/html/latest/driver-api/dma-buf.html
- NVIDIA open-gpu-kernel-modules: https://github.com/NVIDIA/open-gpu-kernel-modules
- AMDVLK: https://github.com/GPUOpen-Drivers/AMDVLK
- Intel media-driver: https://github.com/intel/media-driver