# MVGAL: Multi-Vendor GPU Aggregation Layer for Linux — Design Document

**Date:** 2026-05-23
**Status:** Draft / Phase 1 Complete
**Brainstorm Session:** `ses_kjl2fm25`

---

## Problem

Modern Linux systems frequently have multiple GPUs from different vendors (NVIDIA, AMD, Intel, Moore Threads) installed simultaneously, but existing software stacks treat each GPU as an isolated device. Applications can only use one GPU at a time or require explicit multi-GPU programming per vendor (SLI, CrossFire, NVLink). This wastes available GPU resources and forces users to choose between vendor ecosystems.

**Core problem:** No existing open-source solution provides a transparent, vendor-agnostic layer that aggregates heterogeneous GPUs into a single logical compute device for any application — gaming, rendering, AI/ML, or general compute.

---

## Findings (Brainstorm Results)

The following design decisions were established through the brainstorming session:

### 1. Kernel Architecture: Hybrid (kernel core + userspace plugins)

**Choice:** Minimal kernel module for PCIe/sysfs-based GPU detection and a `/dev/mvgal` ioctl interface. Heavy lifting (memory management, scheduling, API translation) lives in userspace.

**Rationale:**
- Kernel space should do only what kernel space must do: device discovery, interrupt handling, DMA buffer management.
- Userspace is more flexible for rapid iteration, easier to debug, and avoids GPL compatibility issues with vendor userspace drivers (CUDA, ROCm, etc.).
- Vendor GPU drivers already exist in kernel (NVIDIA's nvidia.ko, AMD's amdgpu.ko, Intel's i915.ko). MVGAL should interoperate with them, not replace them.
- A fat kernel module would be harder to maintain across kernel versions and vendor driver changes.

### 2. Userspace API: Custom MVGAL ABI with backend translators

**Choice:** Define a new vendor-agnostic GPU ABI (`libmvgal.so`). Applications link directly or indirectly against libmvgal, which translates calls to vendor-specific backends.

**Rationale:**
- A custom ABI gives full control over multi-GPU semantics rather than trying to retrofit onto Vulkan or OpenGL.
- Backend translators for each vendor API: OpenGL→MVGAL, Vulkan→MVGAL, CUDA→MVGAL via LD_PRELOAD wrappers.
- The ICD loader pattern was rejected because it constrains the design to Vulkan's single-device model.
- The Vulkan-centric interception layer was rejected because not all workloads go through Vulkan (CUDA compute, OpenGL legacy).

### 3. Workload Scheduler: Hybrid capability-aware dynamic scheduler

**Choice:** Profile GPUs at init, build a capability matrix. For gaming: default to Alternate Frame Rendering (AFR) with dynamic rebalancing. For compute: split workload by GPU compute ratio. For rendering: Split Frame Rendering (SFR) with tile-size awareness.

**Rationale:**
- No single scheduling strategy works for all workloads. Gaming needs low latency (AFR works best), rendering needs parallel tiling (SFR), compute needs throughput-based splitting.
- The scheduler auto-detects workload type by observing API calls (e.g., swapchain present → gaming path, compute dispatch → compute path).
- Dynamic rebalancing handles frame time variance across heterogeneous GPUs.

### 4. Memory Management: Hybrid NUMA-like model + explicit API

**Choice:** Expose memory regions to the application. Provide both automatic migration hints and an explicit transfer API. Reads may go remote, writes can be broadcast. The application can choose per-buffer policy: mirror or distribute.

**Rationale:**
- No hardware coherence exists across vendor GPUs (unlike NVLink which is NVIDIA-only).
- Full mirroring wastes VRAM (3 GPUs × 8 GB = 24 GB used for 8 GB of data).
- Pure page migration has high overhead for graphics workloads with random access patterns.
- The hybrid approach lets games mirror framebuffers (simple, fast) while letting compute workloads distribute data (memory-efficient).

### 5. Power Management: Hybrid kernel monitoring + userspace policy daemon

**Choice:** Kernel module monitors GPU activity and exposes utilization metrics via sysfs. Userspace policy daemon (`mvgal-powertune`) makes decisions about which GPUs to enable, power limits, and idle states.

**Rationale:**
- Kernel does measurement (low-level, accurate, real-time).
- Userspace does policy (complex logic, user preferences, PSU awareness).
- Clean separation: kernel exposes metrics, userspace implements the governor algorithm.
- Vendor-specific power interfaces (nvidia-smi, rocm-smi, sysfs) are only accessible from userspace.

### 6. Windows Compatibility: LD_PRELOAD wrapper + Vulkan layer

**Choice:** Use LD_PRELOAD to intercept CUDA/OpenGL calls and redirect to MVGAL. A Vulkan layer handles Vulkan workloads. Separate DLL wrappers for Wine/Proton.

**Rationale:**
- No need to fork Proton — works with upstream Wine/Proton.
- The Vulkan layer (VK_LAYER_MVGAL) intercepts at the Vulkan level, which is where DXVK/VKD3D translate DirectX calls.
- LD_PRELOAD wrappers handle native Linux CUDA and OpenGL applications.
- NTSYNC implemented as a separate kernel patch (upstream since Linux 5.16 as real-time mutex, but expose via futex for Wine).

### 7. Build System: Multi-tool approach

**Choice:** Each component uses the best tool: kbuild for kernel module, Cargo for Rust userspace components, CMake for C++ components, Python setuptools for tools. Top-level Makefile ties everything together.

**Rationale:**
- Linux kernel modules require kbuild (it's not optional).
- Rust ecosystem uses Cargo — fighting it with CMake is counterproductive.
- CMake for C++ components gives excellent IDE support and CPack for packaging.
- Python setuptools for CLI tools matches distro expectations.
- A top-level `Makefile` provides the unified entry point: `make`, `make install`, `make deb`, `make rpm`.

---

## Recommended Architecture

### System Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                        APPLICATION LAYER                        │
│  (Game / Blender / PyTorch / Custom App)                        │
│  Links: libmvgal.so | LD_PRELOAD intercepted | Vulkan layer     │
└───────────────────────────┬─────────────────────────────────────┘
                            │
┌───────────────────────────▼─────────────────────────────────────┐
│                    USERSPACE (libmvgal.so)                        │
│                                                                   │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────────┐  │
│  │ MVGAL Core  │  │  Scheduler   │  │  Memory Manager        │  │
│  │  (C++ ABI)  │──│  Engine      │──│  (UVA + Migration)     │  │
│  └──────┬──────┘  └──────────────┘  └───────────┬────────────┘  │
│         │                                        │               │
│  ┌──────▼────────────────────────────────────────▼──────────┐   │
│  │           Backend Translators                             │   │
│  │  ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐  │   │
│  │  │Vulkan  │ │OpenGL  │ │ CUDA   │ │ OpenCL │ │  HIP   │  │   │
│  │  │ ICD    │ │(glx)   │ │Wrapper │ │Wrapper │ │Wrapper │  │   │
│  │  └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘  │   │
│  └──────┼──────────┼──────────┼──────────┼──────────┼───────┘   │
└─────────┼──────────┼──────────┼──────────┼──────────┼───────────┘
          │          │          │          │          │
┌─────────▼──────────▼──────────▼──────────▼──────────▼───────────┐
│                    KERNEL SPACE (mvgal.ko)                       │
│                                                                   │
│  ┌──────────────┐  ┌────────────────┐  ┌──────────────────────┐  │
│  │ PCIe Prober  │  │ /dev/mvgal     │  │ Power/Thermal        │  │
│  │ (sysfs scan) │  │ ioctl dispatch │  │ Monitor (sysfs)      │  │
│  └──────┬───────┘  └────────────────┘  └──────────────────────┘  │
│         │                                                         │
│  ┌──────▼──────────────────────────────────────────────────────┐  │
│  │              Vendor Kernel Shim Wrappers                     │  │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐       │  │
│  │  │ nvidia.ko│ │ amdgpu.ko│ │ i915.ko  │ │ mtt.ko   │       │  │
│  │  │ (NVIDIA) │ │ (AMD)    │ │ (Intel)  │ │ (MTT)    │       │  │
│  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘       │  │
│  └─────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
          │          │          │          │          │
    ┌─────┴────┐┌───┴────┐┌───┴────┐┌───┴────┐┌───┴────┐
    │ GPU 0    ││ GPU 1  ││ GPU 2  ││ GPU 3  ││ GPU N  │
    │ (NVIDIA) ││ (AMD)  ││ (Intel)││ (MTT)  ││ (...)  │
    └──────────┘└────────┘└────────┘└────────┘└────────┘
```

### Repository Structure

```
mvgal/
├── Makefile                    # Top-level build orchestrator
├── README.md
├── LICENSE
├── docs/
│   ├── architecture.md
│   ├── api/
│   │   ├── libmvgal.md         # libmvgal ABI reference
│   │   └── ioctl.md            # Kernel ioctl interface
│   ├── user-guide.md
│   └── development.md
│
├── kernel/                     # Linux kernel module (C, kbuild)
│   ├── Kbuild
│   ├── Makefile
│   ├── src/
│   │   ├── mvgal_main.c        # Module init/exit, /dev/mvgal char device
│   │   ├── mvgal_pcie.c        # PCIe device discovery via sysfs
│   │   ├── mvgal_ioctl.c       # ioctl dispatch table
│   │   ├── mvgal_dma.c         # DMA buffer management
│   │   ├── mvgal_monitor.c     # Power/thermal/sysfs metrics
│   │   ├── mvgal_sync.c        # Cross-device synchronization primitives
│   │   └── include/
│   │       ├── mvgal.h         # Public kernel header
│   │       └── mvgal_internal.h
│   └── tests/
│       └── test_ioctl.c
│
├── libmvgal/                   # Userspace core library (Rust)
│   ├── Cargo.toml
│   ├── src/
│   │   ├── lib.rs              # libmvgal ABI entry points
│   │   ├── device.rs           # GPU enumeration, capability discovery
│   │   ├── context.rs          # Device context management
│   │   ├── memory.rs           # Unified memory management
│   │   ├── scheduler/          # Workload scheduler (see below)
│   │   ├── sync.rs             # Synchronization primitives
│   │   ├── power.rs            # Power management client
│   │   └── ioctl.rs            # Kernel ioctl wrappers
│   └── tests/
│
├── backends/                   # Vendor backend translators (Rust/C++)
│   ├── backend-vulkan/         # Vulkan ICD + layer (Rust)
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── physical_device.rs     # vkEnumeratePhysicalDevices
│   │       ├── device.rs              # vkCreateDevice → multi-device
│   │       ├── queue.rs               # vkQueueSubmit → scheduler
│   │       └── memory.rs              # vkAllocateMemory → memory manager
│   │
│   ├── backend-opengl/         # OpenGL interposer (LD_PRELOAD, C++)
│   │   ├── CMakeLists.txt
│   │   └── src/
│   │       ├── gl_intercept.cpp
│   │       ├── glx_intercept.cpp
│   │       └── egl_intercept.cpp
│   │
│   ├── backend-cuda/           # CUDA wrapper (LD_PRELOAD, C/Rust)
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs          # cuInit, cuMemAlloc, cuLaunchKernel wrappers
│   │       └── context_split.rs
│   │
│   ├── backend-opencl/         # OpenCL wrapper (C++)
│   │   └── CMakeLists.txt
│   │
│   └── backend-hip/            # HIP wrapper (C++)
│       └── CMakeLists.txt
│
├── power/                      # Power management userspace (Rust)
│   ├── Cargo.toml
│   └── src/
│       ├── main.rs             # mvgal-powertune daemon
│       ├── monitor.rs          # Read kernel metrics
│       ├── policy.rs           # Power policy engine
│       ├── vendor_hooks.rs     # nvidia-smi, rocm-smi wrappers
│       └── dbus_api.rs         # D-Bus config interface
│
├── tools/                      # CLI tools (Python/Rust)
│   ├── mvgal-info/             # GPU status CLI
│   │   └── main.rs
│   ├── mvgal-top/              # Real-time GPU utilization monitor
│   │   └── main.rs
│   ├── mvgal-config/           # Configuration tool
│   │   └── main.rs
│   └── mvgal-bench/           # Benchmark suite
│       └── bench.py
│
├── wine/                       # Wine/Proton integration
│   ├── mvgal-wine-dll/         # Windows DLL wrappers
│   │   └── src/
│   │       ├── d3d12_mvgal.cpp # DX12 explicit multi-adapter → MVGAL
│   │       ├── dxgi_mvgal.cpp  # DXGI adapter enumeration
│   │       └── ntsync.c        # NTSYNC primitives
│   ├── setup_mvgal.sh          # Proton prefix setup script
│   └── README.md
│
├── ci/                         # CI/CD configurations
│   ├── Dockerfile.ubuntu
│   ├── Dockerfile.fedora
│   ├── Dockerfile.arch
│   └── github-actions.yml
│
├── packaging/                  # Distro packaging
│   ├── debian/
│   │   ├── control
│   │   ├── rules
│   │   └── ...
│   ├── rpm/
│   │   ├── mvgal.spec
│   │   └── ...
│   └── arch/
│       └── PKGBUILD
│
└── tests/                      # Integration & performance tests
    ├── integration/
    │   ├── test_device_enum.py
    │   ├── test_memory_mirror.py
    │   ├── test_memory_distribute.py
    │   └── test_scheduler_afr.py
    ├── benchmarks/
    │   ├── bench_render.py
    │   └── bench_compute.py
    └── fixtures/
        └── test_apps/
```

---

## Detailed Design

### 1. Kernel Module (`mvgal.ko`)

#### 1.1 Device Discovery

On module load:
1. Scan `/sys/bus/pci/devices/` for GPU-class devices (PCI class code 0x0300).
2. Match vendor IDs against known vendors (NVIDIA 0x10de, AMD 0x1002, Intel 0x8086, MTT 0x1ed5).
3. For each GPU, read vendor-specific capabilities from sysfs and vendor driver status.
4. Register each GPU as a sub-device in MVGAL's internal device tree.
5. Create `/dev/mvgal` character device (major number allocated dynamically, minor 0).

```c
// mvgal_pcie.c — Simplified device detection
struct mvgal_device {
    struct list_head    list;
    u16                 vendor_id;
    u16                 device_id;
    u8                  bus_nr;
    u8                  devfn;
    u64                 vram_size;
    struct pci_dev      *pdev;
    struct mvgal_ops    *vendor_ops;  // Shim to vendor driver
};

// Vendor shim ops — each vendor plugin fills this
struct mvgal_ops {
    int (*get_power_state)(struct mvgal_device *dev, u32 *mw);
    int (*set_power_state)(struct mvgal_device *dev, enum mvgal_power_state state);
    int (*get_utilization)(struct mvgal_device *dev, struct mvgal_util *util);
    int (*reset_device)(struct mvgal_device *dev);
};
```

#### 1.2 ioctl Interface

The `/dev/mvgal` ioctl interface is the primary kernel-userspace contract:

```c
enum mvgal_ioctl_cmd {
    MVGAL_IOCTL_GET_DEVICE_COUNT,       // Return number of detected GPUs
    MVGAL_IOCTL_GET_DEVICE_INFO,        // Return detailed info for one GPU
    MVGAL_IOCTL_ALLOCATE_DMA_BUF,       // Allocate DMA buffer on specific GPU
    MVGAL_IOCTL_FREE_DMA_BUF,           // Free DMA buffer
    MVGAL_IOCTL_EXPORT_DMA_BUF,         // Export DMA buf for cross-GPU sharing
    MVGAL_IOCTL_IMPORT_DMA_BUF,         // Import DMA buf from another GPU
    MVGAL_IOCTL_GET_METRICS,            // Get power/thermal/util metrics
    MVGAL_IOCTL_SUBMIT_SYNC,            // Submit synchronization fence
    MVGAL_IOCTL_WAIT_SYNC,              // Wait on synchronization fence
};

struct mvgal_ioctl_device_info {
    __u32   device_index;
    __u16   vendor_id;
    __u16   device_id;
    __u8    pci_bus;
    __u8    pci_dev;
    __u8    pci_fn;
    __u64   vram_size_bytes;
    __u32   max_clock_mhz;
    __u32   num_queues;
    char    driver_name[64];             // e.g., "nvidia", "amdgpu", "i915"
};

struct mvgal_ioctl_dma_buf {
    __u32   device_index;
    __u64   size;
    __u64   handle;                      // Opaque handle for userspace
    __u32   flags;                       // MVGAL_DMA_FLAG_*
};
```

#### 1.3 Power/Thermal Monitoring

The kernel module reads GPU power/thermal data from vendor driver interfaces and exposes it via sysfs:

```
/sys/kernel/mvgal/
├── devices/
│   ├── gpu0/
│   │   ├── vendor
│   │   ├── power_mw              // Current power draw (milliwatts)
│   │   ├── temp_celsius          // Current temperature
│   │   ├── utilization_pct       // GPU utilization 0-100
│   │   ├── vram_used_bytes
│   │   ├── power_state           // "active" | "idle" | "suspended"
│   │   └── power_cap_mw          // Power cap (settable)
│   ├── gpu1/
│   │   └── ...
│   └── total_power_mw            // Sum of all GPU power
└── control/
    ├── enable_all         // 0/1 — enable/disable all GPUs
    └── global_power_cap   // Global power budget for all GPUs (settable)
```

### 2. Userspace Library (`libmvgal.so`)

#### 2.1 Core API

The ABI is designed for C linkage so any language can bind to it:

```c
// libmvgal.h — Core MVGAL ABI

// === Device Management ===
int mvgal_init(void);
void mvgal_shutdown(void);
int mvgal_device_count(void);
int mvgal_device_info(int index, struct mvgal_device_info *info);

// === Context Management ===
mvgal_context_t mvgal_create_context(const struct mvgal_context_desc *desc);
void mvgal_destroy_context(mvgal_context_t ctx);

// === Memory Management ===
mvgal_buffer_t mvgal_allocate_buffer(mvgal_context_t ctx, size_t size,
                                      enum mvgal_memory_policy policy,
                                      uint32_t flags);
void mvgal_free_buffer(mvgal_buffer_t buf);
void* mvgal_map_buffer(mvgal_buffer_t buf, enum mvgal_map_access access);
void mvgal_unmap_buffer(mvgal_buffer_t buf);
int mvgal_migrate_buffer(mvgal_buffer_t buf, uint32_t target_device_mask);
int mvgal_copy_buffer(mvgal_buffer_t dst, mvgal_buffer_t src, size_t size);

// === Workload Submission ===
mvgal_command_t mvgal_create_command(mvgal_context_t ctx,
                                      enum mvgal_command_type type);
void mvgal_command_set_kernel(mvgal_command_t cmd, const void *spirv,
                               size_t spirv_size);
void mvgal_command_set_push_const(mvgal_command_t cmd, uint32_t offset,
                                   const void *data, size_t size);
int mvgal_submit(mvgal_context_t ctx, mvgal_command_t cmd,
                  enum mvgal_scheduling_hint hint);

// === Synchronization ===
mvgal_fence_t mvgal_create_fence(mvgal_context_t ctx);
int mvgal_wait_for_fence(mvgal_fence_t fence, uint64_t timeout_ns);
int mvgal_reset_fence(mvgal_fence_t fence);

// === Power Management ===
int mvgal_set_power_cap(int device_index, uint32_t power_mw);
int mvgal_get_power_metrics(int device_index, struct mvgal_power_metrics *m);
int mvgal_set_global_power_cap(uint32_t total_mw);

// === Scheduling Hints (for Application Control) ===
enum mvgal_scheduling_hint {
    MVGAL_SCHED_DEFAULT       = 0,       // Auto-detect
    MVGAL_SCHED_LOW_LATENCY   = 1,       // Gaming: prefer AFR
    MVGAL_SCHED_THROUGHPUT    = 2,       // Compute: split by ratio
    MVGAL_SCHED_RENDER_TILED  = 3,       // Rendering: SFR
    MVGAL_SCHED_SPECIALIZE    = 4,       // By GPU strength
};

// === Memory Policies ===
enum mvgal_memory_policy {
    MVGAL_MEM_MIRRORED       = 0,   // All GPUs have full copy
    MVGAL_MEM_DISTRIBUTED    = 1,   // Pages distributed across GPUs
    MVGAL_MEM_HETEROGENEOUS  = 2,   // Vendor-specific placement
};
```

#### 2.2 Backend Translator Architecture

Each backend translator follows the same pattern:

```
Application call (e.g., cuMemAlloc)
    │
    ▼
LD_PRELOAD wrapper intercepts
    │
    ▼
Translates to MVGAL ABI call (mvgal_allocate_buffer)
    │
    ▼
MVGAL Core determines policy (mirror/distribute)
    │
    ▼
Backend-specific implementation:
    ┌────────────────┬────────────────┬────────────────┐
    │ NVIDIA backend │ AMD backend    │ Intel backend  │
    │ cuMemAlloc     │ hipMalloc      │ zeMemoryAlloc  │
    │ + migration    │ + migration    │ + migration    │
    └────────────────┴────────────────┴────────────────┘
```

### 3. Workload Scheduler

#### 3.1 Capability Matrix

On initialization, the scheduler builds a capability matrix:

```rust
// scheduler/capability.rs
#[derive(Clone, Debug)]
struct GpuCapability {
    device_index: u32,
    vendor: Vendor,
    model: String,
    vram_bytes: u64,
    compute_units: u32,
    // Benchmarked performance
    fp32_flops: f64,
    fp16_flops: f64,
    memory_bandwidth: f64,  // GB/s
    // Feature support
    supports_cuda: bool,
    supports_hip: bool,
    supports_sycl: bool,
    supports_raytracing: bool,
    supports_tensor_cores: bool,
    supports_media_encode: bool,
    // Current state (updated dynamically)
    current_load_pct: f32,
    current_power_mw: u32,
    current_temp: f32,
}

struct CapabilityMatrix {
    gpus: Vec<GpuCapability>,
    total_flops: f64,
    total_vram: u64,
    compute_ratio: Vec<f64>, // Per-GPU fraction of total compute
}
```

#### 3.2 Scheduling Modes

**AFR (Alternate Frame Rendering) — Gaming:**
```
Frame N    → GPU 0 (submit)
Frame N+1  → GPU 1 (submit)
Frame N+2  → GPU 0 (already finishing N)
Frame N+3  → GPU 1 (already finishing N+1)
```
- Dynamic rebalancing: If GPU 0 takes 16ms and GPU 1 takes 24ms, adjust ratio to 60:40.
- Present queue sorts frames by completion order to prevent tearing.
- Fence-based synchronization ensures GPU 1 doesn't start N+1 until GPU 0 has the necessary shared resources ready.

**SFR (Split Frame Rendering) — Rendering/Modeling:**
```
Frame split into tiles:
┌──────────┬──────────┐
│ GPU 0    │ GPU 1    │
│ tile 0,0 │ tile 0,1 │
├──────────┼──────────┤
│ GPU 2    │ tile     │
│ tile 1,0 │ border   │
├──────────┴──────────┤
│ GPU 1    │ GPU 0    │
│ tile 2,0 │ tile 2,1 │
└─────────────────────┘
```
- Tile sizes determined by GPU compute ratio and memory bandwidth.
- Tile borders require synchronization at fragment completion.

**Compute Splitting — AI/ML:**
```
Tensor operation (e.g., matrix multiply on A × B):
┌─────────────────────┐
│ A0×B │ A1×B │ A2×B │  ← GPU 0: rows 0-1
│──────┼──────┼──────│
│ A3×B │ A4×B │ A5×B │  ← GPU 1: rows 2-3
│──────┼──────┼──────│
│ A6×B │ A7×B │ A8×B │  ← GPU 2: rows 4-5
└─────────────────────┘
Row split by compute ratio. All-gather result via DMA.
```

#### 3.3 Scheduler Architecture

```rust
// scheduler/mod.rs
pub enum SchedulerMode {
    Auto,           // Auto-detect from API calls
    Gaming(VideoScheduler),
    Compute(ComputeScheduler),
    Rendering(RenderScheduler),
}

pub struct WorkloadScheduler {
    mode: SchedulerMode,
    capability: CapabilityMatrix,
    stats: SchedulerStats,
    // Per-GPU queue depths for dynamic balancing
    queue_depths: Vec<QueueDepth>,
}

impl WorkloadScheduler {
    /// Called on every workload submission
    pub fn schedule(&mut self, workload: &Workload) -> ScheduleResult {
        match self.auto_detect_mode(&workload) {
            Gaming => self.schedule_afr(workload),
            Compute => self.schedule_compute_split(workload),
            Rendering => self.schedule_sfr(workload),
        }
    }

    /// Auto-detect workload type from API calls
    fn auto_detect_mode(&self, workload: &Workload) -> WorkloadType {
        if workload.has_swapchain_present() {
            WorkloadType::Gaming
        } else if workload.has_compute_dispatch() {
            WorkloadType::Compute
        } else if workload.has_draw_calls() && workload.has_render_pass() {
            WorkloadType::Rendering
        } else {
            WorkloadType::Compute // fallback
        }
    }
}
```

### 4. Memory Management

#### 4.1 Memory Model

```
Application Virtual Address Space
┌────────────────────────────────────────────┐
│ mvgal_allocate_buffer(1GB, DISTRIBUTED)    │
│                                            │
│  ┌─────────────┐  ┌─────────────┐          │
│  │ Pages 0-512 │  │ Pages 513-1023│         │
│  │ on GPU 0    │  │ on GPU 1     │         │
│  │ (NVIDIA)    │  │ (AMD)        │         │
│  └─────────────┘  └─────────────┘          │
│                                            │
│ MVGAL Unified Virtual Address Space        │
│ (Each GPU has a page table mapping         │
│  to local or remote GPU pages)             │
└────────────────────────────────────────────┘
```

#### 4.2 Coherence Protocol

For the `DISTRIBUTED` policy:
- Each page has a **home GPU** (where physical memory resides).
- A **directory** tracks which GPUs have read-only copies.
- On write: invalidate remote copies (lazy: on next read from remote).
- Reads to remote pages: DMA transfer on demand (with prefetch hint API).

For the `MIRRORED` policy:
- All GPUs allocate the same buffer locally.
- Write to any GPU → DMA broadcast to all other GPUs.
- Reads are always local (zero latency).
- Synchronization point: all GPUs must complete write before any reads.

```rust
// memory.rs
pub struct MemoryManager {
    allocations: HashMap<BufferId, BufferAllocation>,
    page_directory: PageDirectory,
    dma_engine: DmaEngine,
}

pub struct BufferAllocation {
    id: BufferId,
    size: u64,
    policy: MemoryPolicy,
    // Per-GPU allocations
    local_allocations: Vec<(u32, DeviceLocalAlloc)>,
    // Page table for distributed mode
    page_table: Option<PageTable>,
}

pub enum MemoryPolicy {
    /// All GPUs have identical copy; writes broadcast
    Mirrored {
        pending_writes: AtomicU32, // Track outstanding DMA broadcasts
    },
    /// Pages partitioned; coherence via directory
    Distributed {
        page_size: u64,
        directory: Directory,
    },
    /// User manages explicitly
    UserManaged,
}
```

#### 4.3 Zero-Copy Paths

When two GPUs share a PCIe root complex:
- Use `dma_buf` export/import to share buffer handles.
- GPU A writes → GPU B reads directly via PCIe peer-to-peer (if supported).
- Fallback: staged copy through system RAM.

```c
// Kernel-level DMA buf sharing (mvgal_dma.c)
int mvgal_export_dma_buf(struct mvgal_device *src_dev, u64 src_handle,
                          int *out_dma_fd) {
    struct dma_buf *dmabuf = /* get from vendor driver via dma_buf ops */;
    *out_dma_fd = dma_buf_fd(dmabuf, O_CLOEXEC);
    return 0;
}

int mvgal_import_dma_buf(struct mvgal_device *dst_dev, int dma_fd,
                          u64 *out_handle) {
    struct dma_buf *dmabuf = dma_buf_get(dma_fd);
    // Map into destination GPU's IOVA space
    *out_handle = /* vendor-specific import */;
    dma_buf_put(dmabuf);
    return 0;
}
```

### 5. Power Management (`mvgal-powertune`)

#### 5.1 Architecture

```
mvgal-powertune daemon
├── Monitor thread          ← reads /sys/kernel/mvgal/devices/gpu*/metrics every 100ms
├── Policy engine           ← evaluates rules & makes decisions
│   ├── Idle detector       ← if GPU < 5% utilization for 5s → D3cold
│   ├── PSU budget          ← sum power ≤ global_power_cap
│   ├── Thermal throttle    ← if temp > threshold → reduce load
│   └── User override       ← D-Bus API for manual control
└── Actuator thread         → writes power caps, triggers sleep states
```

#### 5.2 Idle State Machine

```
                    ┌──────────┐
                    │  Active  │
                    └────┬─────┘
                         │ utilization < 5% for 5s
                         ▼
                    ┌──────────┐
                    │  Idle    │──→ GPU P-state low (P8)
                    └────┬─────┘
                         │ idle > 30s
                         ▼
                    ┌──────────┐
                    │  D3cold  │──→ PCIe D3cold (power removed)
                    └────┬─────┘
                         │ workload arrives
                         ▼
                    ┌──────────┐
                    │  Resume  │──→ P-state high (P0)
                    └────┬─────┘
                         │
                         ▼
                    ┌──────────┐
                    │  Active  │
                    └──────────┘
```

#### 5.3 PSU Budgeting Algorithm

```
Given: GPUs [G0, G1, G2], global_power_cap = 650W
       Current draw: [250W, 200W, 150W] = 600W total

When new workload arrives:
  1. Estimate required power for each GPU based on workload type
  2. If sum ≤ global_power_cap: proceed
  3. If sum > global_power_cap:
     a. Reduce power caps proportionally:
        G0: 250W → 225W  (-10%)
        G1: 200W → 180W  (-10%)
        G2: 150W → 135W  (-10%)
     b. If still over cap: disable least-capable GPU
        (based on compute ratio or user preference)
  4. Re-evaluate every 500ms
```

### 6. Windows Compatibility Layer

#### 6.1 Strategy: LD_PRELOAD + Vulkan Layer

Three interception points work together:

```
Windows Game (via Proton)
    │
    ├── DirectX 12 → VKD3D → Vulkan
    ├── DirectX 11 → DXVK  → Vulkan
    └── DirectX 9  → D9VK  → Vulkan
                              │
                              ▼
                    VK_LAYER_MVGAL (Vulkan layer)
                    Intercepts: vkCreateDevice, vkQueueSubmit,
                    vkAllocateMemory, vkGetSwapchainImages
                              │
                              ▼
                         libmvgal.so
                    Distributes across GPUs
```

#### 6.2 NTSYNC Implementation

NTSYNC (NT synchronization primitives) improves Windows application performance:

```c
// kernel-side NTSYNC via futex (mainline since 5.16)
// Wine uses futex(2) with FUTEX_OWNER_NT | FUTEX_WAIT_NT

// Userspace: Wine's ntdll uses these syscalls
int NtCreateMutant(PHANDLE Handle, ACCESS_MASK DesiredAccess,
                   OBJECT_ATTRIBUTES *Attr, BOOLEAN InitialOwner);
int NtReleaseMutant(HANDLE Handle, PLONG PreviousCount);
int NtCreateSemaphore(PHANDLE Handle, ACCESS_MASK DesiredAccess,
                      OBJECT_ATTRIBUTES *Attr, LONG InitialCount, LONG MaximumCount);

// MVGAL provides: futex-based NTSYNC via kernel patch or the upstream
// real-time mutex + NTSYNC patches that landed in kernel 6.x
```

MVGAL provides:
- A kernel patch (or kernel parameter) enabling NTSYNC futex ops.
- The `wine/ntsync.c` integration shim that detects and uses NTSYNC.
- Fallback to Esync/Fsync if NTSYNC unavailable.

#### 6.3 WoW64 Support

WoW64 (Windows 32-bit on Windows 64-bit) allows 32-bit Windows executables:

- MVGAL's Wine integration includes the standard WoW64 thunking layer.
- 32-bit game → 32-bit Wine loader → thunk to 64-bit → Vulkan → VK_LAYER_MVGAL.
- No MVGAL-specific changes needed — standard Wine WoW64 handles this.

### 7. Build System & CI/CD

#### 7.1 Top-Level Makefile

```makefile
.PHONY: all kernel lib backends tools wine power test clean install deb rpm

all: kernel lib

# Kernel module — kbuild
kernel:
	$(MAKE) -C /lib/modules/$(shell uname -r)/build M=$(PWD)/kernel modules

# Rust userspace library — Cargo
lib:
	cargo build --manifest-path libmvgal/Cargo.toml $(RELEASE_FLAG)

# C++ backends — CMake
backends:
	cmake -S backends -B build/backends $(CMAKE_FLAGS)
	cmake --build build/backends

# Tools — Cargo
tools:
	cargo build --manifest-path tools/mvgal-info/Cargo.toml $(RELEASE_FLAG)
	cargo build --manifest-path tools/mvgal-top/Cargo.toml $(RELEASE_FLAG)
	cargo build --manifest-path tools/mvgal-config/Cargo.toml $(RELEASE_FLAG)

# Power daemon — Cargo
power:
	cargo build --manifest-path power/Cargo.toml $(RELEASE_FLAG)

# Packaging
deb: all
	dpkg-buildpackage -b -uc

rpm: all
	rpmbuild -bb packaging/rpm/mvgal.spec

# Install
install: all
	install -m 644 kernel/mvgal.ko /lib/modules/$(shell uname -r)/extra/
	install -m 755 target/release/libmvgal.so /usr/lib/
	install -m 755 target/release/mvgal-powertune /usr/sbin/
	install -m 755 target/release/mvgal-info /usr/bin/
	install -m 755 target/release/mvgal-top /usr/bin/
	depmod -a
```

#### 7.2 CI/CD Pipeline (GitHub Actions)

```yaml
# .github/workflows/ci.yml
jobs:
  build:
    strategy:
      matrix:
        distro: [ubuntu-22.04, ubuntu-24.04, fedora-40, archlinux]
        compiler: [gcc, clang]
    steps:
      - uses: actions/checkout@v4
      - name: Build kernel module
        run: make kernel
      - name: Build userspace
        run: make lib
      - name: Run tests
        run: make test
      - name: Package
        run: make ${{ matrix.distro == 'ubuntu-*' && 'deb' || 'rpm' }}
```

### 8. Testing Strategy

#### 8.1 Unit Tests

| Component | Framework | What's Tested |
|-----------|-----------|---------------|
| Kernel ioctl | kunit (kernel) | ioctl dispatch, device enumeration, DMA buf lifecycle |
| libmvgal core | Rust `#[test]` | ABI correctness, memory policy enforcement, context lifecycle |
| Scheduler | Rust `#[test]` | AFR frame distribution, compute ratio splitting, load rebalancing |
| Memory manager | Rust `#[test]` | Mirror write broadcast, page migration, directory coherence |
| Power daemon | Rust `#[test]` | State machine transitions, PSU budget calculation |

#### 8.2 Integration Tests

| Test | Setup | What It Validates |
|------|-------|-------------------|
| Device enumeration | 3 GPUs (NVIDIA + AMD + Intel) | All detected, correct capabilities |
| AFR gaming | glxgears + mvgal | Frames distributed, no tearing |
| Compute split | Matrix multiply × 2 GPUs | Correct results, speedup > 1.5× |
| Memory mirror | Allocate 1GB mirror, write, read back | Data consistent across GPUs |
| Memory distribute | Allocate 1GB distributed, scatter write | Pages on correct GPUs |
| Power capping | Heavy load on 2 GPUs, cap at 300W | Total ≤ 300W |
| Wine compatibility | DirectX 11 game via Proton + VK_LAYER_MVGAL | Game runs, uses both GPUs |

#### 8.3 Performance Benchmarks

| Benchmark | Measures |
|-----------|----------|
| Single GPU baseline | FPS, render time, compute throughput |
| MVGAL 2-GPU AFR vs single | FPS scaling, frame time variance |
| MVGAL 3-GPU compute vs single | Throughput scaling, PCIe overhead |
| MVGAL memory mirror vs local | Bandwidth penalty for writes |
| MVGAL memory distribute vs local | Latency for remote page access |
| Idle power savings | Power with all GPUs vs selective D3cold |
| PSU capping overhead | Impact on throughput when power-limited |

---

## Implementation Phases

### Phase 1 (Months 1-3): Foundation
1. Kernel module: device detection, `/dev/mvgal`, basic ioctl interface.
2. `libmvgal.so` core: device enumeration, context management, simple memory allocation.
3. `mvgal-info` tool: display GPU topology.
4. Build system: Makefile, kbuild, Cargo, CMake scaffolding.
5. CI/CD: GitHub Actions for build verification.

### Phase 2 (Months 4-6): Scheduler & Memory
1. Workload scheduler: AFR mode, compute splitting mode.
2. Memory manager: mirrored policy, DMA buf export/import.
3. Vulkan backend: VK_LAYER_MVGAL intercepting physical device enumeration.
4. Power daemon: monitoring, idle detection, D3cold state management.

### Phase 3 (Months 7-9): Backend Translators
1. CUDA LD_PRELOAD wrapper.
2. OpenGL interposer.
3. OpenCL wrapper.
4. HIP wrapper.
5. Memory manager: distributed policy, page migration, directory coherence.

### Phase 4 (Months 10-12): Gaming & Professional
1. Wine/Proton integration testing.
2. NTSYNC kernel patches.
3. Gaming optimization (latency, frame pacing).
4. Blender/Maya/Unreal Engine testing.
5. PyTorch/TensorFlow distributed training support.

### Phase 5 (Months 13+): Polish & Scale
1. More GPU vendor support (Qualcomm, ARM Mali).
2. AI-driven workload prediction.
3. Cloud GPU aggregation.
4. Formal verification of critical kernel paths.
5. Distribution packaging for all major distros.

---

## Risks & Mitigations

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Vendor driver incompatibility with kernel updates | High | High | Abstract via shim layer; CI tests against multiple kernels |
| PCIe peer-to-peer not supported between vendors | High | Medium | Fallback to system RAM staging; explicit in documentation |
| Games detect multiple GPUs and break | Medium | High | Vulkan layer hides multi-GPU from application; presents single device |
| Performance worse than single GPU for some workloads | Medium | Medium | Scheduler auto-detects and falls back to single GPU; power off extra GPUs |
| Kernel module fails to load with signed boot (Secure Boot) | Medium | Medium | Provide signing guide; submit to distro kernel trees |
| Memory coherence overhead kills performance | Medium | High | Profile-guided memory policy; let apps choose explicit API |
| Wine/Proton breakage with each update | High | Medium | No Proton fork needed; Vulkan layer is API-stable |

---

## References

### Kernel & Driver Repositories
- NVIDIA Open GPU Kernel Modules: https://github.com/nvidia/open-gpu-kernel-modules
- AMD AMDVLK: https://github.com/GPUOpen-Drivers/AMDVLK
- Moore Threads mtgpu-drv: https://github.com/dixyes/mtgpu-drv
- Intel media-driver: https://github.com/intel/media-driver

### Multi-GPU Technologies
- NVIDIA NVLink: https://www.nvidia.com/en-us/data-center/nvlink/
- CUDA + NCCL: https://developer.nvidia.com/nccl
- MPI Forum: https://www.mpi-forum.org/
- PyTorch Distributed: https://pytorch.org/docs/stable/distributed.html

### Linux GPU Programming
- Vulkan Loader / ICD: https://github.com/KhronosGroup/Vulkan-Loader
- DXVK: https://github.com/doitsujin/dxvk
- VKD3D-Proton: https://github.com/HansKristian-Work/vkd3d-proton
- Wine/Proton: https://github.com/ValveSoftware/Proton
- Linux kernel GPU documentation: Documentation/gpu/

### Synchronization
- Esync: https://github.com/zfigura/wine/tree/esync
- Fsync: https://github.com/zfigura/wine/tree/fsync
- NTSYNC Linux kernel patches: https://lore.kernel.org/lkml/
