**MVGAL — Task specification (task.md / task.txt)** 
 
 Project name: MVGAL (Multi-Vendor GPU Aggregation Layer for Linux) 
 
 **Overview** 
 - Build a cross-distribution Linux system layer that aggregates and unifies multiple discrete GPUs from different vendors (AMD, NVIDIA, Intel, Moore Threads) so an application or game can use them as a single logical accelerator or as a tightly coordinated multi-device pool. 
 - Goals: vendor-agnostic multi-GPU scheduling and memory movement, driver/userland integrations, rendering/workload orchestration, idle/power optimization, Steam/Proton compatibility, and Windows ABI shims for better Windows-game support (Esync/Fsync-like, WoW64 support, NTSYNC concept). 
 
 **Primary deliverables** 
 1. task.md / task.txt (this file) — spec and actionable checklist for an AI/dev agent. 
 2. Design docs: 
    - Architecture overview (kernel + userland components) 
    - Initialization & device discovery flow 
    - Rendering / compute / workload orchestration flow 
    - Memory & data movement flow, including P2P, staging, and RDMA fallbacks 
    - Power / idle management and per-GPU optimization 
    - Compatibility/adaptation layers for Steam/Proton and Windows ABI shims 
 3. Prototype code: 
    - Kernel module(s) or kernel glue to coordinate vendor kernel modules (where required) 
    - Userland daemon/service (systemd-friendly, polkit/pkexec for privileged actions) 
    - Runtime libraries / APIs for applications (C/C++ and bindings) 
    - Proof-of-concept dispatcher that runs OpenGL/Vulkan/OpenCL/CUDA workloads across mixed GPUs 
 4. Tests and benchmarks: 
    - Unit tests, integration tests, and multi-GPU microbenchmarks (bandwidth, latency, synchronization) 
    - Game and rendering tests via Steam/Proton and a small set of real-world applications (Blender, simple OpenCL/CUDA kernels) 
 5. Packaging and distro integration: 
    - Packages for Debian/Ubuntu, Fedora, Arch (PKGBUILD), and generic tarball/flatpak/snap as appropriate 
 6. Documentation and a contributor guide 
 
 **Assumptions & constraints (decisive choices)** 
 - Target kernel: upstream Linux LTS kernels (support fallback to slightly older kernels). Implement kernel components as out-of-tree modules where required, aim to upstream later. 
 - Userland: systemd-based systems are primary target but support non-systemd via init scripts. 
 - Privileges: use polkit + pkexec for any privileged operations (no direct sudo). 
 - Compatibility: must interoperate with vendor kernel drivers (do not replace vendor drivers; coordinate them). 
 - Prioritize open-source vendor drivers (NVIDIA open-gpu-kernel-modules, AMDVLK/AMDGPU, Intel drivers, MTT driver repo referenced). 
 - Provide fallbacks when P2P/GPU Direct features are absent (host-staging buffers, DMA-BUF). 
 
 **References to inspect (start points; agent must read these repos/docs)** 
 - NVIDIA open-gpu-kernel-modules: `https://github.com/nvidia/open-gpu-kernel-modules` 
 - AMDVLK / AMD GPUOpen drivers: `https://github.com/GPUOpen-Drivers/AMDVLK`  and AMDGPU/ROCm projects 
 - Moore Threads driver (mtgpu-drv): `https://github.com/dixyes/mtgpu-drv` 
 - Intel media/driver: `https://github.com/intel/media-driver` 
 - NVLink, CUDA, NCCL (NVIDIA), RCCL (AMD), UCC/UCX (open collectives), GPUDirect RDMA 
 - Relevant research and tools: NCCL, Horovod, MPI + CUDA workflows, NVML, DRM/KMS, dma-buf, VK_EXT_QUEUE_FAMILY_FOREIGN, Vulkan multi-device extensions, OpenCL ICDs, SYCL 
 - Steam/Proton, Wine forks (Esync/Fsync, WoW64 handling). Investigate NTSYNC idea for NT syscall handling and Esync/Fsync performance patches. 
 
 **High-level architecture (components)** 
 - Kernel-side: 
   - Aggregation kernel module (mvgal-kmod): coordinates device enumeration, exposes unified device nodes (character device /dev/mvgalX and DRM interfaces), and mediates privileged operations (memory pinning, cross-device mapping, dma-buf registration). 
   - Kernel helpers: small hooks to vendor modules (query P2P support, peer mappings, PCI topology, NVLink links, BAR mapping, IOVAs). 
   - Use existing kernel mechanisms: DRM, VFIO, dma-buf, IOMMU, GPUDirect interfaces (where exposed). 
 - Userland: 
   - mvgald (daemon): device manager, policy engine, scheduler, exporter of metadata via D-Bus and Unix sockets, privileged operations via pkexec. 
   - Runtime library: libmvgal (C API) and C++ wrapper providing device listing, memory allocation semantics, cross-device memory copy, synchronization primitives, and workload submission helpers for Vulkan/OpenCL/CUDA/SYCL backends. 
   - Vendor adapters: small plugins per-vendor to translate vendor-specific features into the common MVGAL API (e.g., CUDA -> libmvgal bridge, ROCm bridge). 
   - Application shim: LD_PRELOAD-based helper for intercepting common APIs (OpenGL, Vulkan loader, OpenCL, CUDA runtime) to transparently present an aggregated device or to route work to libmvgal-aware apps. 
   - Proton/Wine integration layer: optional plugin to enable Esync/Fsync-like and WoW64 improvements (NTSYNC concept). Provide patches and userspace patches as a Proton module. 
 - CLI / UI: 
   - mvgalctl: management tool for profiles, power curves, aggregation policies. 
   - Optional GNOME/KDE widget for monitoring. 
 
 **Device discovery & initialization flow** 
 1. Kernel boots, vendor kernel modules register GPUs via DRM / device platform. 
 2. mvgal-kmod detects supported GPUs (AMD, NVIDIA, Intel, MTT), probes PCI topology for peer links (NVLink, PCIe switches). 
 3. Kernel exposes unified device metadata (capabilities, memory sizes, PCI topology, P2P capability) via sysfs and via /dev/mvgalX. 
 4. mvgald queries kernel, applies user/system policies (e.g., aggregate all GPUs by default or only when application requests). 
 5. For aggregation, mvgald: 
    - Negotiates with vendor drivers for memory pinning and dma-buf exports. 
    - Establishes pinned shared buffers or maps remote memory where P2P is supported (GPUDirect), or establishes high-performance host-staged DMA paths otherwise. 
 6. Export a logical device handle to applications via libmvgal or via API shims. 
 
 **Rendering / workload flow (typical patterns)** 
 - Two modes: 
   1. Logical-device mode: present an aggregated logical GPU to the application (best-effort). The runtime splits and schedules workloads across physical GPUs. 
   2. Cooperative multi-device mode: application is aware and uses libmvgal to explicitly distribute work and synchronize. 
 - Scheduling strategies: 
   - Round-robin kernel/command-queue splitting for trivial loads. 
   - Domain decomposition (frame/mesh tiling for rendering; batch/model parallelism for compute). 
   - Work stealing and dynamic load balancing with runtime profiling. 
 - Synchronization: 
   - Map vendor fences to mvgal fences; use dma-buf sync where possible. 
   - When GPUs lack P2P atomicity, use host-mediated synchronization and staged transfers. 
 - Communication: 
   - Prefer direct GPU-GPU peer access (NVLink, PCI P2P). 
   - Use CCLEX-like collectives: build or reuse open libraries (UCC/UCX) for multi-vendor collectives; fall back to host-staged copy if RDMA not available. 
   - For distributed or multi-node scenarios: integrate with NCCL/RCCL only when vendor-specific contexts are available, otherwise use UCC/UCX. 
 
 **Memory & data flow** 
 - Abstractions: 
   - Unified memory descriptors and capability flags per allocation (pinned, exportable, host-staged, remote-mapped). 
 - Fast paths: 
   - GPU P2P direct mappings when vendor drivers and topology permit (use dma-buf + mmap, or vendor-specific peer APIs). 
   - GPUDirect RDMA when NIC and GPUs support it for multi-node. 
 - Fallbacks: 
   - Pinned host buffers + async DMA transfers when direct peer access isn’t available. 
 - Consistency: 
   - Explicit ownership rules; support for cache-coherent accesses where hardware supports it; otherwise explicit invalidation and flush using vendor fences. 
 
 **Driver & vendor integration checklist** 
 - For each vendor (NVIDIA, AMD, Intel, MTT): 
   - Read vendor open-kernel modules and driver architecture. 
   - Identify exposure points: memory pinning, dma-buf export, fence signalling, P2P APIs, NVLink usage. 
   - Implement adapter plugin that: 
     - Queries capability and topology. 
     - Requests pinned export of buffers and handles mapping/unmapping. 
     - Converts vendor sync primitives to mvgal primitives. 
 - Specific repos to inspect and tasks: 
   - NVIDIA: open-gpu-kernel-modules — check kernel module hooks for buffer export and P2P capabilities. 
   - AMD: AMDVLK & AMDGPU/ROCm — check dma-buf export paths and ROCm userland. 
   - Intel: media-driver and GPU driver interfaces — check for VAAPI/Vulkan integration and exports. 
   - MTT: mtgpu-drv — review capabilities and missing hooks to map into MVGAL. 
 - Kernel compatibility: ensure mvgal-kmod monitors vendor driver load/unload and handles race conditions. 
 
 **Steam, Proton, Wine, and Windows-game support** 
 - Goal: Games launched via Steam/Proton should be able to benefit from MVGAL aggregation for performance or better utilization. 
 - Integration points: 
   - LD_PRELOAD shims for Vulkan/OpenGL to alter device enumeration or remap to libmvgal logical device. 
   - Proton patches: implement a module that adapts Esync/Fsync-like features for aggregated GPUs; implement NTSYNC concept for NT syscall management to reduce syscall overhead. 
   - WoW64: provide improved 32-bit on 64-bit handling in Proton/Wine by optimizing syscall paths and supporting 32-bit GPU userland (if needed). 
 - Testing: test with a sample set of Windows games, compare latency and frame-times with and without MVGAL. 
 
 **Power, idle optimization, and PSU power-curve control** 
 - Features: 
   - Per-GPU idle detection and deep-idle states; consolidate workload to fewer GPUs and put others to low-power states when feasible. 
   - Power-curves: expose control for clocks, fan, and power limits via vendor interfaces (nvidia-smi-like controls, AMD power7, Intel controls), coordinated through mvgalctl. 
   - Policies: aggressive consolidation for desktops and laptops; performance-first for gaming; rendering/distributed mode for compute clusters. 
 - Safety: 
   - Respect vendor interfaces and thermal/power limits; avoid user-space overdrives that bypass vendor safety unless explicitly allowed by admin. 
 
 **Languages, compilers, build systems, and tooling (recommended)** 
 - Core languages: 
   - Kernel: C (Linux kernel coding style). 
   - Userland daemon, libraries: C (API), C++ (runtime, wrappers). 
   - Language bindings: Rust for safety-critical components (optional), Python for tests/automation. 
 - GPU and compute APIs: 
   - Vulkan, OpenGL, OpenCL, CUDA (where applicable), ROCm/HIP, SYCL. 
 - Compilers & toolchain: 
   - GCC and Clang/LLVM for C/C++ code; support cross-compilation where needed. 
   - Build systems: CMake primary, with optional Meson/Ninja for faster builds. Use Bazel for large multi-language CI where helpful. 
   - Packaging/build helpers: Make, Ninja, Conan or Hunter for C++ dependencies. 
   - Optional compilers: Intel oneAPI for vendor-specific optimizations for CPU+GPU (when targeting Intel accelerators). 
 - Interop languages: 
   - Rust for runtime modules or safer bindings; use bindgen for C interop. 
   - Python for scripting, tests, and orchestration tools. 
 - CI: 
   - Use GitHub Actions, GitLab CI, and/or self-hosted runners with different GPU vendors to run integration tests. 
 - Suggested libraries: 
   - UCX/UCC for collective communications 
   - libdrm for DRM interactions 
   - dma-buf helpers 
   - Vulkan loader and ICDs 
   - vendor SDKs (only where licensing allows) 
 
 **Implementation plan — phased, actionable** 
 Phase 0 — Research & repo ingestion (2–4 weeks) 
 - Clone and read the referenced vendor repos and kernel docs. 
 - Produce a mapping matrix of which vendor exposes which kernel/userland hooks (dma-buf, fence, mmap, P2P). 
 - Identify missing kernel hooks and list kernel patches needed. 
 
 Phase 1 — Minimal viable prototype (6–10 weeks) 
 - Implement mvgal-kmod with read-only device discovery and reporting via sysfs and /dev/mvgal0. 
 - Implement mvgald that reads device metadata and can export a simple logical device handle. 
 - Implement libmvgal with APIs for listing devices, allocating an exportable buffer, and performing cross-device copy via host-staging. 
 - Test: run a simple OpenCL or Vulkan compute kernel split across two heterogeneous GPUs using host-staged transfers. 
 
 Phase 2 — High-performance paths & vendor adapters (10–16 weeks) 
 - Implement vendor adapters for dma-buf export and P2P mapping for each vendor discovered. 
 - Implement fence conversion and sync primitives. 
 - Implement direct GPU-GPU copy paths where vendor P2P allows. 
 - Integrate UCX/UCC for collectives on mixed GPUs. 
 
 Phase 3 — Rendering/game integration & Proton patches (8–12 weeks) 
 - Build LD_PRELOAD shims for Vulkan/OpenGL device enumeration; prototype with simple Vulkan app and with Steam/Proton. 
 - Implement Esync/Fsync-like improvements and NTSYNC ideas into a Proton module (work with Proton/Wine maintainers if possible). 
 - Test with sample games and measure FPS/latency. 
 
 Phase 4 — Power management, packaging, and distro support (6–10 weeks) 
 - Implement mvgalctl power/profile manager; integrate with vendor tools (nvidia-smi, AMD controls). 
 - Create distro packages, systemd units, and documentation. 
 - Run extended tests across distributions. 
 
 Phase 5 — Hardening, security review, and upstreaming (ongoing) 
 - Prepare patches for upstream kernel (if needed), vendor driver contributions, and library packaging. 
 - Security and stability audits; address race conditions and privilege separation using polkit/pkexec. 
 
 **Security, permissions, and privileged operations** 
 - Use polkit and pkexec for operations that require elevated privileges (no sudo hardcoding). 
 - Keep kernel module attack surface small; limit ioctls and validate inputs. 
 - Sign kernel modules where distro policies require it; document module signing steps. 
 
 **Testing & validation matrix** 
 - Hardware combos: single GPU, 2x mixed (AMD+NVIDIA), 3x mixed (AMD+NVIDIA+MTT), Intel integrated + discrete combos, NVLink present/absent. 
 - OSes: Ubuntu LTS, Fedora, Arch, plus one RHEL/CentOS (for enterprise). 
 - Workloads: 
   - Gaming: selected Steam titles, measure FPS and frame-time variance. 
   - Rendering: Blender render across multiple GPUs. 
   - ML: PyTorch distributed training with mixed GPUs (use UCC/UCX or NCCL/RCCL where applicable). 
   - HPC: MPI + CUDA microbenchmarks. 
 - Metrics: utilization, bandwidth, latency, power consumption, thermal, frame-time P95/P99. 
 
 **Risk analysis & mitigation** 
 - Vendor lock-in & proprietary APIs: mitigate via adapters and open-standards (Vulkan, OpenCL, UCX). 
 - Performance overhead for heterogeneous combos: provide policy-based consolidation and fallbacks. 
 - Kernel stability and upstream resistance: minimize invasive kernel changes; prefer userland solutions and standard kernel primitives (dma-buf, VFIO). 
 - Legal/licensing: ensure license compatibility before using proprietary SDKs; provide optional vendor-specific adapters. 
 
 **Implementation notes and developer instructions** 
 - Always inspect and respect vendor repo licenses and contribution rules. 
 - Prefer non-invasive userland-first designs; use kernel modules only when essential (dma-buf glue or privileged memory pinning). 
 - For multi-node RDMA and GPUDirect support, ensure NICdrivers/hosts allow GPU memory registration (nv_peer_mem equivalents). 
 - When implementing privileged CLI actions, call pkexec to elevate the mvgald helper for those specific tasks. 
 - Document all ABI and API surfaces. Provide stable C ABI for libmvgal. 
 
 **Task assignment checklist (what the AI agent must do)** 
 - Clone and parse the provided repos and docs; produce a capability matrix per vendor. 
 - Produce a small architecture diagram and an implementation roadmap (milestones with time estimates). 
 - Implement Phase 1 prototype (deliver code, build instructions, runbook). 
 - Implement sample app demonstrating aggregated compute across at least two vendors using host-staged fallback. 
 - Produce tests and benchmark scripts and instructions to run them. 
 - Produce packaging for one distro (Ubuntu) and systemd units. 
 - Create contributor guide and coding standards. 
 - Create CI config for multi-runner integration (include GPU tagging). 
 - Provide a security review summary and list of required kernel patches (if any) with minimal diffs. 
 
 **Files to produce** 
 - README.md (project intro, build, run, test) 
 - ARCHITECTURE.md (detailed components + diagrams) 
 - DESIGN.md (initial design with tradeoffs) 
 - IMPLEMENTATION_PLAN.md (milestones, owners, timelines) 
 - mvgal-kmod/ (kernel code) 
 - mvgald/ (userland daemon) 
 - libmvgal/ (C API + C++ wrapper) 
 - plugins/ (vendor adapters) 
 - shims/ (LD_PRELOAD Vulkan/OpenGL shims) 
 - proton-module/ (Proton/Wine patches & instructions) 
 - tests/benchmarks/ (scripts + dataset) 
 - packaging/ (deb, rpm, PKGBUILD) 
 - task.md / task.txt (this deliverable) 
 
 **Acceptance criteria** 
 - Prototype aggregates at least two different-vendor GPUs and runs a split compute workload with measurable improvements or correct split execution (functional correctness required). 
 - libmvgal provides documented APIs and at least one high-level sample showing cross-vendor buffer allocation, copy, and synchronization. 
 - mvgald and mvgal-kmod run on Ubuntu and at least one other distro. 
 - Power/profile controls available via mvgalctl that can consolidate workloads and reduce idle GPU power draw. 
 - Proton module demonstrates improved performance for at least one tested Windows title when aggregation is enabled (or at least no regression and proper device mapping). 
 
 **Notes for the AI/dev agent** 
 - Read all referenced repositories fully (kernel and userland) before coding to identify missing hooks. 
 - When privileged actions are required in user flows, call pkexec for elevation. 
 - Preserve user safety: don’t implement vendor-overriding firmware flashes or unsafe overclocks without explicit admin opt-in. 
 - Be decisive: implement host-staged fallback if any direct P2P or RDMA path is missing. 
 - Provide clear, minimal reproducer cases for each capability. 
 
 **End of task.md / task.txt** 
