# MVGAL Bottlenecks and Optimization Opportunities

**Version:** 1.0  
**Date:** 2026-06-02  

---

## 1. Performance Bottlenecks

### 1.1 Kernel Module Bottlenecks

| Component | Bottleneck | Current Impact | Optimization Opportunity |
|-----------|------------|----------------|----------------------|
| `pci_get_device` scan loop | O(N) PCI scan on every rescan | ~1ms overhead per call | Cache PCI results; use `pci_register_driver` observer pattern |
| Sysfs attribute reads | String parsing overhead | Latency in GPU property queries | Add sysfs cache with TTL; batch reads |
| Sysfs writes (power state) | Acquires global lock | Serialized power changes | Use per-GPU lock; async write |

### 1.2 Memory Manager Bottlenecks

| Component | Bottleneck | Current Impact | Optimization Opportunity |
|-----------|------------|----------------|----------------------|
| Single global mutex | `m_mutex` protects all allocations | Serialized alloc/free | Use sharded locks per allocation range |
| Fake GPU addresses | All addresses `0x1000 + id` | No real GPU mapping | Implement real vendor allocation |
| Missing P2P detection | Always returns `true` | Unnecessary staging copies | Implement `pci_p2pdma_distance()` check |
| NUMA approximation | Uses node index as GPU index | Wrong GPU selection | Read actual GPU NUMA node from sysfs |

### 1.3 Scheduler Bottlenecks

| Component | Bottleneck | Current Impact | Optimization Opportunity |
|-----------|------------|---------------- |------------------------|
| Global workload mutex | `m_workloadMutex` for all workloads | Thread contention under load | Use lock-free queue; per-Priority mutexes |
| Profile mutex | `m_profileMutex` for profiles | Serialized profile lookups | Read-copy-update pattern |
| Work stealing disabled | `stealWork()` returns false | No load balancing | Implement cross-GPU queue migration |
| AI model reload | Model loaded on every `scheduleAI()` | High latency per decision | Load model once in `init()` |

### 1.4 Metrics Collector Bottlenecks

| Component | Bottleneck | Current Impact | Optimization Opportunity |
|-----------|------------|----------------|----------------------|
| 1-second polling interval | Fixed 1000ms interval | Stale metrics | Configurable interval; event-driven |
| JSON generation | String stream per call | CPU overhead | Pre-compute JSON; atomic update |
| Feature extraction | Called per `collectGpuMetrics()` | Repeated computation | Cache features; incremental update |

---

## 2. Scalability Concerns

### 2.1 GPU Scaling

| Limit | Current Value | Issue | Solution |
|-------|---------------|-------|----------|
| `MVGAL_UAPI_MAX_GPUS` | 32 | Arbitrary limit | Extend to 64; use dynamic allocation |
| Priority levels | 16 | May be insufficient for complex workloads | Extend to 32; document levels |
| Queue depth tracking | Missing | No congestion detection | Add per-GPU semaphore depth |

### 2.2 Memory Scaling

| Concern | Current Status | Impact | Solution |
|---------|---------------|--------|----------|
| Large allocations (> VRAM) | No fallback handling | OOM crashes | Add swap tier; host-RAM overflow |
| Buffer fragmentation | Not tracked | Inefficient allocation | Add buddy allocator; defragment |
| DMA-BUF reference cycles | Not detected | Leaks | Add refcounting; cycle detection |

---

## 3. Optimization Opportunities

### 3.1 Memory Allocation Optimizations

#### 3.1.1 Sharded Locking

```cpp
// Current: single mutex
mutable std::mutex m_mutex;
std::unordered_map<uint64_t, MemoryAllocation> m_allocations;

// Proposed: sharded map
static constexpr size_t NUM_SHARDS = 16;
struct Shard {
    mutable std::mutex lock;
    std::unordered_map<uint64_t, MemoryAllocation> allocations;
};
std::array<Shard, NUM_SHARDS> m_shards;

// Hash-based sharding
Shard& getShard(uint64_t id) {
    return m_shards[id % NUM_SHARDS];
}
```

**Expected improvement**: ~4x throughput under concurrent allocation

#### 3.1.2 Lazy UVA Mapping

```cpp
// Current: mmap on every allocate()
void* uvaPtr = mmap(nullptr, size, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

// Proposed: reserve range, map on first access
// Use guard pages for overrun detection
// Overcommit-safe with mremap() fallback
```

**Expected improvement**: Reduced syscall overhead; better fault handling

### 3.2 Scheduler Optimizations

#### 3.2.1 Lock-Free Workload Queue

```cpp
// Current: mutex-protected queue
std::vector<std::queue<std::shared_ptr<Workload>>> m_queueByPriority;

// Proposed: use folly::MPMCQueue or seastar::concurrent_queue
// Or: single-producer single-consumer queues per priority
```

**Expected improvement**: Reduced contention for high-frequency workloads

#### 3.2.2 Predictive Prefetching Integration

```cpp
// Integration point: sampleMetric -> update access pattern
// Feed into allocateOnGpu decision
// Pre-warm target GPU before workload arrives
```

**Expected improvement**: Reduced migration latency; better hit rate

### 3.3 Vulkan Layer Optimizations

#### 3.3.1 Batch AFR Submission

```c
// Current: allocate new structs per submit
VkSubmitInfo *new_submits = malloc(submitCount * sizeof(VkSubmitInfo));

// Proposed: pre-allocated ring buffer
// Thread-local storage for common submit patterns
// Avoid malloc in hot path
```

**Expected improvement**: Reduced allocation overhead; better cache locality

#### 3.3.2 SPIR-V Cache Optimization

```cpp
// Current: 256-slot open-addressing
// Proposed: LRU cache with larger capacity
// Use robin-hood hashing for better probe count
// Shared read-only across processes
```

**Expected improvement**: Better shader hit rate; reduced compilation

---

## 4. Cross-Vendor Performance Considerations

### 4.1 Transfer Path Selection

Current heuristic uses NUMA node equality:

```c
// From mvgal_main.c line 579
if (mvdev->gpus[i].info.numa_node == mvdev->gpus[j].info.numa_node) {
    caps->p2p_matrix[idx] = MVGAL_UAPI_P2P_P2P;
}
```

**Better approach**:
1. Check PCIe root complex via ACPI
2. Verify driver P2P capability via `dma_buf_dynamic`
3. Measure actual bandwidth with test transfer
4. Cache result with timestamp

### 4.2 Memory Placement Policy

Current policy (from MEMORY.md):

| Size | Placement |
|------|-----------|
| < 64MB | GPU with most free VRAM |
| Render target | GPU that writes first |
| > 64MB | GPU with access pattern history |

**Enhancement**:
- Track inter-GPU access frequency
- Correlate with transfer cost matrix
- Use ML model for placement prediction

---

## 5. CPU Utilization Analysis

### 5.1 Thread Pool Usage

The scheduler uses a single processing thread in `processPending()`. Under high load:

- **Issue**: Single-threaded processing
- **Solution**: Thread pool with N worker threads (N = GPU count)

### 5.2 Metrics Collection Overhead

Current poll interval: 1000ms (configurable)

- **Issue**: Fixed polling wastes CPU
- **Solution**: Adaptive polling based on workload; event-driven updates

### 5.3 IPC Overhead

Unix socket with mutex-protected message processing:

- **Issue**: Serialized message handling
- **Solution**: Use `io_uring` for async I/O; lock-free message queue

---

## 6. Memory Bandwidth Analysis

### 6.1 Host-RAM Staging Bottleneck

When P2P unavailable (NVIDIA + AMD baseline):

```
GPU0 → Host (read) → GPU1 (write)
       ↑
   PCIe BW × 2
```

**Mitigation**:
- Batch transfers to amortize PCIe overhead
- Use huge pages (2MB/1GB) for staging
- Compress transfer if beneficial

### 6.2 DMA-BUF Synchronization Cost

Each DMA-BUF operation requires:

1. Kernel `dma_buf_export()` syscall
2. FD passing to other process
3. Kernel `dma_buf_import()` syscall
4. Fence synchronization

**Optimization**:
- Export once, import multiple times
- Use sync timeline semaphores (reusable)
- Cache FD translations

---

## 7. I/O Bottlenecks

### 7.1 Sysfs Reading

Each metric read requires `open()` + `read()` syscall pair.

**Optimization**:
- Use `inotify` to watch for changes
- Memory-map sysfs files where possible
- Batch reads in single `read()` call

### 7.2 Vulkan Loader Dispatch

Current layer uses chain pattern:

```c
next_create_instance = (PFN_vkCreateInstance)next_gipa(VK_NULL_HANDLE, "vkCreateInstance");
```

**Optimization**:
- Cache function pointers per device
- Use static dispatch for hot functions
- Pre-resolve all ICD functions at layer init

---

## 8. Benchmark Targets

### 8.1 Scheduling Latency

| Operation | Target | Current |
|-----------|--------|---------|
| Workload submit | < 10 µs | ~1.81 µs (reported) |
| Dispatch decision | < 5 µs | ~variable |
| IPC round-trip | < 200 µs | Not measured |

### 8.2 Memory Operations

| Operation | Target | Current |
|-----------|--------|---------|
| Allocation (cached) | < 5 µs | Not measured |
| DMA-BUF export | < 100 µs | Stubbed |
| Cross-GPU copy (P2P) | PCIe BW | Not measured |
| Staging copy | Host RAM BW | Not measured |

---

## 9. Profiling Recommendations

### 9.1 Add Instrumentation Points

1. **Scheduler**: `scheduleDynamic()` entry/exit timestamps
2. **Memory**: `allocateOnGpu()` syscall vs kernel time
3. **IPC**: Message serialization/deserialization time
4. **Vulkan Layer**: `vkQueueSubmit` interception overhead

### 9.2 Recommended Tools

| Tool | Purpose |
|------|---------|
| `perf record` | CPU profiling |
| `strace -T` | Syscall latency |
| `Intel VTune` | Memory/CPU analysis |
| `NVIDIA Nsight` | CUDA path analysis |
| `AMD uProf` | AMD GPU analysis |

---

## 10. Priority Optimization Matrix

| Priority | Optimization | Effort | Expected Gain |
|----------|--------------|--------|--------------|
| **Critical** | Real GPU allocation paths | High | Functional correctness |
| **High** | Sharded memory locks | Medium | 2-4x throughput |
| **High** | P2P detection accuracy | Medium | Reduced staging |
| **Medium** | Lock-free queue | Medium | Reduced latency |
| **Medium** | Async IPC | High | Better scalability |
| **Low** | Async metrics collection | Low | Lower CPU usage */