# MVGAL Power Management Specification

**Version:** 1.0  
**Date:** 2026-06-02

---

## 1. Overview

MVGAL provides power management for aggregated GPUs through the `mvgald` daemon's `PowerManager` subsystem. This component manages power states, idle detection, and dynamic frequency scaling for GPUs from multiple vendors.

---

## 2. Power States

### 2.1 State Machine

```
ACTIVE (0)
    │  idle_timeout_ms
    ▼
SUSTAINED (1)
    │  sustained_timeout_ms
    ▼
IDLE (2)
    │  park_timeout_ms
    ▼
PARK (3)
    │  activity detected
    ▼
ACTIVE (loop)
```

### 2.2 State Definitions

| State | Description | Vendor Interface |
|-------|-------------|------------------|
| `ACTIVE` | Normal operation, full performance | Vendor default |
| `SUSTAINED` | High performance mode | AMD: auto, NVIDIA: auto, others: N/A |
| `IDLE` | Reduced clocks, ready to wake | AMD: low, NVIDIA: N/A |
| `PARK` | Low power state | AMD: low, NVIDIA: off, Intel: rc6 |
| `OFF` | GPU powered off | ACPI D3hot |

---

## 3. Configuration

### 3.1 Default Configuration

File: `/etc/mvgal/mvgal.conf`

```ini
[power]
# Time without workload before going idle (ms)
idle_timeout_ms = 5000

# Time in idle before sustained mode (ms)
sustained_timeout_ms = 30000

# Time in sustained before parking (ms)
park_timeout_ms = 60000

# Temperature threshold for throttling (Celsius)
thermal_throttle_temp = 75

# Maximum power limit (Watts, 0 = auto)
max_power_limit_watts = 0

# Enable DVFS (dynamic frequency scaling)
enable_dvfs = true

# Enable PCIe L1 substates for idle GPUs
enable_pcie_l1 = true
```

### 3.2 Runtime Configuration

```bash
# View current power settings
mvgal-config get-power

# Set GPU-specific power state
echo "auto" > /sys/class/mvgal/mvgal0/gpu0/power_state
echo "low" > /sys/class/mvgal/mvgal0/gpu1/power_state

# Set idle timeout
mvgal-config set-idle-timeout 10000
```

---

## 4. Per-Vendor Power Control

### 4.1 AMD GPUs

AMD GPUs are controlled via the amdgpu driver's sysfs interface:

```bash
# Force performance level
echo "low" > /sys/bus/pci/devices/0000:03:00.0/power_dpm_force_performance_level
echo "high" > /sys/bus/pci/devices/0000:03:00.0/power_dpm_force_performance_level
echo "auto" > /sys/bus/pci/devices/0000:03:00.0/power_dpm_force_performance_level

# Power profile
echo "battery" > /sys/class/drm/card0/device/power_dpm_force_performance_level
```

### 4.2 NVIDIA GPUs

NVIDIA GPUs use multiple methods:

```bash
# Power management mode (NVIDIA proprietary driver)
nvidia-smi -pm 1 -i 0

# Set power limit (watts)
nvidia-smi -pl 150 -i 0

# Clock throttling
nvidia-smi -ac 5000,1590 -i 0  # memory, graphics

# AMD kernels: power state via sysfs
echo "auto" > /sys/bus/pci/devices/0000:04:00.0/power_dpm_force_performance_level
```

### 4.3 Intel GPUs

Intel GPUs support both i915 and xe drivers:

```bash
# i915 (legacy)
echo "mid" > /sys/class/drm/card0/device/power_dynamic_control

# xe (new)
echo "static" > /sys/class/drm/card1/device/power/config

# Frequency limits
echo "3000" > /sys/class/drm/card0/device/intel_frequency_max
```

### 4.4 Moore Threads GPUs

MTT GPUs use the MUSA management library:

```bash
# Query power state
musa-smi -q -i 0

# Set power limit
musa-smi -pl 120 -i 0
```

---

## 5. Kernel Module Power Interface

### 5.1 Power State IOCTL

```c
struct mvgal_power_state_request {
    __u32 gpu_index;
    __u32 power_state;  // MVGAL_POWER_STATE_*
    __u32 reserved[4];
};

#define MVGAL_IOC_SET_POWER_STATE _IOW('M', 0x20, struct mvgal_power_state_request)
```

### 5.2 Sysfs Attributes

```bash
# Per-GPU power state
ls /sys/class/mvgal/mvgal0/gpu0/
├── enabled
├── power_state
└── pci_path

# Read current power state
cat /sys/class/mvgal/mvgal0/gpu0/power_state  # "auto", "on", "off"

# Set power state
echo "auto" > /sys/class/mvgal/mvgal0/gpu0/power_state
```

---

## 6. idle Detection and Wake-on-Demand

### 6.1 Idle Detection Flow

```
Monitor GPU utilization every 100ms
    │
    ├── GPU busy? ── Yes ──→ Reset idle timer
    │                    │
    │                    No
    │                    │
    └── Idle time > threshold?
                         │
                         Yes
                         │
                         → Transition to IDLE state
                         │
                         Monitor for activity...
                         │
                         Activity detected
                         │
                         → Wake to ACTIVE state
```

### 6.2 Wake Sources

1. **IPC Socket Activity**: Client connects or submits workload
2. **Vulkan/OpenCL Submission**: API interception detects new work
3. **CUDA Kernel Launch**: CUDA shim detects kernel launch
4. **External Interrupt**: PCIe INTX or MSI from GPU

### 6.3 Wake Implementation

```c
/* In kernel module */
static irqreturn_t mvgal_gpu_irq_handler(int irq, void *data)
{
    struct mvgal_gpu_device *gpu = data;
    
    /* Wake the GPU */
    gpu->power_state = MVGAL_POWER_STATE_ACTIVE;
    wake_up(&gpu->power_waitq);
    
    return IRQ_HANDLED;
}
```

---

## 7. Thermal Management

### 7.1 Temperature Monitoring

```c
struct thermal_reading {
    uint32_t temperature;  /* Celsius */
    uint32_t throttle_level; /* 0-100% */
    uint64_t timestamp_ms;
};

static int mvgal_read_temperature(struct mvgal_gpu_device *gpu, uint32_t *temp)
{
    char path[128];
    struct file *f;
    char buf[16];
    loff_t pos = 0;
    ssize_t n;
    
    snprintf(path, sizeof(path),
             "/sys/class/drm/card%u/device/hwmon/hwmon0/temp1_input",
             gpu->gpu_index);
    
    f = filp_open(path, O_RDONLY, 0);
    if (IS_ERR(f))
        return PTR_ERR(f);
    
    n = kernel_read(f, buf, sizeof(buf) - 1, &pos);
    filp_close(f, NULL);
    
    if (n > 0) {
        *temp = simple_strtoul(buf, NULL, 10) / 1000;
        return 0;
    }
    return -EIO;
}
```

### 7.2 Throttling Response

| Temperature | Action |
|-------------|--------|
| < 70°C | Normal |
| 70-75°C | Log warning, start monitoring |
| 75-80°C | Reduce max clock by 10% |
| 80-85°C | Reduce max clock by 25% |
| > 85°C | Emergency throttle: force IDLE state |

---

## 8. PSU-Aware Power Limiting

### 8.1 Power Budget Concept

The PowerManager tracks total system GPU power draw and enforces limits:

```c
struct power_budget {
    uint64_t total_watts;      /* System GPU power budget */
    uint64_t used_watts;       /* Currently allocated */
    uint64_t reserved_watts;   /* Reserved for active workloads */
};
```

### 8.2 Budget Allocation

```c
static int allocate_power_budget(struct mvgal_gpu_device *gpu, size_t workload_flux)
{
    uint64_t required = estimate_power_for_workload(workload_flux);
    
    if (budget->used_watts + required > budget->total_watts) {
        return -ENOSPC;  /* Power budget exceeded */
    }
    
    budget->used_watts += required;
    gpu->power_allocation = required;
    return 0;
}
```

---

## 9. Monitoring and Telemetry

### 9.1 Metrics Collected

| Metric | Source | Frequency |
|--------|--------|-----------|
| Utilization % | sysfs `gpu_busy_percent` | 100ms |
| Memory used | TTM/DMA-BUF | 100ms |
| Temperature | hwmon | 1000ms |
| Power draw | RAPL/NVML | 1000ms |
| Clock speed | sysfs | 1000ms |
| PCIe bandwidth | erst counters | 5000ms |

### 9.2 Prometheus Format

Endpoint: `http://localhost:7474/metrics`

```
# HELP mvgal_gpu_utilization GPU utilization percentage
# TYPE mvgal_gpu_utilization gauge
mvgal_gpu_utilization{gpu_index="0",vendor="amd"} 45.2
mvgal_gpu_utilization{gpu_index="1",vendor="nvidia"} 23.7

# HELP mvgal_gpu_temperature_celsius GPU temperature in Celsius
# TYPE mvgal_gpu_temperature_celsius gauge
mvgal_gpu_temperature_celsius{gpu_index="0"} 65

# HELP mvgal_power_manager_state Current power state
# TYPE mvgal_power_manager_state gauge
mvgal_power_manager_state{gpu_index="0"} 1  # SUSTAINED
```

---

## 10. CLI Tools

### 10.1 mvgal-status

```bash
$ mvgal-status --watch
GPU 0 (AMD RX 6800): ████████████ 65% util, 72°C, 180W
GPU 1 (NVIDIA RTX 3080): ████████ 42% util, 68°C, 220W

Power States: IDLE, ACTIVE
Total Power: 400W / 550W budget
```

### 10.2 mvgal-config

```bash
$ mvgal-config set-power-state auto 0
$ mvgal-config set-idle-timeout 10000
$ mvgal-config set-max-power 450
```

---

## 11. Safety Considerations

### 11.1 Safe Defaults

1. **Power limits**: Default to 80% of PSU capacity
2. **Thermal limits**: Throttling starts at 75°C
3. **Emergency shutdown**: GPU OFF at 95°C
4. **Fallback**: Single-GPU mode if power budget exceeded

### 11.2 Error Handling

```c
/* Never fail the workload for power management issues */
static void handle_power_error(struct mvgal_gpu_device *gpu, int err)
{
    pr_warn("MVGAL: Power management error on GPU %u: %d\n", gpu->gpu_index, err);
    
    /* Log but don't fail the GPU */
    gpu->power_errors++;
    
    /* Revert to safe defaults */
    gpu->power_state = MVGAL_POWER_STATE_ACTIVE;
}
```

---

## 12. Future Enhancements

### 12.1 Planned Features

1. **NVIDIA GPU Manager (NVMe) integration** for server GPUs
2. **AMD XGMI power topology** for EPYC systems
3. **Intel Iris power optimization** for integrated graphics
4. **Moore Threads power curation** for data center deployments

### 12.2 Research Areas

1. **Machine learning-based power prediction**
2. **Cross-GPU power sharing** (budget borrowing/lending)
3. **Workload-aware frequency scaling**