---

name: gpu-debugger description: "Debug failed GPU CLI runs. Analyze error messages, diagnose OOM errors, fix sync issues, troubleshoot connectivity, and resolve common problems. Turn cryptic errors into actionable fixes."

GPU Debugger

Turn errors into solutions.

This skill helps debug failed GPU CLI runs: OOM errors, sync failures, connectivity issues, model loading problems, and more.

When to Use This Skill

Debugging Workflow

CopyError occurs
     │
     ▼
┌─────────────────────────┐
│ 1. Collect information  │
│    - Error message      │
│    - Daemon logs        │
│    - Exit code          │
│    - VRAM usage         │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ 2. Identify error type  │
│    - OOM                │
│    - Network            │
│    - Model              │
│    - Sync               │
│    - Permission         │
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│ 3. Apply fix            │
│    - Config change      │
│    - Code change        │
│    - Retry              │
└─────────────────────────┘

Information Collection Commands

Check Daemon Logs

Copy# Last 50 log lines
gpu daemon logs --tail 50

# Full logs since last restart
gpu daemon logs

# Follow logs in real-time
gpu daemon logs --follow

Check Pod Status

Copy# Current pod status
gpu status

# Pod details
gpu pods list

Check Job History

Copy# Recent jobs
gpu jobs list

# Specific job details
gpu jobs show <job-id>

Common Errors and Solutions

1. CUDA Out of Memory (OOM)

Error messages:

CopyCUDA out of memory. Tried to allocate X GiB
RuntimeError: CUDA error: out of memory
torch.cuda.OutOfMemoryError

Diagnosis:

Copy# In your script, check VRAM usage
import torch
print(f"VRAM allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"VRAM reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB")

Solutions by severity:

Quick fixes:

Copy# Reduce batch size
BATCH_SIZE = 1  # Start small, increase until OOM

# Enable gradient checkpointing
model.gradient_checkpointing_enable()

# Use FP16
model = model.half()

# Use INT4 quantization
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(load_in_4bit=True)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)

# CPU offloading (for diffusers)
pipe.enable_model_cpu_offload()

# Clear cache between batches
torch.cuda.empty_cache()

Config fix:

Copy{
  // Upgrade to larger GPU
  "gpu_type": "A100 PCIe 80GB",  // Instead of RTX 4090
  "min_vram": 80
}

2. Connection Refused / Timeout

Error messages:

CopyConnection refused
Connection timed out
SSH connection failed
Failed to connect to daemon

Diagnosis:

Copy# Check daemon status
gpu daemon status

# Check if daemon is running
ps aux | grep gpud

# Check daemon logs
gpu daemon logs --tail 20

Solutions:

Restart daemon:

Copygpu daemon stop
gpu daemon start

3. Pod Won't Start / Provisioning Failed

Error messages:

CopyFailed to create pod
No GPUs available
Insufficient resources
Provisioning timeout

Diagnosis:

Copy# Check available GPUs
gpu machines list

# Check specific GPU availability
gpu machines list --gpu "RTX 4090"

Solutions:

Config fixes:

Copy{
  // Use min_vram instead of exact GPU
  "gpu_type": null,
  "min_vram": 24,  // Any GPU with 24GB+

  // Or try different GPU
  "gpu_type": "RTX A6000",  // Alternative to RTX 4090

  // Or relax region constraint
  "region": null,  // Any region

  // Or increase price tolerance
  "max_price": 2.0  // Allow up to $2/hr
}

4. Sync Errors

Error messages:

Copyrsync error
Sync failed
File not found
Permission denied during sync

Diagnosis:

Copy# Check sync status
gpu sync status

# Check .gitignore
cat .gitignore

# Check outputs config
cat gpu.jsonc | grep outputs

Solutions:

Config fixes:

Copy{
  // Ensure outputs are configured
  "outputs": ["output/", "results/", "models/"],

  // Exclude large files
  "exclude_outputs": ["*.tmp", "*.log", "checkpoints/"],

  // Increase storage
  "workspace_size_gb": 100
}

5. Model Loading Errors

Error messages:

CopyModel not found
Could not load model
Safetensors error
HuggingFace rate limit

Diagnosis:

Copy# Check if model is downloading
# Look for download progress in job output

# Check HuggingFace cache on pod
gpu run ls -la ~/.cache/huggingface/hub/

Solutions:

Config fixes:

Copy{
  // Pre-download models
  "download": [
    { "strategy": "hf", "source": "meta-llama/Llama-3.1-8B-Instruct", "timeout": 7200 }
  ],

  // Set HF token in environment
  "environment": {
    "shell": {
      "steps": [
        { "run": "echo 'HF_TOKEN=your_token' >> ~/.bashrc" }
      ]
    }
  }
}

6. Process Hanging / Stuck

Symptoms:

• No output for a long time
• Process doesn't exit
• gpu status shows job running forever

Diagnosis:

Copy# Check if process is actually running
gpu run ps aux | grep python

# Check for infinite loops in logs
gpu jobs logs <job-id> --tail 100

# Check VRAM (might be swapping)
gpu run nvidia-smi

Solutions:

Code fixes:

Copy# Add timeout to model loading
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    low_cpu_mem_usage=True  # Prevent OOM during loading
)

# Add timeout to HTTP requests
import requests
response = requests.get(url, timeout=30)

# Add progress bars to see activity
from tqdm import tqdm
for item in tqdm(items):
    process(item)

7. Exit Code Errors

Common exit codes:

Diagnosis:

Copy# Check last job exit code
gpu jobs list --limit 1

# Get full output including error
gpu jobs logs <job-id>

Solutions:

8. CUDA Version Mismatch

Error messages:

CopyCUDA error: no kernel image is available
CUDA version mismatch
Torch not compiled with CUDA enabled

Diagnosis:

Copy# Check CUDA version on pod
gpu run nvcc --version
gpu run nvidia-smi | head -3

# Check PyTorch CUDA version
gpu run python -c "import torch; print(torch.version.cuda)"

Solutions:

Copy{
  // Use a known-good base image
  "environment": {
    "base_image": "runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04"
  }
}

Or in requirements.txt:

Copy# Install PyTorch with specific CUDA version
--extra-index-url https://download.pytorch.org/whl/cu124
torch==2.4.0

Debug Script Template

Add this to your projects for better error info:

Copy#!/usr/bin/env python3
"""Wrapper script with debugging info."""

import sys
import traceback
import torch

def print_system_info():
    """Print system info for debugging."""
    print("=" * 50)
    print("SYSTEM INFO")
    print("=" * 50)
    print(f"Python: {sys.version}")
    print(f"PyTorch: {torch.__version__}")
    print(f"CUDA available: {torch.cuda.is_available()}")
    if torch.cuda.is_available():
        print(f"CUDA version: {torch.version.cuda}")
        print(f"GPU: {torch.cuda.get_device_name(0)}")
        print(f"VRAM total: {torch.cuda.get_device_properties(0).total_memory / 1e9:.2f} GB")
        print(f"VRAM allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
    print("=" * 50)

def main():
    # Your actual code here
    pass

if __name__ == "__main__":
    print_system_info()
    try:
        main()
    except Exception as e:
        print("\n" + "=" * 50)
        print("ERROR OCCURRED")
        print("=" * 50)
        print(f"Error type: {type(e).__name__}")
        print(f"Error message: {str(e)}")
        print("\nFull traceback:")
        traceback.print_exc()

        # Print memory info for OOM debugging
        if torch.cuda.is_available():
            print(f"\nVRAM at error: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

        sys.exit(1)

Quick Reference: Error → Solution

Output Format

When debugging:

Copy## Debug Analysis

### Error Identified

**Type**: [OOM/Network/Model/Sync/etc.]
**Message**: `[exact error message]`

### Root Cause

[Explanation of why this happened]

### Solution

**Option 1** (Recommended): [solution]
```[code/config change]```

**Option 2** (Alternative): [solution]
```[code/config change]```

### Prevention

To avoid this in the future:
1. [Prevention tip 1]
2. [Prevention tip 2]