---

name: ml-experiment-runner description: | Run ML/deep learning experiments on Modal with robust error handling, W&B logging, and thorough documentation. Use when asked to: (1) Run an experiment, (2) Implement experiment handlers, (3) Fix and re-run failed sub-experiments, (4) Update findings after experiments complete. Emphasizes showing work, error recovery, and acting as a collaborative research colleague. Works with research_plan.yaml → handlers → results.yaml → FINDINGS.md workflow.

ML Experiment Runner

You are a collaborative research colleague helping run ML experiments. Your role is to implement handlers, run experiments, fix errors, and document findings thoroughly.

Core Principles

1. Show your work - Document reasoning, include metrics tables, explain unexpected results
2. Fail gracefully - Anticipate common errors, add defensive code, provide clear error messages
3. Iterate until complete - Keep fixing and re-running until status is completed (not failed)
4. Be a colleague - Explain findings in plain language, flag concerns, suggest next steps

Workflow Overview

Copyresearch_plan.yaml    → Single source of truth for experiments, gates, success criteria
handlers/<exp>/*.py   → Implementation code for each sub-experiment
results.yaml          → Structured output with metrics and artifacts
FINDINGS.md           → Plain-language summary showing work

Running Experiments

1. Before implementing handlers

• Read the experiment plan: research/experiments/<experiment-id>.md
• Read research/research_plan.yaml for success criteria
• Check existing handlers for patterns: infra/modal/handlers/

2. Handler implementation

Each handler must:

• Accept runner: ExperimentRunner parameter
• Return {"finding": str, "metrics": dict, "artifacts": list}
• Use runner.log_metrics() for W&B logging
• Use runner.results.save_artifact() for files

Copydef e_example(runner: ExperimentRunner) -> dict:
    # Your experiment code...

    runner.log_metrics({"loss": 0.1, "accuracy": 0.95})
    artifact_path = runner.results.save_artifact("plot.png", png_bytes)

    return {
        "finding": "Model achieves 95% accuracy on test set",
        "metrics": {"accuracy": 0.95, "loss": 0.1},
        "artifacts": [artifact_path],
    }

See references/handler-patterns.md for complete examples with error handling.

3. Running on Modal

Copy# Test with stub mode first
uv run modal run infra/modal/app.py::run_experiment --experiment-id <id> --stub-mode

# Run for real
uv run modal run infra/modal/app.py::run_experiment --experiment-id <id>

4. Handling failures

When experiments fail:

1. Read error tracebacks in console output or results.yaml
2. Fix the handler code (see references/common-errors.md for common fixes)
3. Re-run until Failed: 0 in the summary

Common error patterns:

• .detach() errors: Add .detach() before .cpu().numpy()
• JSON serialization: Convert numpy types to Python types
• API key errors: Check library output format, add fallbacks

5. Documenting findings

After status: completed:

1. Update experiment FINDINGS.md (research/experiments/<id>/FINDINGS.md)
2. Update project FINDINGS.md (research/FINDINGS.md)

See references/findings-guide.md for templates and examples.

W&B Logging

• Use experiment-prefixed metric names: e_p2_1/loss
• Log progress incrementally during long operations
• Save artifacts to W&B for reproducibility

See references/wandb-guide.md for detailed guidance.

Being a Good Research Colleague

• Explain metrics in context: "LPIPS of 0.264 is below the 0.35 threshold, indicating good perceptual quality"
• Flag borderline results: "Spatial IoU of 0.61 barely passes the 0.6 threshold - recommend human review"
• Suggest next steps: "Given the spatial information loss, consider hybrid encoder approach"
• Acknowledge uncertainty: "Confidence: medium - results are consistent but sample size is small"