SKILL.md

name: exps-performance description: Documentation for the main LLM performance experiments (exps_performance)

exps_performance - Main Performance Experiments

Purpose

Run LLM inference on algorithmic problems, collect CoT rationales and evaluate correctness.

Key Files

File	Purpose
`src/exps_performance/main.py`	Main runner
`src/exps_performance/dataset.py`	Dataset definitions
`src/exps_performance/arms.py`	Evaluation arms (NL, code_sim, code_exec)
`src/exps_performance/llm.py`	LLM interface
`src/exps_performance/analysis.py`	Analysis and plotting
`src/exps_performance/noise.py`	Noise injection experiments

Results Structure

src/exps_performance/results/
  {model}_seed{seed}/
    tb/
      run_{timestamp}/
        res.jsonl  # Main results file

res.jsonl Schema

Field	Description
`kind`	Problem type (e.g., "knap", "bellman_ford")
`digit`	Problem size/digits
`nl_correct`	NL evaluation result
`code_correct`	Code execution result
`sim_correct`	Code similarity result
`model`	Model name
`seed`	Random seed

Data Sources

Performance results: src/exps_performance/results/{model}_seed{seed}/tb/run_*/res.jsonl
Excludes gsm8k from accuracy calculations in logistic analysis

Running Commands

# Run main experiment
uv run python src/exps_performance/main.py --model <model> --seed <seed>

# Run analysis
uv run python src/exps_performance/analysis.py

Score

Total Score

40/100

Based on repository quality metrics

✓SKILL.md

SKILL.mdファイルが含まれている

+20

○LICENSE

ライセンスが設定されている

0/10

○説明文

100文字以上の説明がある

0/10

○人気

GitHub Stars 100以上

0/15

○最近の活動

3ヶ月以内に更新がある

0/10

○フォーク

10回以上フォークされている

0/5

✓Issue管理

オープンIssueが50未満

+5

✓言語

プログラミング言語が設定されている

+5

○タグ

1つ以上のタグが設定されている

0/5

Reviews

💬

Reviews coming soon