
knowledge-base
by shinpr
Compare, improve, and verify prompt changes with evidence — not vibes.
SKILL.md
name: knowledge-base description: Project-specific prompt optimization knowledge management. Use when storing or retrieving learned patterns from comparisons. Provides schema, extraction criteria, capacity management, and retention scoring.
Knowledge Base Skill
Storage Location
{project_root}/.claude/.rashomon/prompt-knowledge.yaml
Schema
patterns:
- name: "Pattern name"
what_to_look_for: |
When this pattern applies
improvement: |
How to improve when detected
learned_from: "Date and context"
confidence: 0.0-1.0
times_applied: 0
anti_patterns:
- name: "Anti-pattern name"
what_to_look_for: |
What to avoid
why_bad: |
Why problematic in this project
learned_from: "Date and context"
confidence: 0.0-1.0
metadata:
last_updated: "ISO-8601 timestamp"
total_comparisons: 0
patterns_count: 0
anti_patterns_count: 0
max_entries: 20
Extraction Criteria
Save as Improvement Pattern
ALL conditions must be true:
- Optimized prompt showed structural improvement (not variance)
- Improvement is project-specific (not explained by BP-001~008)
- Pattern is likely to recur in this project
Confidence Assignment:
| Evidence | Confidence |
|---|---|
| Multiple comparisons confirmed | 0.8+ |
| Single comparison, clear effect | 0.5-0.7 |
| Effect present but uncertain | 0.3-0.5 |
Minimum threshold: 0.3 (entries below this are skipped)
Save as Anti-Pattern
ALL conditions must be true:
- Original had problem specific to this project
- Problem is project-specific (beyond standard patterns BP-001~008)
- Problem likely to recur
Extraction Scope
Save only entries that are:
- Project-specific (beyond standard best practices BP-001~008)
- Likely to recur in this project
- Showing clear effect (structural improvement, confidence ≥ 0.3)
Capacity Management
Maximum: 20 entries (patterns + anti_patterns combined)
Retention Score: confidence * (1 + log(times_applied + 1))
This formula:
- Prioritizes high-confidence entries
- Rewards frequently-used patterns
- Treats all entries equally regardless of age
Key Principle: Old entries are valuable. Retention depends on confidence and usage frequency.
Eviction Process:
- Calculate retention scores for all entries
- Calculate score for new candidate
- If new > lowest existing: remove lowest, add new
- Otherwise: skip new entry
Operations
Retrieval
At start of prompt analysis:
- Read
.claude/.rashomon/prompt-knowledge.yaml(if exists) - For each entry, check
what_to_look_foragainst current prompt - Return relevant entries with relevance scores
- Increment
times_appliedfor patterns used
Storage
After comparison (if structural improvement found):
- Evaluate against extraction criteria
- Generate candidate entries
- Check for duplicates
- Apply capacity management
- Write updated knowledge base
- Update metadata
Example Entry
patterns:
- name: "TypeScript interface reference"
what_to_look_for: |
Code generation prompts creating TypeScript types without
referencing existing type definitions in src/types/
improvement: |
Add: "Reference existing types in src/types/ to maintain
consistency and avoid duplicate type definitions"
learned_from: "2026-01-14: Comparison showed better type reuse"
confidence: 0.7
times_applied: 3
Feedback-Based Adjustments
When comparison results require knowledge base updates:
Confidence Adjustments:
- User confirms improvement: +0.1 (cap at 0.95)
- Pattern led to worse result: -0.2
- Remove entry if confidence < 0.2 after decrease
Entry Management:
- Add new entries from user insight (initial confidence: 0.5)
- Remove entries that fall below confidence threshold
Score
Total Score
Based on repository quality metrics
SKILL.mdファイルが含まれている
ライセンスが設定されている
100文字以上の説明がある
GitHub Stars 100以上
1ヶ月以内に更新
10回以上フォークされている
オープンIssueが50未満
プログラミング言語が設定されている
1つ以上のタグが設定されている
Reviews
Reviews coming soon


