---

name: qa-debugging description: Systematic debugging methodologies, troubleshooting workflows, logging strategies, error tracking, performance profiling, stack trace analysis, and debugging tools across languages and environments. Covers local debugging, distributed systems, production issues, and root cause analysis.

Debugging & Troubleshooting (Jan 2026) — Quick Reference

This skill provides execution-ready debugging strategies, troubleshooting workflows, and root cause analysis techniques.

Core references: Google SRE troubleshooting patterns (Effective Troubleshooting) and SLO-driven reliability/triage (Service Level Objectives); observability standards via OpenTelemetry (Docs) and W3C Trace Context (Spec).

---

Core QA (Default)

Workflow (Reproduce → Isolate → Instrument → Fix → Verify → Regress)

Reproduce:

• Capture exact failure signature: error message, stack trace, request ID/trace ID, timestamp, build SHA, environment, user/tenant, seed/test data IDs.
• Quantify reproducibility: “fails 3/20 runs” is different from “fails 20/20”.

Isolate:

• Reduce scope: minimal input, minimal config, smallest component boundary.
• Bisect changes (git bisect / feature flags) when it started “recently”.

Instrument:

• Prefer structured logs + correlation IDs and traces over ad-hoc print statements (OpenTelemetry, W3C Trace Context).
• Add assertions/guards to fail fast at the true boundary.

Fix:

• Fix root cause, not symptoms; avoid “papering over” with retries/sleeps.

Verify:

• Add regression test at the lowest effective layer; validate in CI-like conditions.

Regress:

• Record the narrative: trigger, root cause, fix, prevention, and what signal would have caught it earlier (Effective Troubleshooting).

Debugging Ergonomics (Make Failures Cheap)

• Standardize a failure bundle:
  • Logs (structured), trace links, key metrics snapshot, and repro steps.
  • Test artifacts (screenshots/trace/video for UI; core dumps/crash reports where relevant).
• REQUIRED: every automated suite defines what artifacts are produced on failure and where they live.

Flaky/Intermittent Failures (Test and Prod)

• Treat flakes as reliability debt, not “noise”.
• First action: classify the flake type:
  • Timing/race: missing waits, async hazards, eventual consistency.
  • Environment: CPU/memory pressure, timezones/locales, throttling.
  • Data: shared state, ordering dependency, non-deterministic fixtures.
• Use controlled repetition: run the test N times with tracing enabled; correlate failures via request/trace IDs.

Debugging Checklist (Universal)

CopyBefore debugging:
[ ] Can you reproduce it consistently?
[ ] Do you have logs/error messages?
[ ] Do you have a minimal test case?
[ ] Do you know when it started?

During debugging:
[ ] Form hypothesis before making changes
[ ] Test one variable at a time
[ ] Document what you've tried
[ ] Use version control (commit working states)

After debugging:
[ ] Fix verified in all environments?
[ ] Regression test added?
[ ] Root cause documented?
[ ] Team notified of findings?

Do / Avoid

Do:

• Start with the smallest reliable reproducer
• Use evidence to support hypotheses (logs, traces, metrics, stack traces)
• Add guardrails and regression tests as part of the fix
• Document the fix for future reference

Avoid:

• Adding sleeps to "stabilize" tests without proving the underlying race
• Disabling tests or lowering assertions to make CI green
• Debugging directly in production without a safe, scoped plan (feature flags, sampling, read-only probes)
• Making random changes without a hypothesis

Quick Reference

---

Decision Tree: Debugging Strategy

CopyIssue type: [Problem Scenario]
    ├─ Application Behavior?
    │   ├─ Crashes immediately? → Check stack trace, error logs
    │   ├─ Slow/hanging? → CPU/memory profiling
    │   ├─ Intermittent failures? → Add logging, reproduce consistently
    │   └─ Unexpected output? → Binary search (add logs to narrow down)
    │
    ├─ Performance Issues?
    │   ├─ High CPU? → CPU profiler to find hot functions
    │   ├─ Memory leak? → Heap snapshots, track over time
    │   ├─ Slow database? → EXPLAIN ANALYZE, check indexes
    │   ├─ Network latency? → Trace external API calls
    │   └─ Frontend slow? → Lighthouse, Web Vitals profiling
    │
    ├─ Production-Only?
    │   ├─ Can't reproduce? → Analyze logs for patterns
    │   ├─ Environment difference? → Compare configs, data volume
    │   ├─ Need safe debugging? → Feature flags for verbose logging
    │   └─ Recent deployment? → Git bisect to find regression
    │
    ├─ Distributed Systems?
    │   ├─ Multiple services involved? → Distributed tracing (Jaeger)
    │   ├─ Request lost? → Search logs by request ID
    │   ├─ Service dependency? → Check health checks, circuit breakers
    │   └─ Async workflow? → Trace message queue, event logs
    │
    └─ Error Type?
        ├─ TypeError/NullPointer? → Check object existence, defensive coding
        ├─ Network timeout? → Check external service health, retry logic
        ├─ Database error? → Check connection pool, query syntax
        └─ Unknown error? → Systematic debugging workflow (observe, hypothesize, test)

---

When to Use This Skill

Use this skill when a user reports:

• Application crashes or errors
• Unexpected behavior or bugs
• Performance issues (slow queries, memory leaks, high CPU)
• Production incidents requiring root cause analysis
• Stack trace or error message interpretation
• Debugging strategies for specific scenarios
• Log analysis and pattern detection
• Distributed system debugging (microservices, async workflows)
• Memory leaks and resource exhaustion
• Race conditions and concurrency issues
• Network connectivity problems
• Database query optimization
• Third-party API integration issues

---

Operational Deep Dives

See references/operational-patterns.md for systematic debugging workflows, logging strategy details, stack trace and performance profiling guides, and language-specific tooling checklists.

---

Templates (Copy-Paste Ready)

Production templates organized by workflow type:

• Debugging Workflow: assets/debugging/template-debugging-checklist.md - Universal debugging checklist with specialized checklists for performance, memory leaks, distributed systems, and production incidents
• Debugging Worksheet: assets/debugging/template-debugging-worksheet.md - One-page worksheet (repro → isolate → instrument → verify) for fast, consistent triage
• Incident Response: assets/incidents/template-incident-response.md - Complete incident response playbook with severity levels, communication templates, and postmortem format
• Logging Setup: assets/observability/template-logging-setup.md - Production logging configurations for Node.js (Pino), Python (structlog), Go (zap), with Docker and CloudWatch integration

---

Resources (Deep-Dive Guides)

Operational best practices by domain:

• Operational Patterns: references/operational-patterns.md - Core debugging workflows, stack trace triage, profiling guides, tool selection, tail sampling strategies, async trace propagation
• Debugging Methodologies: references/debugging-methodologies.md - Scientific method, binary search, delta debugging, rubber duck, time-travel debugging, observability-first approaches, debugging retrospectives (team practice)
• Logging Best Practices: references/logging-best-practices.md - Structured logging, log levels, what to log/not log, implementations by language, request ID propagation, performance optimization
• Production Debugging: references/production-debugging-patterns.md - Safe production debugging techniques, log analysis, metrics, distributed tracing, feature flags, incident response workflow

---

Navigation

Resources

• references/operational-patterns.md
• references/debugging-methodologies.md
• references/logging-best-practices.md
• references/production-debugging-patterns.md

Templates

• assets/debugging/template-debugging-checklist.md
• assets/debugging/template-debugging-worksheet.md
• assets/incidents/template-incident-response.md
• assets/observability/template-logging-setup.md

Data

• data/sources.json — Curated external references

---

AI-Assisted Debugging (2026 Standard)

AI-powered debugging is now core practice, not optional. Use AI to accelerate triage while maintaining evidence-based rigor.

IDE-Integrated AI Debugging

Workflow:

Copy1. Test fails → Right-click → "Debug with Copilot"
2. AI analyzes test, code, and recent changes
3. Forms hypothesis, applies fix, re-runs test
4. Iterates until pass or hands back to developer

LLM Application Debugging

For AI/LLM applications, use specialized observability:

LLM Debugging Checklist:

Copy[ ] Trace full prompt chain (system + user + context)
[ ] Log token counts and latencies per call
[ ] Capture model responses with timestamps
[ ] Track cost per request
[ ] Version prompts separately from code
[ ] Use evals to catch regressions

AI-Assisted RCA Best Practices

Do:

• Use reasoning models (o1, Claude) for complex RCA — they step through codebase layers systematically
• Summarize logs/traces and cluster failures; include "evidence snippets" (IDs, timestamps, top errors)
• Generate hypotheses, then test them with targeted instrumentation
• Let AI suggest fixes, but verify with evidence before applying

Avoid:

• Accepting AI root cause without corroborating evidence (logs, traces, metrics)
• Copying suggested fixes without adding regression tests
• Using AI for production changes without human review
• Trusting AI-generated code that accesses external systems without validation

---

External Resources

See data/sources.json for:

• Debugging tool documentation
• Error tracking platforms (Sentry, Rollbar, Bugsnag)
• Observability platforms (Datadog, New Relic, Honeycomb)
• Profiling tutorials and guides
• Production debugging best practices

---

Quick Decision Matrix

---

Anti-Patterns to Avoid

• Random changes - Making changes without hypothesis
• Inadequate logging - Can't debug what you can't see
• Debugging in production - Always reproduce locally when possible
• Ignoring stack traces - Stack trace tells you exactly where error occurred
• Not writing tests - Fix today, break tomorrow
• Symptom fixing - Treating symptoms instead of root cause
• No monitoring - Flying blind in production
• Skipping postmortems - Not learning from incidents

---

Related Skills

This skill works with other skills in the framework:

Development & Operations:

• git-workflow - Git bisect for finding regressions, version control workflows
• dev-api-design - API debugging, error handling, REST patterns, status codes

Infrastructure & Platform:

• ops-devops-platform - CI/CD pipelines, monitoring, incident response, SRE practices, Kubernetes ops
• data-sql-optimization - Database query optimization, EXPLAIN ANALYZE, index tuning, slow query debugging

---

Success Criteria: Issues are diagnosed systematically, root causes are identified accurately, fixes include regression tests, and debugging knowledge is documented for future reference.