---

name: skriptoteket-devops description: DevOps and server management for Skriptoteket on home server (hemma.hule.education). Branched skill covering deploy, database, users, CLI, security, network, DNS, and troubleshooting.

Skriptoteket DevOps

Compact skill for managing Skriptoteket on home server.

Source of truth for ops in this repo:

• Home server ops: docs/runbooks/runbook-home-server.md
• GPU AI ops: docs/runbooks/runbook-gpu-ai-workloads.md
• Tabby ops: docs/runbooks/runbook-tabby-codemirror.md
• Observability ops: docs/runbooks/runbook-observability.md

ROCm / AMDGPU installer flags (hemma)

• List supported usecases: ssh hemma "amdgpu-install --list-usecase"
• ROCm (headless/compute): ssh hemma "sudo amdgpu-install -y --usecase=rocm"
• Graphics + ROCm (Mesa + compute): ssh hemma "sudo amdgpu-install -y --usecase=graphics,rocm"
• Notes:
  • "Mesa graphics" == graphics (open source Mesa 3D + multimedia libs).
  • workstation is deprecated (and now maps to Mesa anyway); prefer graphics.

When to Use

Activate when the user:

• Needs to deploy changes to home server
• Wants to manage database (backup/restore/migrations)
• Creates or manages users (bootstrap, provision)
• Troubleshoots errors (502, 307, 500)
• Works with SSL/certificates
• Has DNS/DDNS issues
• Needs to run CLI commands in container
• Wants to seed or sync the script bank

---

Critical Configuration (Copy-Paste Ready)

SSH Access (Passwordless)

Copyssh hemma              # paunchygent (non-root default)
ssh hemma-root         # root (use only with explicit approval)
ssh hemma-local        # LAN, non-root
ssh hemma-local-root   # LAN, root

Notes (hemma):

• Default to non-root (ssh hemma); use ssh hemma-root only after explicit approval.
• Non-root key: ~/.ssh/hemma-paunchygent_ed25519 (local).
• Health-gated hardware watchdog handles recovery: /usr/local/bin/health-watchdog.sh + health-watchdog.service pets /dev/watchdog0 only when health checks pass (sshd/port 22, default route, link up, gateway ping). Boot ordering is pinned via /etc/systemd/system/health-watchdog.service.d/10-watchdog-order.conf (starts after sp5100-tco-watchdog.service; waits for /dev/watchdog0 node). systemd watchdog is disabled via /etc/systemd/system.conf.d/99-watchdog.conf, and sp5100_tco is loaded with nowayout=1 heartbeat=60 via /etc/modprobe.d/sp5100_tco.conf. Reboot persistence: watchdog.stop_on_reboot=0 in /etc/default/grub + /etc/default/kdump-tools. Logs in journalctl -t health-watchdog.
• Heartbeat log tag: journalctl -t heartbeat.
• Host incident/SMART logs: /root/logs/incident-*.log, /root/logs/smart/ (cleanup via cleanup-smart-logs.timer).
• Docker commands require sudo (passwordless sudo is configured). If you use sudo -n, the command fails fast instead of prompting.
• Prefer rg (ripgrep) for repo search; if missing: sudo apt-get install -y ripgrep.
• For a friendlier CLI on hemma, expect fd, bat, fzf, jq, tree, and yq (mikefarah/yq v4). Ubuntu packages name some of these as fdfind/batcat, so we symlink fdfind→fd and batcat→bat (see docs/runbooks/runbook-home-server.md).
• If any are missing: sudo apt-get update && sudo apt-get install -y fd-find bat fzf jq tree (and install yq v4 to /usr/local/bin/yq).
• If you need UTC timestamps in Python, avoid datetime.utcnow() (deprecated on 3.12+); use timezone-aware UTC like datetime.now(UTC) instead.
• SSH abuse protection: Fail2ban runs sshd + recidive (3 strikes within 7d => permaban); see docs/runbooks/runbook-home-server.md.
• Edge hardening: nginx-proxy drops common scanner traffic (e.g. /.env, /.git, wp-*, *.php, cgi-bin, WebDAV methods) at the proxy; see docs/runbooks/runbook-home-server.md.
• Observability: Promtail labels nginx-proxy access logs in Loki (vhost, client_ip, method, status); see docs/runbooks/runbook-observability-logging.md.

Watchdog / Crash Recovery Guardrails (hemma)

• Separate ISP loss from local SSH/link failure; only count local failures toward reboot unless user explicitly opts in.

• Always verify runtime state before assumptions: systemctl status ssh-watchdog.timer ssh-watchdog.service and /etc/default/ssh-watchdog.

• Keep the hardware watchdog armed; disable systemd watchdog in the kdump initramfs to avoid petting in the crash kernel.

• Keep watchdog running across warm reboots (especially post-kdump SysRq reboot): watchdog.stop_on_reboot=0 in /etc/default/grub + /etc/default/kdump-tools; apply with update-grub + reboot and kdump-config unload && kdump-config load.

• Run remote commands via script files (no inline quoting) to avoid shell/escape errors.

• Crash-kernel watchdog hardening uses initramfs hooks: /etc/initramfs-tools/hooks/zz-kdump-disable-watchdog (disable systemd petting) and /etc/initramfs-tools/hooks/zz-kdump-watchdog-hardening (include sp5100_tco module + /etc/modprobe.d/sp5100_tco.conf). Rebuild kdump initrd after changes.

• Crash-kernel watchdog must be started (module load alone isn’t enough): kdump-watchdog-arm.service + /usr/local/sbin/kdump-watchdog-arm opens /dev/watchdog0 before kdump-tools-dump.service. Verify: sudo journalctl -b -1 -u kdump-watchdog-arm.service --no-pager includes kdump-watchdog-arm: opened ... (state=active ...).

Production Deployment

Copy# ALWAYS use compose.prod.yaml for production
ssh hemma "cd ~/apps/skriptoteket && git pull && sudo docker compose -f compose.prod.yaml up -d --build"

# With migrations
ssh hemma "cd ~/apps/skriptoteket && git pull && sudo docker compose -f compose.prod.yaml up -d --build"
ssh hemma "sudo docker exec -e PYTHONPATH=/app/src skriptoteket-web pdm run db-upgrade"

Note: On hemma, systemd units may need absolute docker path (/snap/bin/docker) due to PATH differences.

Background Image Builds (REQUIRED)

Run builds in background, log to .artifacts/, and give the user the tail -f command. Template: LOG=.artifacts/runner-build-$(date -u +%Y%m%dT%H%M%SZ).log; nohup <build-cmd> > "$LOG" 2>&1 & echo "$LOG"; tail -f <log>

Env changes (prod)

Copy# Edit: ~/apps/skriptoteket/.env (avoid duplicate keys; last wins)

# Apply env/compose changes (restart does NOT re-read .env)
ssh hemma "cd ~/apps/skriptoteket && sudo docker compose -f compose.prod.yaml up -d --no-deps --force-recreate web"

Container Names

Database Connection

Copy# Production DB (shared-postgres on hule-network)
DATABASE_URL=postgresql+asyncpg://skriptoteket:${SKRIPTOTEKET_DB_PASSWORD}@shared-postgres:5432/skriptoteket

# Dev DB (local container)
DATABASE_URL=postgresql+asyncpg://postgres:postgres@db:5432/skriptoteket

# Connect to prod DB via psql
ssh hemma "sudo docker exec -it shared-postgres psql -U skriptoteket -d skriptoteket"

# Connect to dev DB
ssh hemma "sudo docker exec -it skriptoteket-db-1 psql -U postgres -d skriptoteket"

CLI Command Pattern (ALWAYS USE)

Copy# The -e PYTHONPATH=/app/src is REQUIRED for all CLI commands (web + worker)
sudo docker exec -e PYTHONPATH=/app/src skriptoteket-web pdm run <command>
sudo docker exec -e PYTHONPATH=/app/src skriptoteket-worker pdm run <command>

# Non-interactive (scripts/CI): add -T
sudo docker exec -T -e PYTHONPATH=/app/src skriptoteket-web pdm run <command>
sudo docker exec -T -e PYTHONPATH=/app/src skriptoteket-worker pdm run <command>

# Interactive (prompts): add -it
sudo docker exec -it -e PYTHONPATH=/app/src skriptoteket-web pdm run <command>
sudo docker exec -it -e PYTHONPATH=/app/src skriptoteket-worker pdm run <command>

See also: docs/runbooks/runbook-home-server.md (systemd timer patterns use /snap/bin/docker exec ...).

Worker healthcheck

The production worker container exposes a dependency healthcheck (DB + Docker socket + artifacts volume):

Copyssh hemma "sudo docker exec -e PYTHONPATH=/app/src skriptoteket-worker pdm run python -m skriptoteket.cli healthcheck-execution-worker"

Admin Credentials (Script Bank Seeding)

Copy# Default admin for seeding (set in .env on server)
SKRIPTOTEKET_SCRIPT_BANK_ACTOR_EMAIL=admin@hule.education
SKRIPTOTEKET_SCRIPT_BANK_ACTOR_PASSWORD=<from server .env>

# Seed command
ssh hemma "cd ~/apps/skriptoteket && sudo docker compose -f compose.prod.yaml exec -T -e PYTHONPATH=/app/src web pdm run python -m skriptoteket.cli seed-script-bank --actor-email admin@hule.education --actor-password 'PASSWORD'"

Network Configuration

Critical: Production web container must be on hule-network for nginx to reach it.

File Paths on Server

Copy~/apps/skriptoteket/           # App repo (git pull here)
~/apps/skriptoteket/.env       # Production secrets (never commit)
~/infrastructure/              # nginx-proxy, certbot
~/backups/                     # Database backups
/srv/storage/                  # Long-term data (HDD; models/data/archives)
/srv/backup/                   # Long-term backups (ext4)
/srv/scratch/                  # Fast ephemeral work (SSD; tmp/build/cache)
/root/logs/incident-*.log      # Incident log captures
/root/logs/smart/              # SMART snapshots

---

Most Used Commands

Deploy

Copy# Standard deploy
ssh hemma "cd ~/apps/skriptoteket && git pull && sudo docker compose -f compose.prod.yaml up -d --build"

# With migrations
ssh hemma "sudo docker exec skriptoteket-web pdm run db-upgrade"

# Force recreate (config changes)
ssh hemma "cd ~/apps/skriptoteket && sudo docker compose -f compose.prod.yaml up -d --force-recreate"

Database

Copy# Backup
ssh hemma "sudo docker exec shared-postgres pg_dump -U skriptoteket skriptoteket > ~/backups/skriptoteket-\$(date +%Y%m%d).sql"

# Migrations
ssh hemma "sudo docker exec skriptoteket-web pdm run db-upgrade"

Edit-ops context probe (dev, docker backend required)

Use this to verify that chat + edit-ops see the same virtual file base hashes.

Copy# Ensure the dev stack is running (docker compose)
pdm run dev-start

# Run the probe (uses BOOTSTRAP_SUPERUSER_* from .env)
pdm run python scripts/chat_edit_ops_context_probe.py \
  --scenario scripts/edit_ops_scenarios/chat_edit_ops_context_example.json

# Inspect the captured summary + raw payloads
cat .artifacts/chat-edit-ops-context/<timestamp>/capture_summary.json

Notes:

• The probe talks to the docker-backed API at http://127.0.0.1:8000.
• Captures land under .artifacts/llm-captures/chat_request_context/ and .artifacts/llm-captures/chat_ops_response/.

Cleanup timers (systemd)

We enforce TTL-based cleanup using CLI commands + systemd timers (not cron). Examples:

• Sandbox snapshots cleanup: cleanup-sandbox-snapshots
• Login events cleanup (retention): cleanup-login-events
• Host log cleanup: cleanup-smart-logs.timer (incident + SMART logs, 30-day retention)

Exact unit definitions and schedules: docs/runbooks/runbook-home-server.md.

Users

Copy# Bootstrap superuser (first time)
ssh hemma "sudo docker exec -it -e PYTHONPATH=/app/src skriptoteket-web pdm run python -m skriptoteket.cli bootstrap-superuser --email admin@hule.education"

# Provision user
ssh hemma "sudo docker exec -T -e PYTHONPATH=/app/src skriptoteket-web pdm run python -m skriptoteket.cli provision-user --actor-email admin@hule.education --actor-password 'ADMIN_PASS' --email user@example.com --password 'USER_PASS' --role contributor"

Logs

Copyssh hemma "sudo docker logs -f skriptoteket-web"
ssh hemma "sudo docker logs -f nginx-proxy"

# Host AI services (llama.cpp + tabby)
ssh hemma "sudo systemctl status --no-pager llama-server-rocm.service tabby.service | head -n 60"
ssh hemma "sudo journalctl -u llama-server-rocm.service -n 200 --no-pager"
ssh hemma "sudo journalctl -u tabby.service -n 200 --no-pager"
ssh hemma "curl -s http://127.0.0.1:8082/health"
ssh hemma "curl -s http://127.0.0.1:8083/v1/health"

Status

Copyssh hemma "sudo docker ps | grep -E 'skriptoteket|nginx|postgres'"

---

Local Dev (DX)

Copy# Fast local iteration with log piping
pdm run dev-logs
pdm run fe-dev-logs
pdm run dev-local

# Full dev containers + logs
pdm run dev-start
pdm run dev-containers-logs

# Rebuild containers when deps/reload are flaky
pdm run dev-rebuild

---

GPU Tunnels (Local Workstation)

Copy~/bin/hemma-gpu-tunnel start        # start llama + tabby tunnels
~/bin/hemma-gpu-tunnel start-llama  # start only llama tunnel (:8082)
~/bin/hemma-gpu-tunnel start-tabby  # start only tabby tunnel (:8083)
~/bin/hemma-gpu-tunnel stop         # stop both tunnels
~/bin/hemma-gpu-tunnel stop-llama   # stop only llama tunnel (:8082)
~/bin/hemma-gpu-tunnel stop-tabby   # stop only tabby tunnel (:8083)
~/bin/hemma-gpu-tunnel status       # show tunnel status

---

Branch Routing

---

Quick Troubleshooting

---

Research Documentation

Full research: docs/reference/reports/ref-devops-skill-research.md

Maintenance note

This skill includes deeper branches under .claude/skills/skriptoteket-devops/branches/. Keep those branch docs aligned with the runbooks above when ops patterns change.