Runbook

When a fire misbehaves

Common failure modes and what they mean. Start with hivectl doctor (host CLIs) and hivectl tail <ISSUE> (live pod stdout — the only path from pod to your Mac).

Pod stuck in ImagePullBackOff — after recreating the cluster

The two images are built and imported into the cluster once by hivectl agent-setup, and a fire only references them (imagePullPolicy: IfNotPresent) — it never rebuilds. The images share the secret's lifecycle, so a fresh cluster has neither.

hivectl agent-setup     # rebuild + re-import coding-agent and web-qa-agent

A LoadBalancer service has no external IP — app unreachable on Mac

minikube has no built-in load balancer. Expose services manually:

minikube tunnel -p local-cluster      # in a separate terminal, or:
kubectl port-forward svc/<svc> <port>:<port>

Or use hivectl expose to port-forward prometheus + headlamp + the configured app at once.

The driver refuses to fire an issue — "DoD missing"

Every issue needs a ## Definition of Done H2 with at least one verifiable - [ ] line. The driver will not auto-stub one — it comments asking for it. Add the section and set the issue back to status:ready. (Exception: type:qa-feedback issues, where the driver drafts the DoD itself.)

An issue is stuck in needs-human with a ## Plan (proposed by driver) block

The exact heading is load-bearing: ## Plan fires, ## Plan (proposed by driver) does not (the pod double-checks at startup). A human reviews the proposed plan, renames the heading to plain ## Plan, and flips status back to ready.

A fire stopped early with cost_cap_hit:true

Each fire has a per-task USD ceiling ($5 for coding modes + research, $2 for qa), plus a separate $2 for the worker's in-pod review. It's a ceiling, not a budget. Raise it per-fire:

COST_LIMIT_USD=8 hivectl fire 42 --type=feature-implementation

Or set defaults.cost_limit_usd in .hive/config.yaml as a flat project override.

A pod was OOM-killed or throttled

Per-role limits are guardrails: worker cpu=1, memory=2Gi; qa cpu=2, memory=4Gi. k8s OOM-kills past memory, throttles past CPU. Override per-fire:

MEM_LIMIT=4Gi CPU_LIMIT=2 hivectl fire 42 --type=feature-implementation

hivectl <cmd> says "run hivectl init first"

Project-scoped commands resolve .hive/config.yaml strictly from the current directory — there's no walk-up. cd into the project root (where .hive/ lives), or run hivectl init to scaffold it.

merge refuses with a status error

hivectl merge only merges an issue at status:in-review (i.e. the in-pod review returned PASS). If the review returned CHANGES_REQUESTED, the issue is status:changes-requested and a worker must be re-fired first. Use --force only when you've verified the PR manually.

Dead Sandbox CRs piling up

Pods self-delete on exit (shutdownPolicy: Delete); the Sandbox CRs are GC'd by the hive-gc CronJob (every 10 min, age > 30 min). To clean this project's now:

hivectl gc                 # age > 30m
hivectl gc --age=0m --all  # everything for this app

Missing host CLI

Run the preflight — it reports each tool and an install hint, and marks minikube/vfkit as optional (only needed for the local cluster):

hivectl doctor