Docs

Reconciliation

internal/reconciler runs every 30 seconds (main.go passes 30*time.Second). It is the difference between “SQLite says running” and “Docker has a live container.”

Do not docker stop a tenant container and expect it to stay down. If the row is running, the next ticks will treat that as a crash.

What each tick does

1. Crash detection

For each service with status = 'running' and a container_id:

• Missing container → crash
• Exited / dead → crash
• Transient inspect errors are skipped (not counted)

Crash accounting (two-step SQL, to avoid SQLite evaluation-order bugs):

• Window is 600 seconds
• First crash in a new window sets crash_count = 1
• Further crashes in the window increment
• At 5 crashes inside the window: circuit_open = 1, circuit_open_count++, circuit_retry_at set, container stopped

See Crash-loop & disk guards for backoff.

1a. Unhealthy

If Docker reports unhealthy, the reconciler stops the container (so unless-stopped can start a fresh one) and counts it as a crash. The platform does not inject a wget HEALTHCHECK — RunContainer comments say healthchecks are left to the image author because many images lack curl/wget.

1b. Auto recovery

Rows with circuit_open = 1 and circuit_retry_at <= now are reset: circuit closed, crash_count = 0, status stopped. A later tick / start path can bring them back.

2. Stale deploys

deploying longer than 10 minutes → failed.

3. Missing containers

A stored container_id that Docker cannot find (when not already handled as a running-crash) → stopped.

4. Split-brain

Containers labeled ah.service that are not in the DB are logged for GC.

Garbage collector

internal/gc, interval 5 minutes, first run delayed 2 minutes after process start.

Removes (resource age must be > 10 minutes to avoid racing deploys):

• Orphan service / database containers
• Orphan ah-db-* volumes
• Warm-pool leftovers (ah-pool- volumes, ah.type=warm-pool)
• Build directories older than 1 hour
• Dangling images
• Snapshots over --snapshot-max-per-service (default 10) or --snapshot-max-age (default 30 days)

Warm pool (environments)

Not the reconciler. environments.PoolManager refills every 60s, pool size 2 per enabled template, max 6 total. Default / node / python are pooled; go is cold-start. Ghost rows pointing at missing containers are pruned on refill.

Implications for operators