Skip to content

Runbooks

1. Service health check

curl -sS http://127.0.0.1:8000/health
curl -sS http://127.0.0.1:8000/health/ready
curl -sS http://127.0.0.1:8010/health
docker compose -f docker/docker-compose.yml ps

2. Re-run migrations safely

docker compose -f docker/docker-compose.yml run --rm api bash scripts/migrate.sh

Migrations are additive and idempotent (IF NOT EXISTS patterns).

3. Outbox/DLQ operational check

  • Open admin panel dashboard.
  • Verify:
  • pending_outbox is not growing unexpectedly.
  • dlq_depth is near zero.
  • Use /api/outbox/topics and /api/dlq for machine-readable checks.

4. Worker crash loop debugging

docker compose -f docker/docker-compose.yml logs --tail=200 worker_normalizer_polygon
docker compose -f docker/docker-compose.yml logs --tail=200 worker_normalizer_ton
docker compose -f docker/docker-compose.yml logs --tail=200 worker_state
docker compose -f docker/docker-compose.yml logs --tail=200 worker_projection
docker compose -f docker/docker-compose.yml logs --tail=200 worker_outbox

5. Main site/admin isolation check

For staging/prod ingress:

  • Main portal should stay on its normal public endpoint.
  • Admin panel must be routed to a separate external endpoint/port.
  • Admin endpoint must be protected with basic auth at minimum.

Suggested smoke test:

# main portal
curl -I https://ft.farm/

# admin endpoint (example :8443)
curl -I https://ft.farm:8443/
curl -u "<user>:<pass>" -I https://ft.farm:8443/