Featured image of post Self-healing Docker Compose stacks with healthchecks and autoheal

Self-healing Docker Compose stacks with healthchecks and autoheal

restart: unless-stopped is the first thing everybody adds to a Compose file, and it is the least useful part of a self-healing stack. It restarts a container when the *process* dies. It does nothing when the process is alive but the application is wedged — the classic case for self-hosted services that leak connections, lose their database link, or hang on a dead NFS mount.

Healthchecks fix that, but only if something acts on them. That is what autoheal is for.

1. Write a healthcheck that means something

A healthcheck that curls / and accepts a 500 is not a healthcheck. Aim at an endpoint that touches the real dependencies:

services:
  app:
    image: ghcr.io/example/app:1.4.2
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8080/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 90s
    depends_on:
      db:
        condition: service_healthy

Three details matter more than the command:

  • start_period gives a slow-booting app time to warm up without burning retries. Without it, a 60-second JVM start counts as three failures and the container restarts forever.
  • interval should be slower than your worst expected latency. A 2-second interval on a busy DB-backed endpoint creates its own outage.
  • The check must run inside the container. wget or curl has to exist in the image; if it does not, use a tiny shell probe instead:
      test: ["CMD-SHELL", "timeout 3 bash -c '</dev/tcp/127.0.0.1/8080' || exit 1"]

2. Gate dependencies on health, not start order

condition: service_healthy is the single biggest reliability win in Compose. Without it, depends_on only waits for the container to be *created*, and every stack with a database races on first boot — which shows up later as an application that works after a manual restart and fails after a reboot.

If a service exposes no healthcheck, depends_on falls back to start order. Set condition: service_started explicitly so future readers know it was a choice.

3. Add autoheal

Docker does not restart unhealthy containers by itself. A one-line sidecar does:

  autoheal:
    image: willfarrell/autoheal:1.2.0
    restart: unless-stopped
    environment:
      AUTOHEAL_CONTAINER_LABEL: autoheal
      AUTOHEAL_INTERVAL: 10
      AUTOHEAL_START_PERIOD: 60
      AUTOHEAL_DEFAULT_STOP_TIMEOUT: 30
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Then label the services it may restart:

    labels:
      autoheal: "true"

Using AUTOHEAL_CONTAINER_LABEL instead of all is deliberate: you do not want a sidecar restarting your database because a transient disk stall made it unhealthy for thirty seconds.

4. Prove it works

Break the app on purpose and watch the loop close:

docker compose up -d
docker inspect -f '{{.State.Health.Status}}' app
# healthy

# simulate a hang: freeze the main process
docker compose exec app sh -c 'kill -STOP 1'

watch -n5 'docker inspect -f "{{.State.Health.Status}} {{.RestartCount}}" app'
# unhealthy … then running with RestartCount 1

If RestartCount never moves, autoheal is not seeing the container. Nine times out of ten the label is missing or set on the wrong service; the tenth is a socket permission problem.

5. What healthchecks cannot do

  • They do not fix crashes in the app config. A container that restarts every 30 seconds will also restart every 30 seconds with autoheal, only noisier. Add restart: on-failure:5 for anything that can flap.
  • They are not monitoring. A healthcheck is a local, cheap probe. It will not tell you the certificate expired, the disk is full, or the service answered slowly. Ship logs and metrics somewhere you actually look.
  • They cost resources. Forty containers probing every 5 seconds is real CPU. Thirty seconds is a sane floor.

Healthchecks plus one gated dependency plus autoheal turns a stack that needs a human after every hiccup into one that mostly needs a human when something is genuinely wrong.

Built with Hugo
Theme Stack designed by Jimmy