Docker Healthchecks Report Failure; Restart Policies Wait for Exit

A Docker healthcheck reports whether its command succeeds. It does not stop the container, replace it, or activate a restart policy by itself. Restart policies wait for the container’s main process to terminate—or for Docker’s own lifecycle to change—so an unhealthy container can remain Up indefinitely.

That separation is useful once each mechanism has one job. Health status can gate startup, drive alerts, or supply an orchestrator with evidence. Restart policy can recover a process that actually exits. Trouble begins when operators treat unhealthy, exited, and unavailable as three spellings of the same state.

Teams deciding how much lifecycle behavior belongs in configuration can first compare Docker and Docker Compose deployment tradeoffs. The format changes where the policy is declared, but it does not merge health state with main-process exit.

Read the two state surfaces before changing policy

Docker’s HEALTHCHECK reference gives a checked container an additional status: starting, healthy, or unhealthy. Normal container state still records whether PID 1 is running, paused, restarting, or exited. A probe can fail repeatedly while PID 1 continues serving some requests, waits on a dependency, deadlocks, or loops.

The distinction answers the common incident question directly: Docker leaves an unhealthy standalone container running because healthcheck failure is not container termination. Even restart: always or restart: unless-stopped has no exit event to handle.

Before restarting anything, preserve both state surfaces:

container='app'
docker inspect --format '{{json .State}}' "$container" | jq '{Status,Running,Restarting,OOMKilled,ExitCode,StartedAt,FinishedAt,Health}'
docker inspect --format '{{json .State.Health.Log}}' "$container" | jq '.[-5:] | map({Start,End,ExitCode,Output})'

Docker currently stores only the first 4096 bytes of each probe’s stdout/stderr in health status, so a health command should emit a short diagnostic rather than a full log stream. Preserve application logs separately; bounded Docker log retention keeps that evidence useful without allowing the logging layer to consume the host.

Events reveal whether the lifecycle ever changed

Health transitions generate health_status events. Container lifecycle produces separate die, restart, oom, stop, and start events. Reading both streams prevents a dashboard label from replacing the actual chronology.

docker events --since 30m --filter type=container --filter container=app --format '{{.Time}} {{.Action}} {{json .Actor.Attributes}}'

If the record shows health_status: unhealthy without die, the restart policy correctly did nothing. A later die plus restart means the main process terminated and policy took over. An oom event changes the diagnosis again: memory ownership, not healthcheck wiring, may be the primary failure.

Walk one failure from probe to possible restart

Follow the events in order instead of grouping features by similar names:

1. Docker runs the configured probe on its own schedule.
2. Probe exit status updates starting, healthy, or unhealthy and may emit health_status.
3. Docker does not signal PID 1 merely because that status became unhealthy.
4. An application, operator, or external controller may deliberately stop, kill, or replace the container under a separate policy.
5. Only after the main container stops does its restart policy evaluate the documented exit and daemon-lifecycle rules.

That sequence has an intentional gap between observation and action. It lets an operator preserve evidence, remove a replica from traffic, drain work, or decide that a shared dependency—not the container—is the real failure. Docker’s restart-policy documentation is explicit: on-failure acts when the container exits with a non-zero code. always and unless-stopped also govern stopped containers and daemon restarts, but neither converts a health-state transition into termination.

Compose dependency health is mainly a startup gate

Long-form depends_on with condition: service_healthy tells Compose to wait until a dependency’s healthcheck passes before creating the dependent service. That is valuable for a database migration or application boot sequence; it is not an always-on supervisor.

One similarly named Compose field creates extra confusion. depends_on.<service>.restart: true restarts a dependent after an explicit Compose operation updates or restarts its dependency. Current Compose service documentation excludes automated runtime restarts after a container dies, and it does not promise action on unhealthy.

Restart policy starts after a successful ten-second run

Docker begins monitoring restart policy only after a container has stayed up for at least ten seconds. That guard prevents a container that never starts successfully from spinning without limit. Manual stops also suppress policy until Docker or the container is started again. Those boundaries belong in an incident record because a missing restart may be policy semantics, not a daemon defect.

Make one probe failure mean one thing

A good probe tests the smallest condition required for this instance to serve correctly. It runs quickly, has a timeout below its interval, emits concise evidence, and does not mutate data. The probe binary must exist inside the image; assuming curl or wget is present makes many minimal images permanently unhealthy.

services:
  app:
    image: registry.example/app@sha256:replace-with-reviewed-digest
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "/usr/local/bin/healthcheck"]
      interval: 30s
      timeout: 5s
      start_period: 40s
      start_interval: 5s
      retries: 3

start_period protects legitimate initialization, while start_interval controls probe frequency during that window. A successful check during the start period ends the grace behavior; failures during the grace window do not count toward the unhealthy threshold until the period is over.

Compatibility matters for that last field. Current Compose healthcheck syntax marks start_interval as introduced in Docker Compose 2.20.2, while the Docker Engine 25.0 release notes record Engine support for the extra healthcheck interval during start_period. Verify both boundaries before deployment:

docker version --format 'server={{.Server.Version}}'
docker compose version

Omit start_interval or upgrade the incompatible component when an older parser or runtime rejects it; do not assume the intended health policy was applied.

Readiness, liveness, and user-path availability are not interchangeable

Local readiness might confirm that the process accepts a request and can reach required local state. Liveness should answer whether the instance can make progress. Public availability additionally depends on DNS, TLS, proxy routing, firewall policy, and network reachability—layers an in-container call to 127.0.0.1 cannot prove.

External monitoring therefore remains a separate contract. Private Uptime Kuma monitoring can test a user-visible endpoint without exposing the dashboard, while Docker health status retains the container-local signal.

Dependency checks can create correlated failure

Include a database, queue, or filesystem dependency only when the instance cannot serve its defined workload without it. A probe that calls every upstream service can mark an entire fleet unhealthy during one shared dependency outage. If an auto-remediator then restarts every container, the response adds load precisely when the dependency is weakest.

Resource pressure deserves the same caution. A probe timeout can reflect CPU quota rather than application deadlock. Container CPU-throttling evidence helps separate scheduling delay from logic failure before a restart erases the pattern.

Choose recovery by blast radius, not by label

Choose the action from state safety and failure ownership, not from the presence of an unhealthy label.

1. Alert only when the failure is ambiguous, stateful, or likely shared across dependencies.
2. Manual bounded restart when the process is recoverable, evidence is preserved, and one instance can leave service safely.
3. Application self-termination when the application can prove it cannot progress and exit without corrupting work; Docker restart policy then receives a real termination event.
4. Orchestrator replacement when replicas, readiness removal, disruption budgets, persistent state, and rollback are designed for replacement.

No universal order makes self-termination better than external control. A queue worker may need to finish or return leased work before exit. A stateless HTTP replica behind a load balancer may be safe to replace. A single database container may require investigation rather than an automatic bounce.

Socket-mounted remediation is host control

A container that can command /var/run/docker.sock can ask Docker to create privileged containers or mount host paths. Docker’s daemon attack-surface guidance therefore says only trusted users should control the daemon. Treat any “auto-heal” component with socket access as privileged infrastructure, not a harmless sidecar.

If automatic action is justified, require an allowlist, consecutive-failure threshold, cooldown, maximum attempts per window, dependency check, audit log, and escalation stop. Never let untrusted application input choose container names or Docker arguments.

Avoid two supervisors fighting over one container

Docker warns against combining Docker restart policies with host-level process managers because ownership conflicts. If a host service manager intentionally owns recovery, remove the competing Docker policy and document the single control path. systemd restart-limit behavior provides a useful model for attempt budgets and explicit release after repeated failure, even when Docker remains the runtime.

Run one bounded recovery drill

A restart is an action receipt, not a recovery result. Capture a baseline request, dependency status, pending work, and persistence checks before intervention. Drain traffic or stop new work when the application requires it, then use the graceful stop timeout that matches the workload.

container='app'
docker restart --time 30 "$container"
docker inspect --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}} {{.RestartCount}}' "$container"

Acceptance should cover four surfaces: the local healthcheck returns healthy, the representative application request succeeds, durable state or queued work remains correct, and the external user path is available. Watch through more than one probe interval and long enough to exceed the original failure cycle.

For operators who want direct control of a persistent single-host runtime, Voxfor lifetime VPS infrastructure provides the Linux host boundary; that control also makes health ownership, Docker-daemon security, backups, and restart budgets the operator’s responsibility.

Rollback means restoring the previous image digest or configuration and returning to the prior known-good lifecycle policy. Preserve the failed container’s logs and inspect record until the cause is explained. Do not declare success because RestartCount increased.

At the next alert, follow the event that actually happened

When the next unhealthy alert arrives, do not begin with a restart command or a configuration change. Read the health log and event chronology first. If PID 1 never exited, investigate probe scope, dependencies, resource pressure, and application progress. If a die event occurred without a restart, inspect the selected policy, manual-stop history, successful-start boundary, and daemon state. If a controller performed the action, audit its authority and retry budget.

That incident path ends with evidence rather than a generic ownership card: the observed transition, the component that acted, the user-path acceptance result, and the cause that must be fixed. Further DevOps operations guides can extend the same event-led method into deployment, monitoring, proxies, and container capacity. Health status reports evidence; lifecycle policy decides action; recovery proof belongs to the workload.