Zero-Downtime Releases with Docker Compose on One EC2 Instance

Imagine this scenario. You have one EC2 instance. It runs seven containers: a web application, two background workers, a database, a cache, a RADIUS server, and a reverse proxy. You push a change. Your pipeline connects to the instance and runs this command:

docker compose up -d --build backend

Docker builds the image. Docker then stops the old container and starts the new one. The build takes 90 seconds. The application takes 20 more seconds to open its database connections. For almost two minutes, your users get a 502 error.

For Our WiFi Hotsopt platform, those two minutes have a direct cost. A customer at a hotel front desk pays for a data plan on a phone. If the API is down at that moment, the payment provider sends a webhook to a server that cannot answer it. The customer pays, and the customer gets no internet access.

The usual advice is to move to Amazon ECS or Amazon EKS. That advice is correct at a certain size. It was not correct for me. This article shows the method I use instead: a blue/green release on one EC2 instance, with Docker Compose, a container health check, and an atomic proxy reload.

Why I did not move to ECS

I want to be honest about this decision, because the cost is not only money.

A move to ECS adds an Application Load Balancer, an ECR repository, task definitions, and a service definition. It also adds a new failure surface that I must learn before my first outage, not during it. My platform serves one region and a small number of tenants. One t3 instance holds the full stack, and it has spare capacity.

There is also a technical constraint. My gateways dial into a WireGuard hub. That hub must run on the host, because it needs kernel and network interface access. A FreeRADIUS container answers UDP on the same box, over that tunnel. A move to Fargate would divide this stack between two networks with no advantage.

So the question became simple. Can one EC2 instance do a zero-downtime release? It can. The pattern is older than the container orchestrators, and the orchestrators did not invent it.

The four parts of the design

A blue/green release needs four things. Each one maps to a small piece of code.

1. Two identical slots. Only one slot takes traffic at a time.

2. A health check. The new slot must prove that it works.

3. A switch. One action moves all traffic from the old slot to the new slot.

4. A record of the active slot. The next release must know where to go.

Here is the flow:

  release starts
       │
       ├─ build the image, tag it as the idle slot (green)
       ├─ run the database migration one time
       ├─ start green, and wait for its health check
       │      └─ green is unhealthy? stop green, keep blue live, exit
       ├─ tell the proxy to send traffic to green   ← the switch
       ├─ write "green" to the active-slot file
       ├─ wait 30 seconds for in-flight requests
       └─ stop blue

Note the order. The proxy switches only after the health check passes. Every failure before the switch leaves the old version live.

Two slots with Compose profiles

Docker Compose supports YAML anchors and profiles. An anchor removes the duplication between the two slots. A profile keeps the idle slot out of a normal docker compose up.

x-backend-common: &backend-common
  restart: unless-stopped
  env_file: .env.prod
  stop_grace_period: 45s
  depends_on:
    postgres:
      condition: service_healthy
  healthcheck:
    test: ["CMD", "python", "-c",
      "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/internal/health/ready', timeout=2)"]
    interval: 3s
    timeout: 3s
    retries: 20
    start_period: 60s

services:
  backend_blue:
    <<: *backend-common
    image: app-backend-blue
    profiles: ["deployment"]

  backend_green:
    <<: *backend-common
    image: app-backend-green
    profiles: ["deployment"]

Two settings in this block prevent common release errors.

start_period: 60s gives the container time to start before Docker counts a failed check. Without it, a slow start looks like a broken release.

stop_grace_period: 45s sets the time between the graceful stop signal and the forced stop signal. My web server has a 30-second graceful timeout. The grace period must be longer than that timeout. If it is shorter, Docker stops a process by force during a request.

The health check must query the database

A health check that only proves that the process is alive is not sufficient. My readiness endpoint makes one query:

@require_GET
@never_cache
def ready(request):
    """The process is ready for traffic only when its database is reachable."""
    try:
        with connection.cursor() as cursor:
            cursor.execute("SELECT 1")
            cursor.fetchone()
    except Exception:
        return JsonResponse({"status": "unavailable"}, status=503)
    return JsonResponse({"status": "ok"})

This endpoint answers the only question the release script asks: can this container serve a real request? A liveness probe cannot answer that question. I keep both endpoints, and I use only the readiness endpoint for the cutover.

Migrations are a release action, not a start command

Many teams put the database migration in the container start command. Do not do this in a blue/green release.

During the release, the old version and the new version are both live. If the new container changes the schema as it starts, the old container can fail immediately. The migration must run one time, as its own step:

docker compose run --rm --no-deps backend_green python manage.py migrate --noinput

This rule has a consequence. Every schema change must use the expand/contract method. First you add the new column, and you keep the old column. You release the code that writes to both. Only in a later release do you remove the old column. The old version must continue to operate against the new schema for the length of the cutover.

The switch

My reverse proxy is Caddy, in a container. The configuration file names the upstream by its Compose service name. A template holds the name backend:8000, and one command writes the file for the correct slot:

sed "s/backend:8000/${upstream}:8000/g" caddy/Caddyfile.template > caddy/Caddyfile.next

The next step is the part I recommend most. The script does not send the new file to the live proxy. It first validates the file in a disposable container:

docker compose run --rm --no-deps caddy \
  caddy adapt --config /etc/caddy/Caddyfile.next \
  --adapter caddyfile --pretty --validate > caddy/Caddyfile.next.json

A bad configuration file now fails in a container that serves no traffic. Only a valid file becomes the live file:

mv -f caddy/Caddyfile.next caddy/Caddyfile
docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile.json --force

mv on the same filesystem is atomic. No process can read a file that is half written. The proxy reload is also atomic: the proxy loads the new configuration into memory, and it applies the change in one action. If the load fails, the previous configuration continues to serve traffic. The script then puts the previous file back on disk, so a later restart cannot start the rejected configuration.

The failure paths

A release with no error is the easy part. These are the failures that the script must also handle.

The new slot never becomes healthy. An error trap stops the new slot and prints its last 200 log lines. The old slot never stopped, so there is no outage.

The proxy rejects the new configuration. The reload fails, the old configuration stays live, and the script exits. The trap then stops the new slot.

The disk write of the active-slot file fails. This one is easy to miss. At this point the proxy already sends traffic to the new slot. If the script exits here, the trap stops the slot that now serves all traffic. So the script switches the proxy back to the old slot first:

if ! { printf '%s\n' "$target" > .backend-active.next && mv -f .backend-active.next .backend-active; }; then
  ./scripts/reload-caddy.sh "$active" || true
  exit 1
fi

Requests arrive during the switch. The proxy accepts a request one microsecond before the reload. The old container must answer it. This is why the script waits 30 seconds before it stops the old slot.

What this method does not give you

I must be clear about the limits of this method. Do not use it in a system that has different requirements.

This method gives you zero-downtime releases. It does not give you high availability. The instance is one point of failure. A hardware event, a full disk, or a kernel panic stops everything.

It does not give you horizontal scale. When one instance is not sufficient, you need a load balancer and more than one host. At that point the orchestrator is worth its complexity, and ECS is the shorter path.

This method also keeps state on the instance. My database runs in a container with a volume, so my backup procedure, not my instance, is my recovery plan.

Use this method when one instance holds your load, and your releases still cause downtime. Move to ECS when one instance is no longer sufficient. Do not move earlier only because the release procedure is the problem. The release procedure has a smaller answer.