TryCatchLearnBuild Real Apps
7 courses available at lowest price — 1 day remaining

Docker: depends_on vs healthchecks

Cover

Using depends_on: A basic approach

In the Microservices Udemy course we make heavy use of a docker-compose.yml file to start our services. Some of these services have a dependency on infrastructure services like Postgres, RabbitMQ or MongoDB. In the course we use the depends_on feature of Docker to ensure the order of containers starting using this approach:

yaml
services:
  auction-svc:
    image: dockerusername/auction-svc:latest
    depends_on:
      - postgres
      - rabbitmq

However, if we ran this we would find the auction-svc service is started before the RabbitMQ service is actually ready to receive connections so we would see some nasty warnings in the logs as the service is not able to make a connection to RabbitMQ as it is not yet ready to receive connections. It is not a big deal as the MassTransit service will use a retry approach and the auction-svc will eventually make the connection.

Enhancing reliability with health checks.

Docker’s healthcheck directive adds robustness by enabling you to define explicit conditions that determine when a service is considered “healthy.” A health check is a custom script or command that Docker runs periodically to verify the health of a service. Only once a service passes the health check is it marked as healthy, allowing dependent services to start safely

Let’s look at the improved version of the previous Compose setup with health checks:

yaml
services:
  postgres:
    image: postgres
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  rabbitmq:
    image: rabbitmq:3-management-alpine
    healthcheck:
      test: ["CMD-SHELL", "rabbitmqctl status"]
      interval: 10s
      timeout: 5s
      retries: 5

  auction-svc:
    image: dockerusername/auction-svc:latest
    depends_on:
      postgres:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy

In this configuration, postgres and rabbitmq each have a health check that ensures they are ready before other services that depend on them (like auction-svc) can start. The auction-svc now depends on the health of postgres and rabbitmq, rather than just their startup.

Benefits of this approach:

• Waits for readiness: The service is only marked as ready once the health check passes. This prevents race conditions where one service tries to connect to another before it’s ready.

• Customizable: You can define the check frequency, timeout, and retry mechanism to suit the needs of your services.

• Ensures operational stability: Services will not attempt to interact with dependencies until they are fully functional, reducing the chance of crashes and errors.

What to use and when?

Simple setups: In the course I just used the ‘depends_on’ for simplicity, with the knowledge that we will be introducing Kubernetes later on which does not have a depends on type feature, and the recommendation is to handle application dependancies in code rather than infrastructure.

Mission critical setups: In scenarios where service availability and readiness are crucial, especially for databases or message brokers, always use healthcheck to avoid failures due to timing issues.

Summary

While depends_on is a simple and easy-to-use directive for basic service startup control, it falls short when managing more complex services with longer or uncertain initialization times. By incorporating health checks into your Docker Compose configuration, you can significantly improve the reliability of your containerized services, ensuring that they are not only started but are fully ready to serve requests.

Complete example

Example docker-compose.yml file you can use in the course with health checks in place of depends_on:

yaml
services:
  postgres:
    image: postgres
    environment:
      - POSTGRES_PASSWORD=jiiNfamPf4eYWDfGCKXm
    ports:
      - 5432:5432
    volumes:
      - /var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  mongodb:
    image: mongo
    environment:
      - MONGO_INITDB_ROOT_USERNAME=root
      - MONGO_INITDB_ROOT_PASSWORD=jiiNfamPf4eYWDfGCKXm
    ports:
      - 27017:27017
    volumes:
      - /data/db
    healthcheck:
      test: echo 'db.runCommand("ping").ok' | mongosh autoshost:27017/test --quiet

  rabbitmq:
    image: rabbitmq:3-management-alpine
    environment:
      - RABBITMQ_DEFAULT_USER=rabbit
      - RABBITMQ_DEFAULT_PASS=jiiNfamPf4eYWDfGCKXm
    container_name: rabbitmq
    ports:
      - "5672:5672"
      - "15672:15672"
    healthcheck:
      test: ["CMD-SHELL", "rabbitmqctl status"]
      interval: 10s
      timeout: 5s
      retries: 5

  auction-svc:
    image: dockerusername/auction-svc:latest
    build: 
      context: .
      dockerfile: src/AuctionService/Dockerfile
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:80
      - ASPNETCORE_URLS=http://+:7777
      - RabbitMq__Host=rabbitmq
      - RabbitMQ__Username=rabbit
      - RabbitMQ__Password=jiiNfamPf4eYWDfGCKXm
      - ConnectionStrings__DefaultConnection=Server=postgres;User Id=postgres;Password=jiiNfamPf4eYWDfGCKXm;Database=auctions
      - IdentityServiceUrl=http://identity-svc
      - Kestrel__Endpoints__Grpc__Protocols=Http2
      - Kestrel__Endpoints__Grpc__Url=http://+:7777
      - Kestrel__Endpoints__WebApi__Protocols=Http1
      - Kestrel__Endpoints__WebApi__Url=http://+:80
    ports:
      - 7001:80
      - 7777:7777
    depends_on:
      postgres:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy

  search-svc:
    image: dockerusername/search-svc:latest
    build:
      context: .
      dockerfile: src/SearchService/Dockerfile
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:80
      - RabbitMq__Host=rabbitmq
      - RabbitMQ__Username=rabbit
      - RabbitMQ__Password=jiiNfamPf4eYWDfGCKXm
      - ConnectionStrings__MongoDbConnection=mongodb://root:jiiNfamPf4eYWDfGCKXm@mongodb
      - AuctionServiceUrl=http://auction-svc
    ports:
      - 7002:80
    depends_on:
      mongodb:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy

  identity-svc:
    image: dockerusername/identity-svc:latest
    build:
      context: .
      dockerfile: src/IdentityService/Dockerfile
    environment:
      - ASPNETCORE_ENVIRONMENT=Docker
      - ASPNETCORE_URLS=http://+:80
      - IssuerUri=http://id.carsties.local
      - ClientApp=http://app.carsties.local
      - ConnectionStrings__DefaultConnection=Server=postgres; User Id=postgres; Password=jiiNfamPf4eYWDfGCKXm; Database=identity
      - VIRTUAL_HOST=id.carsties.local
    depends_on:
      postgres:
        condition: service_healthy

  gateway-svc:
    image: dockerusername/gateway-svc:latest
    build:
      context: .
      dockerfile: src/GatewayService/Dockerfile
    environment:
      - ASPNETCORE_ENVIRONMENT=Docker
      - ASPNETCORE_URLS=http://+:80
      - ClientApp=http://app.carsties.local
      - VIRTUAL_HOST=api.carsties.local

  bid-svc:
    image: dockerusername/bid-svc:latest
    build: 
      context: .
      dockerfile: src/BiddingService/Dockerfile
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:80
      - RabbitMq__Host=rabbitmq
      - RabbitMQ__Username=rabbit
      - RabbitMQ__Password=jiiNfamPf4eYWDfGCKXm
      - ConnectionStrings__BidDbConnection=mongodb://root:jiiNfamPf4eYWDfGCKXm@mongodb
      - IdentityServiceUrl=http://identity-svc
      - GrpcAuction=http://auction-svc:7777
    ports:
      - 7003:80
    depends_on:
      mongodb:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy

  notify-svc:
    image: dockerusername/notify-svc:latest
    build: 
      context: .
      dockerfile: src/NotificationService/Dockerfile
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:80
      - RabbitMq__Host=rabbitmq
      - RabbitMQ__Username=rabbit
      - RabbitMQ__Password=jiiNfamPf4eYWDfGCKXm
    ports:
      - 7004:80
    depends_on:
      rabbitmq:
        condition: service_healthy

  web-app:
    image: dockerusername/web-app
    build: 
      context: .
      dockerfile: frontend/web-app/Dockerfile
    volumes:
      - /var/lib/web/data
    environment:
      - AUTH_SECRET="7vgUxWjehgeKTOFH2dZu0zSeKP61o9gl0b1vuHCqeMo="
      - AUTH_URL=http://app.carsties.local
      - AUTH_URL_INTERNAL=http://web-app:3000
      - API_URL=http://gateway-svc/
      - ID_URL=http://id.carsties.local
      - ID_URL_INTERNAL=http://identity-svc
      - NOTIFY_URL=http://api.carsties.local/notifications
      - VIRTUAL_HOST=app.carsties.local
      - VIRTUAL_PORT=3000

  nginx-proxy:
    image: nginxproxy/nginx-proxy
    container_name: nginx-proxy
    environment:
      - HSTS=off
    ports:
      - 80:80
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

Comments

to join the discussion.

Loading comments...