Docker Compose Advanced Topics¶

• This lab covers advanced Docker Compose features that are essential for production-ready deployments.
• Learn about profiles, secrets, configs, resource constraints, logging drivers, init containers, variable substitution, and more.
• Each topic includes practical examples and hands-on tasks.

CTRL + click to open in new window

Docker Compose Advanced Topics¶

This lab explores the advanced features of Docker Compose that transform simple multi-container setups into production-grade, secure, and efficient deployments. You’ll learn patterns used in real-world applications to manage complexity, enforce security, and optimize resource usage.

Description¶

This lab covers 9 advanced Docker Compose topics, each demonstrated with practical examples and hands-on exercises. The topics range from service profiles to variable substitution, and from security patterns to the modern include: feature.

Prerequisites¶

• Docker and Docker Compose installed (v2.20+ recommended for include: feature)
• Completion of the basic Docker Compose labs (001-003)
• Familiarity with YAML syntax
• Basic Linux command-line skills

What You’ll Learn¶

By the end of this lab, you will be able to:

• Use profiles to conditionally enable services
• Manage secrets securely in compose files
• Use configs for external configuration
• Set resource constraints (CPU/memory) per service
• Configure logging drivers for different environments
• Use init containers for startup dependencies
• Master variable substitution with defaults and validation
• Merge and override multiple compose files
• Use the modern include: feature to compose modular apps

Table of Contents¶

1. Profiles
2. Secrets Management
3. Configs
4. Resource Constraints
5. Logging Drivers
6. Init Containers
7. Variable Substitution Deep Dive
8. Merge and Override Multiple Compose Files
9. Include Feature

1. Profiles¶

Profiles allow you to define which services start based on the environment or use case. Services with matching --profile flags are included.

# docker-compose.yml
version: "3.8"
services:
  app:
    image: nginx:alpine
    profiles:
      - dev
      - staging
      - production

  db:
    image: postgres:16-alpine
    profiles:
      - dev
      - staging

  monitoring:
    image: prom/prometheus
    profiles:
      - staging
      - production
      - monitoring

  debug-tool:
    image: busybox
    profiles:
      - debug
    command: sleep 3600

Profile Behavior¶

Hands-On Task 1: Profiles¶

1. Create a docker-compose.yml with 4 services: web, db, redis, monitoring
2. Assign profiles: web → dev/production, db → dev/staging/production, redis → dev/staging, monitoring → staging/production
3. Start only the dev services: docker compose --profile dev up -d
4. Verify which services are running: docker compose ps
5. Add the monitoring profile: docker compose --profile dev --profile monitoring up -d
6. Stop and clean up: docker compose down

Expected Outcome: You can selectively start subsets of services based on profile flags.

2. Secrets Management¶

Docker Compose supports secrets for sensitive data like passwords, API keys, and TLS certificates. Secrets are mounted as files inside containers.

# docker-compose.yml
version: "3.8"
services:
  app:
    image: nginx:alpine
    secrets:
      - db_password
      - api_key
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
      API_KEY_FILE: /run/secrets/api_key

secrets:
  db_password:
    file: ./secrets/db_password.txt
  api_key:
    file: ./secrets/api_key.txt

# Create the secret files
mkdir -p ./secrets
echo "SuperSecretPassword123!" > ./secrets/db_password.txt
echo "sk-abc123def456" > ./secrets/api_key.txt

Secret Types¶

Hands-On Task 2: Secrets¶

1. Create the secrets/ directory with db_password.txt and api_key.txt
2. Write a docker-compose.yml with a postgres service that uses a secret for POSTGRES_PASSWORD
3. Verify the secret is mounted: docker compose exec postgres cat /run/secrets/db_password
4. Update the secret value and recreate: docker compose up -d
5. Clean up: docker compose down

Expected Outcome: Secrets are securely mounted as files, not exposed in environment variables.

3. Configs¶

Configs allow you to manage non-sensitive configuration files externally, similar to secrets but without encryption.

# docker-compose.yml
version: "3.8"
services:
  nginx:
    image: nginx:alpine
    configs:
      - source: nginx_conf
        target: /etc/nginx/conf.d/default.conf
      - source: app_settings
        target: /etc/nginx/app.conf

configs:
  nginx_conf:
    file: ./config/nginx.conf
  app_settings:
    file: ./config/app.json

mkdir -p ./config
cat > ./config/nginx.conf << 'EOF'
server {
    listen 80;
    server_name example.com;
    location / {
        proxy_pass http://app:3000;
    }
}
EOF

Configs vs Volumes¶

Hands-On Task 3: Configs¶

1. Create a config/ directory with nginx.conf and app.json
2. Write a docker-compose.yml with an nginx service that mounts both configs
3. Verify the configs are mounted: docker compose exec nginx cat /etc/nginx/conf.d/default.conf
4. Modify a config and recreate the service
5. Clean up

Expected Outcome: Config files are mounted into the container at specified paths.

4. Resource Constraints¶

Docker Compose allows you to limit CPU and memory usage per service using the deploy.resources section.

# docker-compose.yml
version: "3.8"
services:
  web:
    image: nginx:alpine
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: "256M"
        reservations:
          cpus: "0.25"
          memory: "128M"

  heavy-worker:
    image: alpine
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: "1G"
    command: dd if=/dev/zero of=/dev/null bs=1M count=1000

Resource Units¶

Hands-On Task 4: Resource Constraints¶

1. Write a docker-compose.yml with two services: web (limited) and stress (CPU-intensive)
2. Run docker stats in another terminal and observe the resource usage
3. Start the stress service: docker compose up -d stress
4. Observe that web stays within its 0.5 CPU limit while stress may consume more
5. Clean up

Expected Outcome: Services respect their configured CPU and memory limits.

5. Logging Drivers¶

Docker Compose supports multiple logging drivers to control how container logs are collected and stored.

# docker-compose.yml
version: "3.8"
services:
  app:
    image: nginx:alpine
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  web:
    image: nginx:alpine
    logging:
      driver: "local"
      options:
        max-size: "10m"
        max-file: "3"

# Global logging defaults
x-logging: &default-logging
  logging:
    driver: "json-file"
    options:
      max-size: "10m"
      max-file: "3"

Logging Driver Comparison¶

Hands-On Task 5: Logging Drivers¶

1. Create a docker-compose.yml with 3 nginx services using different drivers: json-file, local, and none
2. Generate some traffic: curl http://localhost:8080
3. Compare log output:
4. docker compose logs json-svc → shows formatted JSON logs
5. docker compose logs local-svc → shows logs from local driver
6. docker compose logs none-svc → no logs (driver: none)
7. Check disk usage: docker system df
8. Clean up

Expected Outcome: Different logging drivers produce different log formats and storage footprints.

6. Init Containers¶

Init containers run to completion before the main service starts, ideal for database migrations, schema setup, or waiting for dependencies.

# docker-compose.yml
version: "3.8"
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_PASSWORD: secret
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 10

  app:
    image: nginx:alpine
    depends_on:
      app-init:
        condition: service_completed_successfully
      db:
        condition: service_healthy
    ports:
      - "8080:80"

  app-init:
    image: busybox:1.36
    command: >
      sh -c "echo 'Running migrations...' && sleep 5 && echo 'Migrations complete!'"
    depends_on:
      db:
        condition: service_healthy

Init Container Patterns¶

Hands-On Task 6: Init Containers¶

1. Write a docker-compose.yml with: db (postgres with healthcheck), db-migrate (init container that depends on healthy db), and app (depends on completed migration)
2. Start: docker compose up -d
3. Verify startup order: docker compose logs db-migrate
4. Check that app only starts after db-migrate completes
5. Clean up

Expected Outcome: Services start in the correct order: db → db-migrate → app.

7. Variable Substitution Deep Dive¶

Docker Compose supports powerful variable substitution in YAML files.

# docker-compose.yml
version: "3.8"
services:
  app:
    image: ${IMAGE_NAME:-nginx:alpine}
    ports:
      - "${APP_PORT:-8080}:80"
    environment:
      - ENV_NAME=${ENV_NAME:-development}
      - DB_URL=${DB_URL:?error: DB_URL is required}
      - LOG_LEVEL=${LOG_LEVEL:-info}
      - ESCAPED_DOLLAR=$${not_a_variable}

# Variable precedence (lowest to highest):
# 1. Shell environment variables
# 2. .env file in project directory
# 3. --env-file flag
# 4. environment: section in compose file

Variable Substitution Syntax¶

.env File Resolution Order¶

Project root (.env)          ← Lowest priority
  └── --env-file FILE        ← Explicit env file
       └── Shell env vars    ← From the current shell
            └── compose file environment: section  ← Highest priority

Hands-On Task 7: Variable Substitution¶

1. Create a .env file with: APP_PORT=9090, LOG_LEVEL=debug
2. Write a docker-compose.yml using ${APP_PORT:-8080}, ${LOG_LEVEL:-info}, and ${DB_URL:?error}
3. Start without DB_URL set → should fail with error message
4. Set DB_URL=postgres://localhost:5432/mydb and retry
5. Override with shell: APP_PORT=7070 docker compose up -d
6. Verify the port: docker compose port app 80
7. Clean up

Expected Outcome: Variable substitution works with defaults, required variables, and shell overrides.

8. Merge and Override Multiple Compose Files¶

Docker Compose allows you to split configurations across multiple files and merge them.

# docker-compose.yml (base)
version: "3.8"
services:
  app:
    image: nginx:alpine
    ports:
      - "80:80"
    environment:
      - ENV=production

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: myapp

# docker-compose.override.yml (dev overrides)
services:
  app:
    ports:
      - "8080:80"
    environment:
      - ENV=development
    volumes:
      - ./src:/usr/share/nginx/html

  db:
    environment:
      POSTGRES_PASSWORD: devpassword
    ports:
      - "5432:5432"

# docker-compose.prod.yml (production additions)
services:
  app:
    restart: always
    deploy:
      resources:
        limits:
          memory: "512M"

  db:
    restart: always
    volumes:
      - db_data:/var/lib/postgresql/data

volumes:
  db_data:

Merge Rules¶

# Usage patterns
docker compose up -d                          # Uses docker-compose.yml + docker-compose.override.yml
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d   # Custom merge
docker compose -f docker-compose.yml -f docker-compose.prod.yml -p myapp up -d  # With project name

Hands-On Task 8: Merge Files¶

1. Create docker-compose.yml (base: app + db)
2. Create docker-compose.override.yml (dev: add volumes, change ports)
3. Start: docker compose up -d → automatically uses override
4. Verify: docker compose config shows merged config
5. Now start without override: docker compose -f docker-compose.yml up -d
6. Compare: docker compose ps shows different ports
7. Create docker-compose.prod.yml and merge all three: docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
8. Clean up

Expected Outcome: Multiple compose files merge together, with later files taking precedence.

9. Include Feature¶

The include: feature (Docker Compose v2.20+) allows you to compose applications from separate compose files, similar to extends: but more powerful.

# docker-compose.yml
include:
  - path: ./web/docker-compose.yml
  - path: ./api/docker-compose.yml
  - path: ./db/docker-compose.yml

services:
  proxy:
    image: nginx:alpine
    ports:
      - "80:80"
    depends_on:
      web:
        condition: service_started

# ./web/docker-compose.yml
services:
  web:
    image: nginx:alpine
    ports:
      - "8080:80"

# ./api/docker-compose.yml
services:
  api:
    image: node:20-alpine
    command: node server.js

# ./db/docker-compose.yml
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: secret

Include vs Extends¶

Hands-On Task 9: Include¶

1. Create directory structure:

├── docker-compose.yml (main, uses include)
├── web/docker-compose.yml
├── api/docker-compose.yml
└── db/docker-compose.yml

1. Each sub-compose file defines one service with basic configuration
2. The main file includes all three and adds a proxy service
3. Start: docker compose up -d
4. Verify all services are running: docker compose ps
5. Test cross-service connectivity: docker compose exec proxy curl http://web:8080
6. Clean up

Expected Outcome: Services from included files are composed together, with automatic network connectivity.

Hands-On Tasks Summary¶

Verification Checklist¶

• Profiles control which services start based on environment
• Secrets are mounted as files, not exposed in env vars
• Configs provide external configuration without encryption overhead
• Resource constraints limit CPU and memory per service
• Different logging drivers produce different output formats
• Init containers run to completion before dependent services start
• Variable substitution supports defaults and required variables
• Multiple compose files merge with the correct precedence
• Included compose files create a unified application

Additional Resources¶

• Docker Compose Profiles Documentation
• Docker Compose Secrets Documentation
• Docker Compose Configs Documentation
• Docker Compose Resource Constraints
• Docker Compose Logging Drivers
• Docker Compose Include Reference
• Docker Compose Variable Substitution

Cleanup¶

# Stop and remove all containers, networks, and volumes
docker compose down --volumes --remove-orphans

# Remove any created directories
rm -rf ./secrets ./config ./web ./api ./db