Skip to content

Multi-Environment Deployment Implementation Guide

Quick Start

Your infrastructure is now organized into three environment tiers with automatic progression:

Development (rpi-dev-*)  →  Staging (rpi-staging-*)  →  Production (rpi-*)
     Test Here              Validate Here               Deploy Here

Environment Overview

Development

  • Purpose: Test configuration changes and new features
  • Naming: rpi-dev-XXX (e.g., rpi-dev-1883c4049f6c)
  • Strategy: Immediate deployment
  • Backups: Disabled
  • Health Checks: Every 5 minutes
  • Use Case: Active development, breaking changes acceptable

Staging

  • Purpose: Pre-production validation
  • Naming: rpi-staging-XXX (e.g., rpi-staging-1883c404a130)
  • Strategy: Validated deployment
  • Backups: Enabled
  • Health Checks: Every 5 minutes (same as dev)
  • Use Case: Production-like testing, stable changes

Production

  • Purpose: Customer-facing infrastructure
  • Naming: rpi-prod-<serial> (e.g., rpi-prod-1883c404a12f, rpi-100)
  • Strategy: Approval required
  • Backups: Enabled (30-day retention)
  • Health Checks: Every 10 minutes (conservative)
  • Use Case: Stable, tested, approved changes only

Getting Started with Environments

1. Add Your First Development Device

Edit ansible/inventory/production/hosts.yml:

development:
  hosts:
    rpi-dev-1883c4049f6c:  # Development device for testing
      local_host: 192.168.1.50
      tunnel_host: rpi-dev-1883c4049f6c-ssh.tagai.uk
      device_region: development

Then deploy: 

task ansible:deploy-dev DEVICE=rpi-dev-1883c4049f6c

2. Deploy to All Development Devices

task ansible:deploy-dev

3. Add Staging Device

Edit ansible/inventory/production/hosts.yml:

staging:
  hosts:
    rpi-staging-1883c404a130:
      local_host: 192.168.1.26
      tunnel_host: rpi-staging-1883c404a130-ssh.tagai.uk
      device_region: staging
    rpi-staging-002:
      local_host: 192.168.1.27
      tunnel_host: rpi-staging-002-ssh.tagai.uk
      device_region: staging
  vars:
    environment: staging
  children:
    staging_node_red:
      hosts:
        rpi-staging-1883c404a130:
        rpi-staging-002:
    staging_cloudflared:
      hosts:
        rpi-staging-1883c404a130:
        rpi-staging-002:
    staging_mqtt:
      hosts:
        rpi-staging-1883c404a130:
        rpi-staging-002:

Deploy to staging: 

task ansible:deploy-staging

4. Add Production Devices

Edit ansible/inventory/production/hosts.yml:

production:
  hosts:
    rpi-prod-1883c404a12f:
      local_host: 192.168.1.100
      tunnel_host: rpi-prod-1883c404a12f-ssh.tagai.uk
      device_region: production
    rpi-staging-1883c404a130:
      local_host: 192.168.1.101
      tunnel_host: rpi-staging-1883c404a130-ssh.tagai.uk
      device_region: production
    rpi-dev-1883c4049f6c:
      local_host: 192.168.1.102
      tunnel_host: rpi-dev-1883c4049f6c-ssh.tagai.uk
      device_region: production
  # ... add devices 004-100 following the pattern

Production requires approval: 

task ansible:deploy-prod DEVICE=rpi-prod-1883c404a12f
# You will be prompted to confirm

Deployment Workflow

Typical Change Flow

1. Edit configuration (e.g., Node-RED settings, MQTT config)
   └─ Update docs/configuration.md

2. Deploy to Development
   └─ task ansible:deploy-dev
   └─ Test thoroughly, verify no breaking changes

3. Deploy to Staging
   └─ task ansible:deploy-staging
   └─ Run production-like validation
   └─ Check healthchecks, monitor logs

4. Deploy to Production
   └─ task ansible:deploy-prod
   └─ Confirm with approval prompt
   └─ Monitor health checks post-deployment

Useful Commands by Scenario

Testing a Configuration Change

# 1. Deploy to dev for testing
task ansible:deploy-dev DEVICE=rpi-dev-1883c4049f6c

# 2. Verify health
task ansible:health-dev

# 3. Check specific service
task ansible:shell DEVICE=rpi-dev-1883c4049f6c CMD="systemctl status nodered"

# 4. If good, promote to staging
task ansible:deploy-staging DEVICE=rpi-staging-1883c404a130

# 5. Final health check before production
task ansible:health-staging

Emergency Service Restart

# Restart in dev (test immediately)
task ansible:restart-dev

# Restart in staging (affects pre-prod)
task ansible:restart-staging

# Restart in production (requires confirmation)
task ansible:restart-prod

Monitor Fleet Health

# Dev environment status
task ansible:health-dev

# Staging environment status
task ansible:health-staging

# Production environment status
task ansible:health-prod

# All environments at once
task ansible:health

Certificate Renewal

# Automatically renews certificates when < 30 days to expiry
task ansible:renew-certs

# Checks all environments (dev, staging, prod)

Environment-Specific Variables

Each environment has unique settings in group_vars/:

Development (group_vars/development.yml)

environment: development
environment_tier: 1
deploy_strategy: immediate
backup_enabled: false
healthcheck_interval: 300  # 5 minutes
log_level: debug
monitoring_enabled: false

Staging (group_vars/staging.yml)

environment: staging
environment_tier: 2
deploy_strategy: validated
backup_enabled: true
healthcheck_interval: 300  # 5 minutes
log_level: info
monitoring_enabled: true
alert_thresholds:
  cpu: 80
  memory: 85
  disk: 90

Production (group_vars/production.yml)

environment: production
environment_tier: 3
deploy_strategy: approval_required
backup_enabled: true
backup_retention: 30  # days
healthcheck_interval: 600  # 10 minutes
log_level: info
monitoring_enabled: true
alert_thresholds:
  cpu: 75
  memory: 80
  disk: 85
critical_alerts: true
alert_channels:
  - email
  - slack

Role Application by Environment

The same roles apply to all environments, but behave differently based on environment settings:

  • node_red: Always version 3.1.3, but dev has debug logs, prod has strict monitoring
  • cloudflared: Uses same tunnel config, but staging validates connectivity, prod has failover
  • mqtt_client: Manages certificates across all environments, renewal threshold consistent
  • healthcheck: Runs with different intervals (5min dev/staging, 10min prod)

Testing the Three-Tier System

Quick Test (10 minutes)

# 1. Verify development is empty
ansible-inventory -i ansible/inventory/production/hosts.yml --graph | grep development

# 2. List staging (has rpi-staging-1883c404a130)
ansible-inventory -i ansible/inventory/production/hosts.yml --graph | grep staging

# 3. Verify production is empty (ready for your devices)
ansible-inventory -i ansible/inventory/production/hosts.yml --graph | grep production

# 4. Test health check output format (should show no hosts)
task ansible:health-dev

# 5. Test staging health (will run on rpi-staging-1883c404a130 if online)
task ansible:health-staging

Full Test (30 minutes)

  1. Add a development device to inventory
  2. Run: task ansible:deploy-dev DEVICE=rpi-dev-1883c4049f6c
  3. Run: task ansible:health-dev (verify services started)
  4. Add the same device to staging under a different name
  5. Run: task ansible:deploy-staging DEVICE=rpi-staging-1883c404a130
  6. Run: task ansible:health-staging (verify staging device works)
  7. Test production deployment with approval prompt: 
    task ansible:deploy-prod DEVICE=rpi-prod-1883c404a12f
# Type 'yes' when prompted

Troubleshooting

Devices Not Showing in Environment

Check inventory syntax: 

ansible-inventory -i ansible/inventory/production/hosts.yml --host rpi-dev-1883c4049f6c

Deployment Hangs Waiting for Confirmation

This is normal for production deployments - read the prompt carefully and type exactly yes to proceed.

Health Check Shows No Hosts

Make sure devices are added to inventory. Empty environments are normal during initial setup.

Environment Variables Not Applied

Verify the device's environment group matches its name prefix: - rpi-dev-1883c4049f6c must be under development group - rpi-staging-1883c404a130 must be under staging group - rpi-prod-1883c404a12f must be under production group

Next Steps

  1. Add your development devices to development group in hosts.yml
  2. Test deployment with task ansible:deploy-dev
  3. Add staging device for pre-production validation
  4. Configure production devices when changes are validated
  5. Set up monitoring to watch health checks across environments

For detailed operations, see ansible-environment-workflow.md.