Multi-Environment Deployment Implementation Guide¶

Quick Start¶

Your infrastructure is now organized into three environment tiers with automatic progression:

Development (rpi-dev-*)  →  Staging (rpi-staging-*)  →  Production (rpi-*)
     Test Here              Validate Here               Deploy Here

Environment Overview¶

Development¶

Purpose: Test configuration changes and new features
Naming: rpi-dev-XXX (e.g., rpi-dev-1883c4049f6c)
Strategy: Immediate deployment
Backups: Disabled
Health Checks: Every 5 minutes
Use Case: Active development, breaking changes acceptable

Staging¶

Purpose: Pre-production validation
Naming: rpi-staging-XXX (e.g., rpi-staging-1883c404a130)
Strategy: Validated deployment
Backups: Enabled
Health Checks: Every 5 minutes (same as dev)
Use Case: Production-like testing, stable changes

Production¶

Purpose: Customer-facing infrastructure
Naming: rpi-prod-<serial> (e.g., rpi-prod-1883c404a12f, rpi-100)
Strategy: Approval required
Backups: Enabled (30-day retention)
Health Checks: Every 10 minutes (conservative)
Use Case: Stable, tested, approved changes only

Getting Started with Environments¶

1. Add Your First Development Device¶

Edit ansible/inventory/production/hosts.yml:

development:
  hosts:
    rpi-dev-1883c4049f6c:  # Development device for testing
      local_host: 192.168.1.50
      tunnel_host: rpi-dev-1883c4049f6c-ssh.tagai.uk
      device_region: development

Then deploy:

just ansible deploy-dev rpi-dev-1883c4049f6c

2. Deploy to All Development Devices¶

just ansible deploy-dev

3. Add Staging Device¶

Edit ansible/inventory/production/hosts.yml:

staging:
  hosts:
    rpi-staging-1883c404a130:
      local_host: 192.168.1.26
      tunnel_host: rpi-staging-1883c404a130-ssh.tagai.uk
      device_region: staging
    rpi-staging-002:
      local_host: 192.168.1.27
      tunnel_host: rpi-staging-002-ssh.tagai.uk
      device_region: staging
  vars:
    environment: staging
  children:
    staging_node_red:
      hosts:
        rpi-staging-1883c404a130:
        rpi-staging-002:
    staging_cloudflared:
      hosts:
        rpi-staging-1883c404a130:
        rpi-staging-002:
    staging_mqtt:
      hosts:
        rpi-staging-1883c404a130:
        rpi-staging-002:

Deploy to staging:

just ansible deploy-staging

4. Add Production Devices¶

Edit ansible/inventory/production/hosts.yml:

production:
  hosts:
    rpi-prod-1883c404a12f:
      local_host: 192.168.1.100
      tunnel_host: rpi-prod-1883c404a12f-ssh.tagai.uk
      device_region: production
    rpi-staging-1883c404a130:
      local_host: 192.168.1.101
      tunnel_host: rpi-staging-1883c404a130-ssh.tagai.uk
      device_region: production
    rpi-dev-1883c4049f6c:
      local_host: 192.168.1.102
      tunnel_host: rpi-dev-1883c4049f6c-ssh.tagai.uk
      device_region: production
  # ... add devices 004-100 following the pattern

Production requires approval:

just ansible deploy-prod rpi-prod-1883c404a12f
# You will be prompted to confirm

Deployment Workflow¶

Typical Change Flow¶

1. Edit configuration (e.g., Node-RED settings, MQTT config)
   └─ Update docs/configuration.md

2. Deploy to Development
   └─ just ansible deploy-dev
   └─ Test thoroughly, verify no breaking changes

3. Deploy to Staging
   └─ just ansible deploy-staging
   └─ Run production-like validation
   └─ Check healthchecks, monitor logs

4. Deploy to Production
   └─ just ansible deploy-prod
   └─ Confirm with approval prompt
   └─ Monitor health checks post-deployment

Useful Commands by Scenario¶

Testing a Configuration Change¶

# 1. Deploy to dev for testing
just ansible deploy-dev rpi-dev-1883c4049f6c

# 2. Verify health
just ansible health-dev

# 3. Check specific service
just ansible shell rpi-dev-1883c4049f6c "systemctl status nodered"

# 4. If good, promote to staging
just ansible deploy-staging rpi-staging-1883c404a130

# 5. Final health check before production
just ansible health-staging

Emergency Service Restart¶

# Restart in dev (test immediately)
just ansible restart-dev

# Restart in staging (affects pre-prod)
just ansible restart-staging

# Restart in production (requires confirmation)
just ansible restart-prod

Monitor Fleet Health¶

# Dev environment status
just ansible health-dev

# Staging environment status
just ansible health-staging

# Production environment status
just ansible health-prod

# All environments at once
just ansible health

Certificate Renewal¶

# Automatically renews certificates when < 30 days to expiry
just ansible renew-certs

# Checks all environments (dev, staging, prod)

Environment-Specific Variables¶

The only per-environment variable the roles read is deploy_environment (mapped to dev/staging/prod in the Node-RED env templates). It is set on each environment group in ansible/inventory/production/hosts.yml and hosts-lite.yml:

production:
  vars:
    deploy_environment: production

Shared settings live in ansible/inventory/production/group_vars/all/ (main.yml, nodered.yml).

Role Application by Environment¶

The same roles apply to all environments, but behave differently based on environment settings:

node_red: Always version 3.1.3, but dev has debug logs, prod has strict monitoring
cloudflared: Uses same tunnel config, but staging validates connectivity, prod has failover
healthcheck: Runs with different intervals (5min dev/staging, 10min prod)

Testing the Three-Tier System¶

Quick Test (10 minutes)¶

# 1. Verify development is empty
ansible-inventory -i ansible/inventory/production/hosts.yml --graph | grep development

# 2. List staging (has rpi-staging-1883c404a130)
ansible-inventory -i ansible/inventory/production/hosts.yml --graph | grep staging

# 3. Verify production is empty (ready for your devices)
ansible-inventory -i ansible/inventory/production/hosts.yml --graph | grep production

# 4. Test health check output format (should show no hosts)
just ansible health-dev

# 5. Test staging health (will run on rpi-staging-1883c404a130 if online)
just ansible health-staging

Full Test (30 minutes)¶

Add a development device to inventory
Run: just ansible deploy-dev rpi-dev-1883c4049f6c
Run: just ansible health-dev (verify services started)
Add the same device to staging under a different name
Run: just ansible deploy-staging rpi-staging-1883c404a130
Run: just ansible health-staging (verify staging device works)

Test production deployment with approval prompt:

just ansible deploy-prod rpi-prod-1883c404a12f
# Type 'yes' when prompted

Troubleshooting¶

Devices Not Showing in Environment¶

Check inventory syntax:

ansible-inventory -i ansible/inventory/production/hosts.yml --host rpi-dev-1883c4049f6c

Deployment Hangs Waiting for Confirmation¶

This is normal for production deployments - read the prompt carefully and type exactly yes to proceed.

Health Check Shows No Hosts¶

Make sure devices are added to inventory. Empty environments are normal during initial setup.

Environment Variables Not Applied¶

Verify the device's environment group matches its name prefix: - rpi-dev-1883c4049f6c must be under development group - rpi-staging-1883c404a130 must be under staging group - rpi-prod-1883c404a12f must be under production group

Next Steps¶

Add your development devices to development group in hosts.yml
Test deployment with just ansible deploy-dev
Add staging device for pre-production validation
Configure production devices when changes are validated
Set up monitoring to watch health checks across environments

For detailed operations, see ansible-environment-workflow.md.