MQTT mTLS Implementation for Gaisro Technika Fleet¶

Summary¶

Implemented a passwordless, certificate-based authentication system for 100 Raspberry Pi devices connecting to EMQX v6 broker at mqtts.tagai.xyz using MQTT v5. Devices authenticate using mutual TLS with 2-year validity certificates, eliminating per-device password management.

What Was Implemented¶

1. Certificate Authority (CA)¶

Location: config/mqtt-ca/
Validity: 10 years
Purpose: Issues and signs device client certificates
Status: Auto-initialized on first run

2. Certificate Provisioning Script¶

File: scripts/provision-mqtt-certs.sh

Generates per-device mTLS certificates with CN=device-id (e.g., CN=rpi-prod-1883c404a12f):

# Initialize CA (one-time)
./scripts/provision-mqtt-certs.sh --init

# Generate certs for all devices (reads from Terraform)
./scripts/provision-mqtt-certs.sh --all

# Generate cert for single device
./scripts/provision-mqtt-certs.sh rpi-prod-1883c404a12f

Output per device: - config/<device>/mqtt/client.crt — Device certificate (730 days) - config/<device>/mqtt/client.key — Device private key (secret) - config/<device>/mqtt/ca.crt — CA bundle for verification

3. Provisioning Pipeline Integration¶

Extended `scripts/export-tunnel-credentials.sh`¶

Now calls provision-mqtt-certs.sh --all after tunnel credentials
Single command provision all tunnel + MQTT certs for a fleet

Updated `scripts/sync-device-configs.sh`¶

Syncs MQTT certs to remote ${remote_dir}/mqtt-certs/ via SCP
Validation: warns if certs are missing but continues sync

Typical workflow:

# 1. Export tunnel credentials + generate MQTT certs
export CLOUDFLARE_API_TOKEN=<token>
./scripts/export-tunnel-credentials.sh --all config

# 2. Sync to all devices (tunnel + MQTT certs)
./scripts/sync-device-configs.sh --all --user tagai --remote-base /opt/tagai

4. Node-RED MQTT Configuration¶

Reference file: node-red/flows-mqtt-config.json

Provides example MQTT broker node with TLS: - Broker: mqtts.tagai.xyz:8883 - Client cert: ~/.mqtt/client.crt - Client key: ~/.mqtt/client.key - CA cert: ~/.mqtt/ca.crt - Client ID: Device hostname (auto-mapped to CN in cert) - Protocol: MQTT v5 - Last Will: Publish device status (online/offline)

Integration: Merge config into Node-RED flows or use as a template. Update client ID and topics per device.

5. EMQX Configuration Guide¶

File: docs/reference/emqx-mtls-configuration.md

Comprehensive setup covering:

TLS Listener: Port 8883 with verify_peer=true, fail_if_no_peer_cert=true
Certificate Authenticator: Extract username from certificate CN
ACL Rules: Topic-based access control using ${username} placeholder
Revocation (optional): CRL support for emergency device lockout
Monitoring: Commands to check TLS status, connections, cert expiry
Troubleshooting: Diagnostics for TLS, auth, ACL, and expiration issues

Certificate Lifecycle¶

Issuance (Now, 2026)¶

Issue 2-year certificates for all 100 devices
Validity: Jan 7, 2026 – Jan 7, 2028
Distribution via provisioning sync during device setup

No Interim Rotation¶

Certificates remain valid for full 2-year term
No renewal, refresh, or rotation needed
Minimal operational overhead

Year 2 Reissuance (2028)¶

~6 months before expiry: Issue new 2-year batch
Sync to all devices in a coordinated maintenance window
Restart Node-RED MQTT connections
No changes to EMQX config if same CA is used

Revocation (Emergency Only)¶

If device is compromised: Revoke cert via CA
Publish updated CRL to EMQX
Restart listener for CRL reload (brief downtime)
EMQX will reject connections from revoked device

Security Considerations¶

Strengths¶

No passwords: Device identity is cryptographically bound to certificate
Offline validation: EMQX can validate certs without external service
Scalability: Easy management of 100 devices with same policies
MQTT v5: Full protocol support; no compatibility gaps

Key Storage¶

Private keys stored in config/<device>/mqtt/client.key (dev/test)
On devices: stored in /opt/tagai/<device>/mqtt-certs/ (ensure read-only)
In production: consider encrypted at-rest storage or key management service

CA Security¶

CA private key (config/mqtt-ca/ca.key) must be safeguarded
Store offline or in HSM for production
Backup issuance artifacts securely

File Structure¶

config/
  mqtt-ca/
    ca.key              # CA private key (keep secret)
    ca.crt              # CA certificate (distribute to devices + EMQX)
  rpi-prod-1883c404a12f/mqtt/
    client.crt          # Device certificate
    client.key          # Device private key
    ca.crt              # CA cert (copy)
  rpi-staging-1883c404a130/mqtt/
    ...

scripts/
  provision-mqtt-certs.sh  # New: Issue device certificates
  export-tunnel-credentials.sh  # Updated: Calls provision-mqtt-certs.sh
  sync-device-configs.sh  # Updated: Syncs MQTT certs

node-red/
  flows-mqtt-config.json  # Example MQTT broker TLS configuration

docs/
  05-EMQX-MTLS-CONFIGURATION.md  # EMQX TLS setup guide

Quick Start¶

Step 1: Initialize CA (if not already done)¶

cd /Users/tautcius/Projects/Tagai/gaisro-technika-infra
./scripts/provision-mqtt-certs.sh --init

Step 2: Generate Certificates for All Devices¶

export CLOUDFLARE_API_TOKEN=<your-api-token>
./scripts/export-tunnel-credentials.sh --all config
# This now automatically calls provision-mqtt-certs.sh --all

Step 3: Review Generated Certs¶

ls -la config/rpi-prod-1883c404a12f/mqtt/
# Shows: client.crt, client.key, ca.crt

# Check certificate details
openssl x509 -in config/rpi-prod-1883c404a12f/mqtt/client.crt -noout -text | grep -A 5 "Validity\|Subject"

Step 4: Sync to Devices¶

./scripts/sync-device-configs.sh --all --user pi --remote-base /opt/tagai

Step 5: Configure EMQX¶

Follow docs/reference/emqx-mtls-configuration.md
Place CA cert at /etc/emqx/certs/ca.crt
Enable TLS listener on port 8883
Enable certificate authenticator

Step 6: Update Node-RED Flows¶

Use node-red/flows-mqtt-config.json as reference
Import example MQTT broker node (adjust client ID and topics)
Test connection from a single device first

Validation¶

Test TLS on a Device¶

# From device (e.g., rpi-prod-1883c404a12f):
mosquitto_sub -h mqtts.tagai.xyz -p 8883 \
  -t "devices/rpi-prod-1883c404a12f/#" \
  --cert /opt/tagai/rpi-prod-1883c404a12f/mqtt-certs/client.crt \
  --key /opt/tagai/rpi-prod-1883c404a12f/mqtt-certs/client.key \
  --cafile /opt/tagai/rpi-prod-1883c404a12f/mqtt-certs/ca.crt

Verify EMQX Listener¶

# On EMQX broker:
emqx_ctl listeners  # Check SSL listener on 8883

emqx_ctl clients list  # List connected clients

emqx_ctl clients info rpi-prod-1883c404a12f  # Inspect connection details

Check Certificate Expiry¶

# On device:
openssl x509 -in /opt/tagai/rpi-prod-1883c404a12f/mqtt-certs/client.crt -noout -enddate
# Output: notAfter=Jan  7 08:54:15 2028 GMT

Future Enhancements¶

Automated renewal (if interval < 2 years): Implement ACME via step-ca for short-lived certs
Key rotation: Periodic private key regeneration on devices
Hardware security: Store keys in TPM or HSM on Raspberry Pi
Monitoring: Alert on cert expiry (e.g., 6 months, 1 month, 1 week)
Audit logging: Track cert issuance, revocation, and connection events

References¶

EMQX Docs: https://docs.emqx.io/en/latest/
MQTT v5 Spec: https://docs.oasis-open.org/mqtt/mqtt/v5.0/
OpenSSL: https://www.openssl.org/
mTLS Primer: https://smallstep.com/hello-mtls/

Status: ✅ Implementation complete. Ready for EMQX configuration and device testing.