Skip to content

Architecture Alternatives — Research & Proposal

Date: 2026-07-05 Question: Is Ansible + Cloudflare Tunnels the right architecture for this fleet?

Revised 2026-07-05: two recommendations below were rejected after review — end users do need per-device Node-RED dashboards (to fine-tune parameters), and updates stay push-based Ansible with on-demand connect (no ansible-pull). See 19-NETBIRD-HEADSCALE-CLOUDFLARE-COMPARISON.md for the updated access-layer decision.

Requirements (restated)

  1. Fleet of ~100 remote Raspberry Pi devices connecting to MQTT (EMQX, mTLS).
  2. A way to connect back to devices for updates and management.
  3. Node-RED with two user levels: admins (full editor) and users (restricted, no access to all paths).

Current Architecture — Assessment

Component Current Verdict
MQTT Devices → EMQX outbound, mTLS client certs ✅ Solid, keep as-is
Remote access 1 Cloudflare Tunnel per device, 3 hostnames each (dashboard / editor / SSH) ⚠️ Works but heavy: 300 DNS records, 100 cloudflared configs, Terraform churn per device
User auth Cloudflare Access: Google IdP (users), GitHub IdP (admins) ⚠️ Free tier caps at 50 seats; beyond that $7/user/mo for all users
Node-RED "RBAC" settings.js middleware sniffs Host header to decide dashboard vs editor ❌ Not real authorization. Anyone who reaches port 1880 with the right Host header gets the full editor. Security depends 100% on Cloudflare staying in front.
Updates Ansible push over CF tunnel SSH ⚠️ Requires devices online during push; offline devices never converge; slow at 100 devices

Core insight: devices already maintain an outbound connection (MQTT). The management plane can also be outbound-only — the per-device inbound tunnel is only truly needed for interactive admin sessions (SSH, editor), not for updates or for end users.

Options Researched

Option A — Keep architecture, harden it (low effort, ~$0)

  • Replace Host-header middleware with real Node-RED adminAuth: admins get "*" permission, users get "read" or no editor account at all. Editor then requires login even if Cloudflare is bypassed. (Node-RED securing docs)
  • Switch Ansible push → ansible-pull on a systemd timer. Devices clone the repo and converge themselves; offline devices catch up when they reconnect. Keep push mode for ad-hoc debugging.
  • The git-based Node-RED project deployment already implemented (see NODERED-GIT-DEPLOYMENT-SUMMARY.md) becomes the flow-update channel — device pulls, restarts service, health-checks.

Fixes: real two-level auth, offline convergence. Keeps: tunnel sprawl, CF seat limits.

Option B — Mesh VPN for the management plane (Tailscale / Headscale / NetBird)

Replace 100 per-device tunnels with one mesh network; admins SSH / open editor over the mesh.

  • Tailscale (SaaS): unlimited user devices, but tagged resources (servers/IoT) — 50 included, then $1/mo each → ~$50–100+/mo at 100 devices. Zero coordination-server maintenance. (pricing analysis)
  • Headscale (self-hosted, free): official Tailscale clients, own coordination server. No per-device cost, but you operate the server.
  • NetBird (self-hosted, free): fully open source stack incl. dashboard and IdP integration. (comparison)

Only solves the admin access path. End users still need something for dashboards — which leads to Option C.

Option C — Centralize user-facing dashboards (architectural fix)

Users never touch device-local Node-RED at all.

  • Devices already publish state to EMQX. Build one central dashboard (central Node-RED instance or Grafana) on the VPS next to EMQX, subscribed to all device topics.
  • End users get accounts on the central dashboard only. Grafana gives real RBAC (Viewer/Editor/Admin, per-folder permissions) for free; a central Node-RED + FlowFuse Dashboard 2.0 multi-user is an alternative.
  • Device-local Node-RED editor becomes an admin-only tool — one hostname per device instead of three, or reachable only over the mesh VPN.

Eliminates: CF Access seat-count problem (users hit one app, not 100), the Host-header middleware entirely, and 2/3 of the DNS/ingress sprawl. Command/control back to devices flows through MQTT topics with EMQX ACLs.

Option D — FlowFuse (purpose-built Node-RED fleet platform)

The exact product category for this use case: Device Agent connects outbound (no tunnels needed), snapshot-based flow deployment to the whole fleet, remote editor access through the platform, audit logs, and true RBAC (viewer/member/owner roles).

  • Cost is the blocker: Cloud Pro ≈ $3,700+/mo at 100 devices ($425 base + ~$35/extra instance). Self-hosted open source lacks RBAC and team features; Enterprise self-hosted is custom-priced. (pricing)
  • Still needs a separate SSH path for OS-level work (mesh VPN or a few retained tunnels).

Right product, wrong price point for this fleet unless the business case supports it.

Option E — OS-level fleet platforms (balena / Mender)

  • balena: balenaOS + containers, full device lifecycle, per-device pricing, hard vendor lock-in to balenaOS. (overview)
  • Mender: open-source A/B OTA updates for the OS, works on Raspberry Pi OS via mender-convert; remote terminal is a paid add-on. (fleet management)

Neither addresses Node-RED user levels. Worth revisiting only if atomic OS updates with rollback become a hard requirement. (comparison)

Recommendation

Ansible + Cloudflare Tunnels is not wrong — but it is doing two jobs it shouldn't: serving end users per-device, and pushing updates. Restructure into three planes:

Plane Today Proposed
Data MQTT → EMQX (mTLS) Unchanged
Users CF Access → 100 device dashboards, Host-header middleware One central dashboard (Grafana or central Node-RED) behind CF Access. Users exist only there.
Admin/updates CF tunnel SSH + Ansible push ansible-pull + git flow deployment (self-converging). Interactive access: keep CF tunnels admin-only (1 hostname/device) or migrate to Headscale/Tailscale.

Phased migration

  1. Phase 1 (defense in depth, days): add adminAuth to device Node-RED; admins "*", no user accounts. Editor safe even without Cloudflare.
  2. Phase 2 (update plane, ~1 week): convert existing roles to ansible-pull + systemd timer; git-based flow deployment (already half-built) as the standard channel. Devices report deploy status over MQTT.
  3. Phase 3 (user plane, 1–2 weeks): stand up central dashboard on the EMQX VPS; move end users there; delete per-device dashboard hostnames + middleware from settings.js.
  4. Phase 4 (optional): replace per-device tunnels with Headscale/Tailscale for admin SSH/editor; keep 2–3 CF tunnels as break-glass.

Cost comparison (100 devices)

Option Monthly cost Effort
Current, >50 users $350+ (CF Access seats) —
Recommended (C + A) ~$0 (CF free tier: users hit 1 app; VPS already exists) 3–4 weeks phased
+ Tailscale mesh ~$50–100 +1 week
FlowFuse Cloud ~$3,700 low
balena ~$100–200+ (per-device) high (OS migration)

Sources