Skip to content

Node-RED over Cloudflare Tunnel — Performance Diagnosis & Fix Plan

Status: Phases 1–3 shipped 2026-07-06 (PR #2, closed issue #1). Phase 4 implemented 2026-07-10 (code written, live-verified device investigation, deploy to prod pending owner confirmation). Phase 5 not triggered — no evidence phases 1–3 are insufficient. Device tested: rpi-lite-prod-1883c404a132 (RPi CM4, Techbase, hostname techbase, cellular uplink DEFAULT_ROUTE=GPRS)

Problem

Opening the Node-RED editor via *-nodered.tagai.uk (Cloudflare Tunnel + Access) takes minutes, sometimes never finishes. Not user-side network — measured hop by hop.

Measured evidence

Hop Result
Cloudflare edge (DNS/TLS/Access) 173 ms — fine
Node-RED origin, localhost on device red.min.js 984 KB in 12 ms — fine
Site raw uplink (speed.cloudflare.com from device) 135 KB/s up / 229 KB/s down (~1–2 Mbit/s, cellular)
Same asset through tunnel 35 KB/s — only ~25 % of available uplink

Root causes (stacked)

  1. QUIC throttled by kernel UDP buffers. cloudflared runs protocol=quic; device has default net.core.rmem_max = 212992 (208 KB). Cloudflare requires ~7.5 MB (docs). Tunnel achieves only ~25 % of uplink.
  2. No compression at origin. Node-RED/Express serves editor assets raw (no Content-Encoding). Full editor ≈ 6–10 MB uncompressed over ~1 Mbit/s. cloudflared journal shows stream canceled by remote with error code 0 on /nodes — browser gave up waiting.
  3. No edge caching. Every editor open re-downloads all assets through tunnel (cf-cache-status: DYNAMIC).
  4. Flaky site DNS. Single resolver 83.181.60.97 (listed twice in resolv.conf); cloudflared logs lookup region1.v2.argotunnel.com: i/o timeout every 10–20 min + connection churn → the "sometimes never loads" cases.
  5. Minor: leftover per-request debug logging in node-red/settings.js ([ACCESS] Host headers, marked "remove after testing") — ~100 journald/SD writes per editor load.

Expected outcome after fixes

  • First editor load: ~3–5 min → ~15 s (compressed ~2 MB at ~135 KB/s)
  • Repeat loads: seconds (served from edge cache)
  • Fewer intermittent stalls (DNS fallback + buffers)

Fix plan

Phase 1 — UDP buffers for QUIC (biggest win, all devices)

ansible/roles/prerequisites/tasks/main.yml — persistent sysctl:

- name: "Increase UDP buffer sizes for cloudflared QUIC"
  become: yes
  sysctl:
    name: "{{ item }}"
    value: "7500000"
    sysctl_file: /etc/sysctl.d/99-cloudflared-quic.conf
    reload: yes
  loop:
    - net.core.rmem_max
    - net.core.wmem_max
  tags: [prerequisites, network, cloudflared]

Restart cloudflared afterwards (buffers bound at socket creation).

Phase 2 — Gzip at origin

  • ansible/roles/node_red/tasks/main.yml: npm install compression task (copy dotenv task pattern, creates: node_modules/compression).
  • node-red/settings.js:
  • load compression defensively (try/catch like dotenv block);
  • call it from both httpAdminMiddleware and httpNodeMiddleware after the read-only redirect logic;
  • delete the console.log('[ACCESS] Host headers:...') debug line.
  • Deploy: ansible-playbook playbooks/update-nodered-settings.yml -e use_tunnel=true.

Phase 3 — Cloudflare edge cache rule (Terraform)

New terraform/cloudflare-cache.tf — cloudflare_ruleset, phase http_request_cache_settings, zone tagai.uk:

  • Expression: (http.host wildcard "*-nodered.tagai.uk") and (http.request.uri.path.extension in {"js" "css" "woff2" "woff" "svg" "png" "map" "ico"})
  • Action: cache eligible, edge TTL 86400 s, respect origin ETag.
  • Safe with Access: Access check runs before cache serve; editor assets identical for all users.
  • Fallback: same rule manually via dashboard (Caching → Cache Rules).

Phase 4 — DNS resilience on device (confirmed 2026-07-10, implemented)

Investigated live on rpi-lite-prod-1883c404a132. Mechanism: NetworkManager is running but not the DNS source (no ipv4.dns set on any connection — the device has no wired uplink active, only ppp0/tun0). The real DNS manager is the vendor gprs.service (systemd unit wrapping /home/gprs/bin/gprs, which drives /usr/sbin/pppd). It reads GPRS_AUTO_DNS from /home/core/syscfg:

  • GPRS_AUTO_DNS=Y (the observed value) → uses only the single DNS server negotiated from the cellular carrier via pppd (usepeerdns-style) — this is the flaky 83.181.60.97 seen in /etc/resolv.conf, listed twice.
  • The file already had GPRS_DNS_1=8.8.8.8, GPRS_DNS_2=208.67.222.222, GPRS_DNS_3=8.8.4.4 configured and sitting unused, ignored while auto is on.

Fix: flip GPRS_AUTO_DNS=N via the same lineinfile pattern already used for DEFAULT_ROUTE in ansible/roles/prerequisites/tasks/main.yml, then restart gprs.service (only when the file actually changed, failed_when: false — same pattern as the cloudflared restart in Phase 1). No new resolver values introduced — this just stops ignoring the three already configured. Toggle: techbase_gprs_static_dns_enabled (default true).

Caveat for fleet-wide rollout: only this one device was reachable to verify (see side finding below). Other devices' GPRS_DNS_1/2/3 may be unset — confirm before/during first deploy to each; the vendor gprs script's behavior when AUTO_DNS=N but the DNS_1/2/3 keys are missing is unverified.

Phase 5 — Optional (only if still slow after 1–3)

  • codeEditor: { lib: "ace" } in settings.js — drops Monaco payload (~5 MB).
  • Test --protocol http2 in cloudflared.service.j2 on one device — cellular networks sometimes mangle UDP; TCP may beat QUIC even with fixed buffers.

Verification

H=rpi-lite-prod-1883c404a132-nodered.tagai.uk
TOKEN=$(cloudflared access token --app=https://$H)

# 1. Throughput: expect total <3 s (was 7.2 s), speed 100+ KB/s (was 35 KB/s)
curl -so /dev/null --compressed -H "cf-access-token: $TOKEN" \
  -w "code=%{http_code} total=%{time_total} speed=%{speed_download}B/s\n" https://$H/red/red.min.js

# 2+3. Origin gzip + edge cache: expect content-encoding: gzip, cf-cache-status: HIT on 2nd fetch
curl -sI --compressed -H "cf-access-token: $TOKEN" https://$H/red/red.min.js | grep -i "cf-cache-status\|content-encoding"

# 4. On device: sysctl net.core.rmem_max => 7500000; journalctl -u cloudflared clean of buffer warnings

# 5. Browser: editor full load target ~15 s first open, ~2–5 s repeat

Rollback: phases independent — remove sysctl.d file, redeploy previous settings.js, delete cache rule, revert service template.

Baseline measurements (2026-07-05, keep for before/after comparison)

/                  code=200 ttfb=0.318s total=0.318s
/red/red.min.js    code=200 total=7.197s  dl=255118B (gzip at edge)  speed=35447B/s
/red/style.min.css code=200 total=3.251s  dl=41280B                  speed=12697B/s
/nodes             code=200 total=9.672s  dl=212952B                 speed=22016B/s
localhost red.min.js: 984732B in 0.0125s (origin fine)
site uplink: 135407B/s up, 229106B/s down
sysctl: net.core.rmem_max = 212992, net.core.wmem_max = 212992
cloudflared 2026.3.0, protocol=quic, connector up since 2026-06-23

Side finding (separate issue)

Tunnels rpi-prod-1883c404a12f, rpi-dev-1883c404a12f, rpi-dev-1883c4049f6c, rpi-staging-1883c404a130 had zero connections on 2026-07-05 — devices/cloudflared offline (hostnames return HTTP 530). Needs separate investigation.