#4130·arcane

Bug: Excessive network egress / outbound bandwidth since v2.11.0

Author: jostrasserCreated Sep 19, 2026Updated Sep 20, 2026
Labelstriage

Bug Description

Since upgrading to v2.11.0 (and verified persisting in the just-released v2.13.0), running the Arcane Agent on a remote host generates persistent and aggressive outbound network traffic (egress).

In a distributed deployment (Arcane Agent running on a remote cloud VM communicating with the Arcane control plane over WAN), the agent produces a continuous outbound rate of 80–230 KB/s (~20 GB egress per 24 hours), causing unexpected cloud egress billing costs (>45 GB within a week).

Identified drivers from logs and settings:

  1. Overly aggressive default automation schedules running with 6-part second-precision crons (*/30 * * * * * for Docker Client Refresh and Auto Heal).
  2. Hardcoded 2-minute API polling routines (/api/container-registries/sync, /api/backups/s3/sync, /api/git-repositories/sync).
  3. A feedback loop between GitOps sync and the continuous Filesystem Watcher: GitOps updates Compose files on disk -> Filesystem Watcher detects changes immediately -> triggers project re-sync.
  4. Job History Bloat in Job State / KV Store: Debug logs reveal that every execution of docker-client-refresh and auto-heal updates a KV record containing an unbounded/excessive history array (runs and receipts containing dozens of past executions with full timestamps, UUIDs, and attempts). When /api/environments/0/activities or /jobs endpoints are queried, this bloated JSON payload is transmitted over WAN.

Steps To Reproduce

  1. Deploy arcane_agent via Docker Compose on a remote Linux host connected to Arcane over WAN.
  2. Configure at least one GitOps sync target (e.g. Compose files).
  3. Leave automation schedules at their defaults (Docker Client Refresh: */30 * * * * *, Auto Heal: */30 * * * * *, Filesystem Watcher: Continuous).
  4. Monitor outbound bandwidth on the host network interface (/sys/class/net/<nic>/statistics/tx_bytes).
  5. Observe continuous baseline outbound bandwidth of 80 to 230 KB/s without active user interaction.

Expected Behavior

  1. Remote agents should use conservative default polling intervals to minimize WAN traffic (e.g., hourly/daily for Docker Client Refresh; 5–10 minutes for Auto Heal).
  2. The Filesystem Watcher should ignore or debounce changes written by the internal GitOps sync engine to prevent immediate cyclic re-syncing.
  3. Idle network egress between agent and central server should remain minimal (<5 KB/s).
  4. Job history in KV store / API responses should be paginated, truncated, or pruned to prevent JSON payload bloat over WAN.
  5. Transmit only delta changes or state flags instead of the full execution history array.

Actual Behavior

  1. Default schedules run tasks like Docker Client Refresh and Auto Heal every 30 seconds.
  2. The agent executes sync tasks every 2 minutes and loops between GitOps file writes and Filesystem Watcher events every 5 minutes.
  3. Network traffic on the container's bridge network (arcane_agent_default) accounts for over 96% of all host egress traffic, generating ~20 GB of outbound data per 24 hours.

Screenshots

ImageImageImage

Arcane Version

v2.13.0

Installation Method

Docker Compose (Recommended)

Environment Type

Both Local and Remote Agents

Database Type

SQLite (Default)

Operating System

Linux

Docker Version

29.8.1

Browser & Version

Chrome latest

Debug Logs

bash
### 1. Default Sub-Minute Schedules & KV Store History Bloat (from Debug Logs)
The agent executes `docker-client-refresh` and `auto-heal` every 30 seconds (`*/30 * * * * *`). On every single tick, it updates a KV entry with an unpruned execution history containing dozens of past runs and receipts, generating large payloads:

Sep 19 22:18:00.060 DBG SQL query executed [19.02ms] query="UPDATE `kv` SET `value`='{\"jobId\":\"docker-client-refresh\",\"schedule\":\"*/30 * * * * *\",\"nextRun\":\"2026-09-19T22:18:30+02:00\",\"runs\":[{\"id\":\"271a30d9...\",\"status\":\"succeeded\"}, ... dozens of historical runs ...],\"receipts\":{...}}' WHERE key = 'jobs.df03bfe23f26e657fc7a2cbc5aad713658c1aa3b17137039b9d9eed4386b1d32'"
Sep 19 22:18:00.086 DBG SQL query executed [4.57ms] query="UPDATE `kv` SET `value`='{\"jobId\":\"auto-heal\",\"schedule\":\"*/30 * * * * *\",\"nextRun\":\"2026-09-19T22:18:30+02:00\",\"runs\":[{\"jobId\":\"auto-heal\",\"status\":\"succeeded\"}, ... dozens of historical runs ...]}' WHERE key = 'jobs.1a433b118e8afafd01123e2a0519d7775a7c6dabf6157a47c15a03190c420019'"
Sep 19 22:18:00.116 DBG Docker client refresh completed
Sep 19 22:18:00.122 DBG SQL query executed query="UPDATE `activities` SET `duration_ms`=104,`latest_message`='Auto Heal: succeeded',`status`='success' WHERE id = 'd6a022f5-9f0e-5ac2-b2eb-7f6e316fb602'"

### 2. GitOps Sync <-> Continuous Filesystem Watcher Loop
Every GitOps evaluation touches Compose files on disk, which immediately wakes the continuous filesystem watcher and triggers an immediate secondary sync cycle:

Sep 19 20:03:20.990 INF Using single file sync mode syncId=1db15a73 composePath=docker/uptimekuma_gcp_compose.yml
Sep 19 20:03:21.058 INF Using single file sync mode syncId=74d8be79 composePath=docker/netbird_gcp_01_compose.yml
Sep 19 20:03:21.114 INF Using single file sync mode syncId=b6fef13e composePath=docker/cftunnel_gcp_compose.yml
Sep 19 20:03:21.146 INF Updated project files projectName=uptime_kuma_gcp
Sep 19 20:03:21.749 INF Filesystem change detected, syncing projects
Sep 19 20:03:21.795 INF Project sync completed after filesystem change

### 3. Hardcoded 2-Minute Sync Workers
The agent repeatedly queries remote registry, S3, and repository endpoints on a fixed 120-second cycle:

Sep 19 20:04:00.371 INF Incoming request method=POST path=/api/container-registries/sync duration=8.27ms status=200
Sep 19 20:04:00.413 INF Incoming request method=POST path=/api/backups/s3/sync duration=639µs status=200
Sep 19 20:04:00.463 INF Incoming request method=POST path=/api/git-repositories/sync duration=954µs status=200
Sep 19 20:04:00.503 INF Incoming request method=PUT path=/api/environments/0/templates/variables duration=540µs status=200

### 4. 5-Second UI Polling Over WAN
When the Arcane web dashboard is open, polling occurs every 5 seconds, transmitting the bloated activities state over the network:

Sep 19 20:00:39.698 INF Incoming request method=GET path=/api/environments/0/activities duration=44.81ms status=200
Sep 19 20:00:44.694 INF Incoming request method=GET path=/api/environments/0/activities duration=45.68ms status=200
Sep 19 20:00:49.785 INF Incoming request method=GET path=/api/environments/0/activities duration=141.97ms status=200
Sep 19 20:00:50.012 INF Incoming request method=GET path=/api/environments/0/dashboard duration=373.66ms status=200

### 5. Measured Network Impact & Isolation
* Baseline outbound throughput on host interface (ens4): ~230 KB/s continuous (~19.8 GB egress per 24 hours).
* Network isolation: Bridge `br-7f2b430c3e8c` (`arcane_agent_default`) accounted for 3.67 GB of 3.80 GB total VM egress (~96.5%).
* Direct verification: Executing `docker stop arcane_agent` immediately drops host outbound egress from ~230 KB/s to 0–1 KB/s.

Docker Compose Configuration

yaml
services:
  arcane-agent:
    image: ghcr.io/getarcaneapp/arcane-headless:latest
    container_name: arcane_agent
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - arcane_agent_data:/app/data
      - arcane_agent_cert:/app/certs:ro
      - /docker-data/projects:/docker-data/projects
    ports:
      - 3553:3553
    extra_hosts:
      - "arcane.strasser.internal:ip"
    environment:
      - PUID=0
      - PGID=0
      - TZ=Europe/Vienna
      - SSL_CERT_FILE=/app/certs/rootCA.pem
      - AGENT_MODE=true
      - AGENT_TOKEN=arc_removed
      - MANAGER_API_URL=https://arcane.strasser.internal:3552
      - ENVIRONMENT=production
      - ALLOW_DOWNGRADE=false
      - LOG_LEVEL=info
      - LOG_JSON=false
      - ANALYTICS_DISABLED=false
      - PROJECTS_DIRECTORY=/docker-data/projects
      - ARCANE_BACKUP_VOLUME_NAME=arcane_backup

volumes:
  arcane_agent_data:
    name: arcane_agent_data
    driver: local
  arcane_agent_cert:
    name: arcane_agent_cert
    driver: local

Environment Configuration (.env file)

bash

Additional Context

No response