#16146 Two RELATED_ENTITIES_AGGREGATION calculated fields on the same entity recalculate each other forever (deterministic reproduction, root cause in code)

Component

Rule Engine / Calculated Fields (RELATED_ENTITIES_AGGREGATION)

Environment

OS: Ubuntu 24.04 (deb package) — also seen in production on Kubernetes (AKS)
ThingsBoard: CE 4.3.1.5 (deb, monolith, TB_QUEUE_TYPE=in-memory, PostgreSQL 16, Java 21). Same behaviour in 4.3.1.3 in production. The relevant code is identical in v4.3.1.3, v4.3.1.5 and master (2026-09-10).
Reproduced via REST API only (no UI involved). Script and raw evidence attached (tb-cf-loop-repro.zip).

Related: #16027 (closed for inactivity). This issue supersedes it with a deterministic reproduction and a root-cause analysis.

Description

When an entity (or profile) has two RELATED_ENTITIES_AGGREGATION calculated fields whose outputs are consumed by another calculated field of the same entity, a single pair of input updates makes both aggregations recalculate indefinitely, each one exactly every deduplicationIntervalInSec, with no further input and frozen values, until a config save stops them. In production this produced tens of thousands of identical rows per day and drove downstream CFs at the same cadence.

Mechanism (verified with DEBUG logs and by reading the source):

Every calculation of an aggregation schedules a one-shot CalculatedFieldReevaluateMsg at +dedup (RelatedEntitiesAggregationCalculatedFieldState.performCalculation() → scheduleReevaluation()). The reevaluation recalculates if lastArgsRefreshTs > lastMetricsEvalTs (shouldRecalculate()).
RelatedEntitiesAggregationCalculatedFieldState.update() sets lastArgsRefreshTs = now unconditionally, even when the argument map is empty, and CalculatedFieldEntityMessageProcessor.processArgumentValuesUpdate() does not return when newArgValues is empty (it logs "No new argument values to process for CF." and continues into state.update()). So any telemetry delivered to the entity actor — including keys that are not arguments of the aggregation — counts as "arguments changed".
The previousCalculatedFields filter only protects a CF from its own cascade. The output of aggregation A (pushed to the CF pipeline because a CF consumes it) is delivered to aggregation B as an "empty update" and vice versa. Each one re-arms the other's deferred recalculation at every cycle.

How to reproduce in ~10 minutes (one command)

Fresh ThingsBoard CE 4.3.1.x with the demo tenant on a Linux host (e.g. docker run --rm -p 8080:8080 thingsboard/tb-postgres:4.3.1.5, or the deb package as used here).
TB_URL=http://localhost:8080 python3 repro.py all (script in the attachment, Python 3 stdlib only).

It creates everything, first verifies that the documented deferred recalculation works on the host, then ignites, observes 180 s, prints every row read back via REST with the period between rows, prints the verdict and extinguishes. Full output in the attachment (evidence/run_all_output.txt); the verdict line of that run:

RESULT: LOOP REPRODUCED - in_flow written 10 times and out_flow 9 times in 180 s with no input; in_flow periods: 20.000, 20.001, 20.001, 20.000, 20.001, 20.001, 20.000, 20.001, 20.000

Reproduced 4 times out of 4 on this instance (two of them after a full restart). If the script's environment check prints FAIL, the host drops deferred recalculations (see side finding) and the loop cannot be shown there — use a Linux host.

Manual steps (same thing without the script)

Device profile tank. Devices T1 (profile tank), M1, M2 (default profile). Relations T1 --Feeds--> M1, T1 --Drains--> M2.
On profile tank, create:
- agg_in: RELATED_ENTITIES_AGGREGATION, relation FROM Feeds, argument flow (TS_LATEST, default 0), metric in_flow = SUM(flow), deduplication 20 s, useLatestTs=false, output TIME_SERIES / IMMEDIATE / processCfs=true.
- agg_out: same, relation FROM Drains, metric out_flow, deduplication 20 s.
- net: SCRIPT, arguments in_flow, out_flow (TS_LATEST), expression return { "net_flow": out_flow - in_flow };, output TIME_SERIES / IMMEDIATE / processCfs=true. (Exact JSON of the three CFs in the attachment.)
Wait 30 s. Post {"flow": 10} to M1, wait 5 s, post {"flow": 7} to M2. Send nothing else.

Expected

in_flow=10, out_flow=7, net_flow=-3 written once (plus at most one deferred recalculation per aggregation), then silence.

Actual

Both aggregations recalculate every 20.001 s forever, values frozen (ts_kv of T1, UTC):

in_flow  13:15:54.452 | 13:16:14.454 (+20.002) | 13:16:34.456 (+20.002) | 13:16:54.457 (+20.001) | ... | 13:20:54.469 (+20.001)   value 10 throughout
out_flow 13:15:58.476 | 13:16:18.477 (+20.001) | 13:16:38.477 (+20.000) | 13:16:58.479 (+20.002) | ... | 13:20:58.491 (+20.001)   value 7 throughout
net_flow recalculated after each of them (-3 throughout)

Period = deduplication interval + ~1 ms with monotonic +1 ms/cycle drift (a re-armed one-shot timer). It ran 5+ minutes and stopped only when I saved agg_in with processCfs=false (13:21:07); the other chain died at its next reevaluation.

DEBUG log (org.thingsboard.server.actors.calculatedField, org.thingsboard.server.service.cf):

13:15:54,452 CalculatedFieldCtx  [agg_in]  Scheduling CF reevaluation in 20000 ms
13:15:54,466 ...MessageProcessor [agg_out] No new argument values to process for CF.   <- agg_in's output delivered to agg_out
13:15:58,475 CalculatedFieldCtx  [agg_out] Scheduling CF reevaluation in 20000 ms
13:15:58,484 ...MessageProcessor [agg_in]  No new argument values to process for CF.   <- agg_out's output delivered to agg_in
13:16:14,452 ...MessageProcessor [T1][agg_in]  Reevaluating CF state
13:16:14,453 CalculatedFieldCtx  [agg_in]  Scheduling CF reevaluation in 20000 ms      <- recalculated, no argument changed
13:16:14,489 ...MessageProcessor [agg_out] No new argument values to process for CF.
13:16:18,476 ...MessageProcessor [T1][agg_out] Reevaluating CF state
13:16:18,476 CalculatedFieldCtx  [agg_out] Scheduling CF reevaluation in 20000 ms
... repeats indefinitely

Controls (same instance)

Foreign key re-arms the deferred recalculation. Post flow=30 to M1 (agg_in calculates at 13:14:37.425); 8 s later post {"out_flow": 99} directly on T1 (not an argument of agg_in). Result: a second in_flow=30 row at 13:14:57.426 (+20.001 s). Log: [agg_in] No new argument values to process for CF. at 13:14:45.436, then Reevaluating CF state + Scheduling CF reevaluation at 13:14:57.
No consumer → no loop. Delete net, ignite again (11/8, 5 s): exactly one in_flow and one out_flow row. (Outputs no CF consumes are never pushed to the CF pipeline, so the sibling is never re-armed.)
Unequal intervals (20 s / 40 s), same 5 s offset: one deferred re-run each, then silence. With unequal intervals the loop needs the sibling's cascade to land inside a narrow window, which is why it is intermittent in production (remote DB, ~100–300 ms cascade latency) and rarely seen with a local DB.
Documented deduplication works: flow=21 then flow=22 8 s later → in_flow=21 immediately, in_flow=22 exactly 20.001 s later.

Root cause (file/line, v4.3.1.3 == v4.3.1.5 == master)

application/.../service/cf/ctx/state/aggregation/RelatedEntitiesAggregationCalculatedFieldState.java L136–139: update() sets lastArgsRefreshTs = now unconditionally; L185–215: scheduleReevaluation() after every calculation and shouldRecalculate() = intervalPassed && lastArgsRefreshTs > lastMetricsEvalTs.
application/.../actors/calculatedField/CalculatedFieldEntityMessageProcessor.java L434–437: missing return when newArgValues.isEmpty(), so the empty update reaches state.update() for every CF of the entity/profile (L466–467); L359: previousCalculatedFields filter only covers the CF's own chain.

Suggested fix: return early (or do not touch lastArgsRefreshTs) when the incoming message contains no argument key of the CF.

Side finding: deferred recalculation silently dropped when the timer fires ≤ the scheduled millisecond

shouldRecalculate() compares lastMetricsEvalTs <= System.currentTimeMillis() - dedup, while the reevaluation is scheduled at exactly +dedup on a ScheduledExecutorService (monotonic clock). On a Docker Desktop / WSL2 host the reevaluation was processed 19.999 s after scheduling (Scheduling ... 12:59:44,936 → Reevaluating ... 13:00:04,935), the check fails and the deferred update is lost — the documented deduplication behaviour does not work at all on that host (a second update inside the window is never applied). Scheduling at +dedup + 1 ms (or comparing with a 1 ms tolerance) would make this robust.

Attachment

tb-cf-loop-repro.zip: repro.py (REST harness: setup, ignite, post, mute, dedup, delete), judge.sh (psql query with periods), exact JSON of the three CFs, ts_kv rows from ignition to extinction, DEBUG log of the loop.

tb-cf-loop-repro.zip