[Bug] Cassandra historical telemetry queries return no data after long runtime; read executor stuck with Deque full

Component

• ThingsBoard CE 4.1.0
• Cassandra as historical time-series storage
• Redis cluster as latest time-series storage

Description After the platform runs for a period of time, the application is still alive and telemetry ingestion appears normal, but historical telemetry queries return no data from Cassandra.

Restarting the ThingsBoard service restores the ability to query historical telemetry immediately.

The deployment uses Redis for latest telemetry:

• database.ts.type=cassandra
• database.ts_latest.type=redis-cluster

So the issue is not about Redis latest telemetry. The failure happens when querying historical time-series data from Cassandra, for example dashboards or WebSocket/API historical telemetry queries.

Observed behavior

• ThingsBoard process continues running.
• Latest telemetry/rule-chain related logic may still appear active.
• Historical telemetry queries return no result / cannot load data.
• Restarting ThingsBoard fixes the issue temporarily.
• Before restart, the Cassandra read executor queue is full and stuck.
• After restart, the read executor queue returns to normal.

Evidence Before restart:

[Read] Permits queueSize = [200000] ... currBuffer = [1002]

After restart:

[Read] Permits queueSize = [0] ... currBuffer = [0]

Thread dump captured during the bad state shows both read dispatcher threads sleeping at:

org.thingsboard.server.dao.util.AbstractBufferedRateExecutor.dispatch(AbstractBufferedRateExecutor.java:227)

That corresponds to the branch where curLvl > concurrencyLimit, so the dispatcher does not consume new tasks from the queue.

The application logs also contain repeated exceptions:

java.lang.IllegalStateException: Deque full
    at java.util.concurrent.LinkedBlockingDeque.add(...)
    at org.thingsboard.server.dao.util.AbstractBufferedRateExecutor.submit(...)
    at org.thingsboard.server.dao.nosql.CassandraAbstractDao.executeAsyncRead(...)
    at org.thingsboard.server.dao.timeseries.CassandraBaseTimeseriesDao.fetchPartitions(...)
    at org.thingsboard.server.dao.timeseries.CassandraBaseTimeseriesDao.getPartitionsFuture(...)
    at org.thingsboard.server.dao.timeseries.CassandraBaseTimeseriesDao.findAndAggregateAsync(...)
    at org.thingsboard.server.dao.timeseries.CassandraBaseTimeseriesDao.findAllAsync(...)

Why this looks like a bug The system can get into a state where:

• currBuffer remains above CASSANDRA_QUERY_CONCURRENT_LIMIT
• read dispatcher threads only sleep
• queued Cassandra read tasks are no longer consumed
• queue eventually reaches CASSANDRA_QUERY_BUFFER_SIZE
• new historical telemetry reads fail with Deque full
• only service restart clears the in-memory executor state

This means the application does not self-recover from the Cassandra read executor stuck/full state.

Configuration

database:
  ts:
    type: cassandra
  ts_latest:
    type: redis-cluster

cassandra:
  query:
    buffer_size: 200000
    concurrent_limit: 1000
    permit_max_wait_time: 120000
    read_timeout: 20000
    use_ts_key_value_partitioning_on_read: true

Expected behavior Historical telemetry queries should either:

• complete successfully,
• fail fast with a clear timeout/rate-limit error,
• or recover automatically after Cassandra pressure is gone.

The executor should not remain stuck until the ThingsBoard process is restarted.

Actual behavior Historical telemetry queries stop returning data. The read executor remains stuck with:

queueSize=[200000]
currBuffer=[1002]

Related issue This looks related to:

• https://github.com/thingsboard/thingsboard/issues/15063

However, in this case the user-visible symptom is not only log spam or overload. The main symptom is that historical telemetry data cannot be queried until ThingsBoard is restarted.

Suggested investigation Please check whether AbstractBufferedRateExecutor can leak or permanently retain concurrencyLevel when Cassandra read futures do not complete as expected.

Potential hardening:

• avoid curLvl <= concurrencyLimit overshooting the configured limit
• use offer instead of queue.add to avoid repeated IllegalStateException
• add a watchdog or recovery path when currBuffer > concurrent_limit for a long time
• expose clearer metrics/alerts for Cassandra read executor stuck state