#14659·onyx

Slack indexing silently skips older history after the first 900-message page

Author: shandysulenCreated Sep 11, 2026Updated Sep 14, 2026

Problem

Slack indexing can report SUCCESS after fetching only the first history page of a channel. Older messages remain unindexed.

Reproduced on September 11, 2026, with a self-hosted Docker Compose deployment using onyxdotapp/onyx-backend:latest. The running container contains the same faulty logic as current main.

Evidence

An initial historical run, with a poll range starting at the Unix epoch, reported:

  • 116 discovered channels visited.
  • 27,059 documents and 27,788 chunks indexed.
  • 1,747/1,747 processing batches completed; no recorded indexing errors.
  • 20 channels each returned 900 history messages, then zero messages on the next request.

Representative log values (channel details omitted):

First request:  len(message_batch)=900 oldest=None latest='1789151380.913839'
Second request: len(message_batch)=0 oldest='1789095238.633369' latest='1789151380.913839'

The second request searches after the newest message already retrieved, rather than fetching the older page. Processing-batch completion therefore does not prove complete history coverage.

Cause and reproduction

Slack returns channel history newest first. _get_messages consumes one page and returns whether a cursor exists, but does not retain that cursor. Checkpoint resume then:

  1. Saves message_batch[0]["ts"] (the newest message).
  2. Uses that timestamp as oldest for the next request.
  3. Receives an empty page and marks the channel complete.

To reproduce, index a channel with more than 900 top-level messages and enough human messages to avoid the bot-channel bypass. Compare the first two history requests and check whether older messages were indexed.

The timestamp changes appear to originate in #5427. They remain in the connector. This differs from the incremental thread-reply issue in #14012.

Proposed fix

Keep the original oldest boundary. Resume backward using the last (oldest) message's timestamp as latest, with exclusive boundaries. Alternatively, persist and reuse Slack's pagination cursor.

Add a regression test spanning multiple history pages and checkpoint resume, checking for missing or duplicate messages. Update progress calculations accordingly. Existing affected indexes will need historical backfill or a full re-index after the fix.