Sink log store lag panel reports misleading multi-year lag during create sink snapshot backfill fake epochs

Description

During CREATE SINK snapshot backfill, the Dev Dashboard Sink Metrics -> Log Store Lag panel can report an extremely large lag, e.g. multiple years, even when the sink is actively consuming log store data.

The panel currently computes lag as:

(max(log_store_latest_write_epoch) by (sink_id, actor_id, sink_name)
-
max(log_store_latest_read_epoch) by (sink_id, actor_id, sink_name)) / (2^16) / 1000

This assumes both read and write epochs are on the normal wall-clock epoch timeline. However, during create sink snapshot backfill, the reader may still be consuming log-store data written under fake epochs generated by CreatingStreamingJobStatus::new_fake_barrier.

Relevant code:

let prev_epoch = Epoch::from_physical_time(*prev_epoch_fake_physical_time);
*prev_epoch_fake_physical_time += 1;
let curr_epoch = Epoch::from_physical_time(*prev_epoch_fake_physical_time);

These fake epochs start from small physical times, so log_store_latest_read_epoch can be orders of magnitude smaller than a normal wall-clock epoch, while log_store_latest_write_epoch may already be on the normal epoch timeline.

As a result, the dashboard subtracts a fake epoch from a real epoch and displays a meaningless multi-year lag.

Symptoms

Observed behavior:

• log_store_latest_write_epoch is on the normal epoch scale.
• log_store_latest_read_epoch is on the fake snapshot-backfill epoch scale.
• Log Store Lag can be shown as multiple years.
• log_store_read_rows may be higher than log_store_write_rows, but the displayed epoch lag can still increase.

This is misleading because the sink may simply be draining create-sink snapshot backfill fake-epoch data, rather than lagging by years.

Expected Behavior

The Log Store Lag metric/panel should not report wall-clock lag when read and write epochs are from different timelines.

Possible expected behavior:

• Hide or mark lag as unavailable while the reader is still on fake snapshot-backfill epochs.
• Avoid the calculation when log_store_latest_read_epoch is clearly not a normal wall-clock epoch.
• Expose a separate metric/state indicating whether the log reader is consuming create sink snapshot fake epochs.
• Use a snapshot-backfill-aware lag metric instead of directly subtracting fake and real epochs.

Impact

This makes it difficult to diagnose whether a sink is actually falling behind.

In this state:

• log_store_read_rows > log_store_write_rows may indicate the sink is catching up.
• But Log Store Lag still increases because latest_read_epoch is on the fake epoch timeline.
• Operators may incorrectly conclude that the sink/log-store reader is severely lagging.

Suggested Fix

Make the dashboard/query or metric semantics snapshot-backfill aware.

One possible approach:

1. Detect fake epoch range for log_store_latest_read_epoch.
2. Avoid displaying Log Store Lag as wall-clock duration until latest_read_epoch reaches the real snapshot_epoch.
3. Optionally add another panel:
   • Log Store Read/Write Epoch Raw
   • Create Sink Snapshot Backfill Fake Epoch Progress
   • Log Store Lag (valid only for real epochs)

At minimum, update the dashboard description to warn that Log Store Lag is invalid when read epoch is still a create-sink snapshot backfill fake epoch.