#659·flue

Configurable `CANONICAL_FLUSH_DELAY_MS`, or flush the first delta immediately (first text is always ~1s late)

Author: shoot16625Created Sep 11, 2026Updated Sep 14, 2026

Version: @flue/runtime 2.0.4 (source checked); measured on 2.0.3. Cloudflare target.

Summary

ConversationRecordWriter coalesces streamed deltas for CANONICAL_FLUSH_DELAY_MS = 1e3 before appending them to the durable stream. The constant is module-level, not exported, and not part of FlueConfig. The source comment already names the trade-off ("Lower = smoother live streaming … at the cost of more durable writes"), but applications cannot choose where they sit on it.

Observed

We measured on the Cloudflare target with a real model through @flue/sdk:

  • A 2,351-character answer arrived as 1,715 deltas packed into 15 SSE frames.
  • Frame intervals were fixed at 1,014–1,025 ms, with 94–232 deltas per frame.
  • The median delta was 1 character, so the model streams per token and the batching happens entirely in the writer.
  • message-started → first delta batch: 1,010 ms. Every response's first visible text is delayed by about 1 s, and client-side smoothing cannot recover that.

Proposal

Either of these would solve it for us:

  • Leading-edge flush: flush the first delta of a block immediately, then coalesce the rest at the current cadence. Time-to-first-text improves and the write count barely changes.
  • Make the delay configurable, for example durability.streamFlushMs or a FlueConfig field, so that latency-sensitive chat UIs can trade writes for smoothness.