#17452·langfuse

Dual-write Event Propagation Job cursor stuck for 48+ hours, no errors logged

Author: DanielRuVeCreated Sep 14, 2026Updated Sep 17, 2026
Labelsbugself-hostingfeat-ingestion

Environment:

Self-hosted, Kubernetes (EKS), Helm chart langfuse-k8s v2.0.0 Langfuse v4.27.0, migrated from v3.224.1 via chart v1→v2 → app v3→v4 (following official upgrade guides) ClickHouse 26.4.5 (operator-managed, 3 replicas), PostgreSQL 17.9 (RDS), Redis 7.0.7 (ElastiCache) Migration write mode: LANGFUSE_MIGRATION_V4_WRITE_MODE=dual, with historic backfill enabled and completed successfully (all 4 background_migrations steps show "phase": "completed")

Problem:

The dual-write "Event Propagation Job" (worker cron, logs as [DUAL WRITE], Executing Event Propagation Job every ~60s) has its cursor stuck at a fixed timestamp for over 48 hours, with zero errors or warnings logged:

[DUAL WRITE] Last processed partition: 2026-09-09 18:57:00 [DUAL WRITE] No partitions available for processing (last processed: 2026-09-09 18:57:00)

The /api/health?failIfEventPropagationStuck=true endpoint does not flag this as stuck ("stuck": false), but reports:

json "propagationDelaySeconds": 179039

(~49.7 hours)

Confirmed NOT the cause:

Source ClickHouse partitions (traces/observations, monthly-partitioned) are actively receiving new inserts — confirmed via system.parts (recent modification_time). Not a pod-local state issue — the cursor value persisted identically across a worker pod restart (new pod, same stuck timestamp). Not related to background_migrations (Postgres) — that table only tracks the one-time historic backfill steps (all completed); this is a separate, ongoing mechanism. No errors/warnings in worker logs matching propagat|dual over a 48h window.

Impact measured: 17 traces + 191 observations created after the stuck timestamp are not yet reflected in events_full/events_core.

Question for the team: what storage backend holds this job's cursor/watermark (Postgres table we're missing, Redis key, other), and is there a supported way to inspect/reset it manually without risking the migration state?