Dual-write Event Propagation Job cursor stuck for 48+ hours, no errors logged
Environment:
Self-hosted, Kubernetes (EKS), Helm chart langfuse-k8s v2.0.0 Langfuse v4.27.0, migrated from v3.224.1 via chart v1→v2 → app v3→v4 (following official upgrade guides) ClickHouse 26.4.5 (operator-managed, 3 replicas), PostgreSQL 17.9 (RDS), Redis 7.0.7 (ElastiCache) Migration write mode: LANGFUSE_MIGRATION_V4_WRITE_MODE=dual, with historic backfill enabled and completed successfully (all 4 background_migrations steps show "phase": "completed")
Problem:
The dual-write "Event Propagation Job" (worker cron, logs as [DUAL WRITE], Executing Event Propagation Job every ~60s) has its cursor stuck at a fixed timestamp for over 48 hours, with zero errors or warnings logged:
[DUAL WRITE] Last processed partition: 2026-09-09 18:57:00 [DUAL WRITE] No partitions available for processing (last processed: 2026-09-09 18:57:00)
The /api/health?failIfEventPropagationStuck=true endpoint does not flag this as stuck ("stuck": false), but reports:
json "propagationDelaySeconds": 179039
(~49.7 hours)
Confirmed NOT the cause:
Source ClickHouse partitions (traces/observations, monthly-partitioned) are actively receiving new inserts — confirmed via system.parts (recent modification_time). Not a pod-local state issue — the cursor value persisted identically across a worker pod restart (new pod, same stuck timestamp). Not related to background_migrations (Postgres) — that table only tracks the one-time historic backfill steps (all completed); this is a separate, ongoing mechanism. No errors/warnings in worker logs matching propagat|dual over a 48h window.
Impact measured: 17 traces + 191 observations created after the stuck timestamp are not yet reflected in events_full/events_core.
Question for the team: what storage backend holds this job's cursor/watermark (Postgres table we're missing, Redis key, other), and is there a supported way to inspect/reset it manually without risking the migration state?
Source: langfuse/langfuse