Baike.dev
Log in
Back to tool/Back to issues
#402·pentagi

[Bug]: langfuse-clickhouse — "Too many open files" (errno 24) on queries, get_mempolicy log flood, and MergeMutationsExecutor threads spinning at ~100% CPU with no pending mutations

Author: psyrayCreated Sep 1, 2026Updated Sep 8, 2026
Labelsbug

Affected Component

  • Analytics Platform Integration (Langfuse)

Describe the bug

The bundled langfuse-clickhouse container degrades after running for a while under sustained trace ingestion and eventually makes the Langfuse stack unusable while slowing down the whole host. Observed on two independent deployments (16-core dev host and 4-vCPU production host). Three interrelated symptoms — plus the root cause, which we identified with live diagnostics and confirmed by fixing it:

1. Queries fail with Too many open files (errno 24)

The ClickHouse process runs with the Docker default nofile limit of 4096. Under load, queries exhaust it and fail with errno 24 on data part / skip-index files. The Langfuse trace-list endpoint still works, but the trace-detail endpoint returns HTTP 500:

{"message":"Internal Server Error","error":"Cannot open file /var/lib/clickhouse/store/61f/61fced17-be34-4b17-b3eb-d044cdca7e4b/202609_407_407_0/skp_idx_idx_project_dataset_run.idx: , errno: 24, strerror: Too many open files. "}

This silently breaks any automation that reads trace details (e.g. per-flow usage metrics come back empty).

2. MergeMutationsExecutor threads at ~100% CPU — root cause: infinite retry loop of a failing merge (not an empty-work spin)

Correction of our initial reading: the threads are NOT spinning "with no pending mutations". top -H shows two MergeMutationsExecutor threads at ~100% CPU while system.mutations WHERE NOT is_done returns zero rows — because the loop is a merge, not a mutation. Live diagnosis on 24.10.2.80:

  • The two executor threads replay the same merge task on system.trace_log (e.g. {41475ecd-5255-44bc-b861-4dcdad1149bb::202608_1_13050_273}, covering ~13k historic parts) which fails on every attempt while writing the output part:

    Exception while executing background task {41475ecd-...::202608_1_13050_273}:
Code: 76. DB::ErrnoException: Cannot open file /var/lib/clickhouse/store/414/41475ecd-.../tmp_merge_202608_1_13050_273/CurrentMetric_IOUringPendingEvents.sparse.idx.cmrk2: , errno: 24, strerror: Too many open files. (CANNOT_OPEN_FILE)

  • 825 failures in 5 minutes (~2.75 retries/s), each attempt re-reading the 11 source parts → 52,339 MergeMutate log lines in 10 minutes in system.text_log (~87 lines/s), which explains the 200% CPU (400% on the 16-core prod host) and the constant background I/O churn.

  • Why EMFILE with only ~130 FDs open at idle: the trace_log Wide-part merge writer opens all output streams upfront. ProfileEvents/CurrentMetrics are Map columns that expand to hundreds of sparse-serialized subcolumns (.bin + .sparse.idx/.cmrk2 per key) plus skip indexes — several thousand files at once, above the 4096 limit.

  • Leftover delete_tmp_tmp_merge_* directories are cleaned at startup — further evidence of repeated failed merge attempts.

  • docker restart clears the EMFILE condition temporarily, but the persisted part set re-schedules the same merge → the loop returns within ~25 minutes.

3. Container log flooded with get_mempolicy: Operation not permitted — noise correlated with executor activity, not an independent problem

  • The message is written directly to stderr (it never appears in system.text_log), ~4 lines/min while background executors are churning. On an idle healthy server it disappears entirely (0 lines in 3 min after the fix below, vs 99.8% of all container log lines during the incident).
  • Adding cap_add: [SYS_NICE] puts the capability in the bounding set only: with the compose's user: "101:101" the server process has CapEff = 0 (verified via /proc/1/status and a scratch container), so the EPERM persists. In rootless Docker, CAP_SYS_NICE is not obtainable at all. The spam is harmless at that rate and vanishes once the executor loop is fixed.

4. system.trace_log grows unbounded — 8.08 GB after ~3 days on a low-traffic box (monthly partitions, no TTL in the default config). This is also what keeps scheduling the huge, EMFILE-prone merges of symptom 2.

Failure chain:

nofile = 4096 (Docker default)
  → trace_log big merges cannot open their output streams (thousands of sparse subcolumns)
    → merge fails with errno 24 → hot-retried ~2.75x/s, forever, at ~100% CPU per executor thread
      → FD pressure + debug-log flood break foreground queries (trace-detail 500)
unbounded trace_log (13k historic parts)
  → giant merges re-scheduled permanently → the EMFILE-failing merge never goes away

Actual behavior: trace-detail API returns 500 (errno 24), container log is a single repeating error line, clickhouse pins 2–4 CPU cores doing no useful work, host slows down, Langfuse stack ends up disabled.

Expected behavior: the clickhouse container should run with sane FD limits out of the box (as already done for the observability clickstore in docker-compose-observability.yml), system tables should have bounded retention, idle CPU should stay below ~10%, and background executors should back off when a task repeatedly fails.

Steps to Reproduce

  1. Deploy PentAGI with the bundled Langfuse analytics stack (docker-compose-langfuse.yml; langfuse/langfuse:3 = 3.225.5, clickhouse/clickhouse-server:24 = 24.10.2.80) with default settings — no ulimits, no extra capabilities on the clickhouse service
  2. Enable tracing in PentAGI (OTLP export to Langfuse) and run a normal workload — in our case ~1–3k spans/day from automated pentest benchmark flows
  3. After ~1 day: docker stats shows the clickhouse container at ~200% CPU (400% on the 16-core prod host) even at idle
  4. Find the failing merge and its error rate:
    bash
    docker exec langfuse-clickhouse clickhouse-client --query \
  "SELECT count() FROM system.text_log WHERE message LIKE '%Exception while executing background task%' AND event_time > now() - 300"
# → ~825 in 5 min
docker exec langfuse-clickhouse clickhouse-client --query \
  "SELECT substring(message, 1, 220) FROM system.text_log WHERE message LIKE '%Exception while executing background task%' ORDER BY event_time DESC LIMIT 1"
# → Code: 76 ... errno: 24 ... tmp_merge_.../...sparse.idx.cmrk2
  5. docker logs langfuse-clickhouse --since 5m | grep -c get_mempolicy → hundreds of lines while executors churn
  6. docker exec langfuse-clickhouse sh -c 'ulimit -n; ls /proc/1/fd | wc -l' → limit 4096, low FD usage between retries (the merge opens its thousands of streams transiently)
  7. Call the Langfuse trace-detail API → 500 with errno 24 on a skip-index file of a data part
  8. docker restart langfuse-clickhouse → EMFILE cleared, but the CPU loop returns within ~25 minutes

System Configuration

  • PentAGI Version: main; Langfuse stack from the official Langfuse v3 compose: langfuse/langfuse:3 (3.225.5), clickhouse/clickhouse-server:24 (24.10.2.80)
  • Deployment Type:
    • Docker Compose
  • Environment:
    • Docker Version: 28.3.2
    • Docker Compose Version: 2.38.2
    • Host OS: Linux (Lightwhale 3.0.5)
    • Available Resources:
      • RAM: 15.6 GB
      • CPU: 16 cores
      • Disk Space: btrfs data volume, plenty free
  • Enabled Features:
    • Langfuse Analytics
    • Grafana Monitoring
    • Custom LLM Server
  • Active Integrations:
    • LLM Provider: Custom (Ollama Cloud via OpenAI-compatible endpoint, routed by LiteLLM)
    • Search Systems: SearxNG

Logs and Artifacts

Before (incident active, dev 16-core host):

# CPU at idle:
$ docker stats --no-stream --format '{{.Name}} {{.CPUPerc}}' | grep clickhouse
langfuse-clickhouse   199.88%      # 139-200% during the incident; 400% on the 16-core prod host

# Threads:
$ docker exec langfuse-clickhouse top -b -n1 -H | head -6
    PID USER      %CPU  TIME+     COMMAND
    642 clickho+  99.9  7:49.22   MergeMu...     <- MergeMutationsExecutor
    640 clickho+  99.9  7:59.20   MergeMu...

# The retry loop (system.text_log, 10 min):
$ ... SELECT thread_name, count() FROM system.text_log WHERE event_time > now() - 600 GROUP BY thread_name ORDER BY 2 DESC LIMIT 3
MergeMutate        52339
BgSchPool          10172
SystemLogFlush      2433

# Full exception (system.text_log):
Exception while executing background task {41475ecd-...::202608_1_13050_273}:
Code: 76. DB::ErrnoException: Cannot open file /var/lib/clickhouse/store/414/41475ecd-.../
tmp_merge_202608_1_13050_273/CurrentMetric_IOUringPendingEvents.sparse.idx.cmrk2: ,
errno: 24, strerror: Too many open files. (CANNOT_OPEN_FILE)
  ... MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() -> WriteBufferFromFile -> DiskLocal::writeFile

# Trace detail API (500):
{"message":"Internal Server Error","error":"Cannot open file /var/lib/clickhouse/store/61f/.../202609_407_407_0/skp_idx_idx_project_dataset_run.idx: , errno: 24, strerror: Too many open files. "}

# Log flood over 24h (correlated with executor churn):
$ docker logs langfuse-clickhouse --since 24h 2>&1 | wc -l             -> 5774
$ docker logs langfuse-clickhouse --since 24h 2>&1 | grep -c get_mempolicy -> 5762   (99.8%)

# FD situation (between retries):
$ docker exec langfuse-clickhouse sh -c 'ulimit -n; ls /proc/1/fd | wc -l'
4096
132

# Unbounded system logs:
system.trace_log: 7.66 GiB (13 parts, monthly partitions, no TTL)

After applying the fixes below (same box, ~30 min later):

$ docker stats --no-stream --format '{{.Name}} {{.CPUPerc}}' | grep clickhouse
langfuse-clickhouse   5.75%        # no MergeMutationsExecutor thread in top -H anymore

$ docker exec langfuse-clickhouse sh -c 'grep "open files" /proc/1/limits'
Max open files            262144               262144

$ ... SELECT count() FROM system.text_log WHERE message LIKE '%errno: 24%' AND event_time >= '<restart time>'
0

$ docker logs langfuse-clickhouse --since 3m 2>&1 | grep -c get_mempolicy
0

system.trace_log (fresh table): ~0.1 GB, TTL in DDL (event_date + toIntervalDay(3))
legacy system.trace_log_0 (renamed by the server at startup): 8.34 GB, TTL added manually, drains within 3 days

Proposed fixes — APPLIED & VALIDATED (PR ready, happy to send)

  1. docker-compose-langfuse.yml: add ulimits to the clickhouse service — ulimits: { nofile: { soft: 262144, hard: 262144 } } (ClickHouse docs recommend ≥ 262k; Docker's 4096 default is the direct cause of the errno-24 failures). This is the root-cause fix: with 262k FDs the trace_log merge can open its streams and completes; the retry loop and the query EMFILE failures both disappear. Note it also works in rootless Docker (verified) and is already done for clickstore in docker-compose-observability.yml.
  2. Ship a config.d file bounding system logs — new langfuse/clickhouse/system-logs.xml bind-mounted to /etc/clickhouse-server/config.d/system-logs.xml:ro, with TTLs (trace_log/text_log 3 days, metric_log/query_log/part_log/processors_profile_log/asynchronous_metric_log 7 days) and <logger><level>warning</level></logger> to cut the 87 lines/s debug flood. Caveats found during validation on a pre-existing volume: at startup 24.10.2 renames the existing system log tables to <name>_0 (old data preserved but frozen, without TTL) and creates fresh tables from the config — and it even created system.query_log/system.processors_profile_log without TTL (quirk), so those two needed a one-shot ALTER TABLE ... MODIFY TTL. The legacy <name>_0 tables must also get a TTL (or be dropped) or the old data stays on disk forever (8.34 GB of trace_log_0 in our case) — use ALTER TABLE system.<name>_0 MODIFY TTL event_date + INTERVAL N DAY DELETE SETTINGS materialize_ttl_after_modify = 0 (metadata-only; without that setting the ALTER synchronously runs a full MATERIALIZE TTL rewrite that pinned ~130% CPU for minutes on the multi-GB table).
  3. cap_add: [SYS_NICE] on the clickhouse service — keep it (documented ClickHouse-on-docker requirement, harmless), but be aware it only reaches the bounding set when the service runs as user: "101:101" (CapEff stays 0) and is ineffective in rootless Docker; it does not silence get_mempolicy by itself. The spam disappears with fix 1+2 because it is produced by the executor churn.
  4. No ClickHouse upgrade needed for the CPU spin — it is fully explained by 1+2 (EMFILE-failing merge hot-retried forever). Optional follow-up worth reporting to ClickHouse upstream: a background merge that repeatedly fails (e.g. EMFILE) is retried ~2.75×/s with no backoff, pinning an executor thread at 100% CPU while masking the real error from system.merges/system.mutations.

Screenshots or Recordings

No response

Verification

  • I have checked that this issue hasn't been already reported
  • I have provided all relevant configuration files (with sensitive data removed)
  • I have included relevant logs and error messages
  • I am running the latest version of PentAGI

Source: vxcontrol/pentagi

View original on GitHubView discussion on GitHub