ESP32-S3 example gate takes 33 minutes: where the time goes

Summary

The ESP32-S3 example gate (build_esp32s3.yml, now one serial job after #4414) takes 33 minutes on a 4-vCPU ubuntu-24.04 runner. Only about 11 minutes of that is compiling and linking sketch code. The rest is one silent 5-minute recompile of Arduino framework libraries that the cache should have covered, the FastLED library rebuilt from scratch in every job, and roughly 9 seconds of identical per-build overhead repeated for each of 98 examples.

The same sweep takes 34.5 minutes locally on a 16-core machine with a warm cache, which shows the cost is per-build overhead and serial structure, not CPU count.

A master run of the same job (34768611588) also exposed two defects: the fbuild daemon never became healthy, and --max-failures 3 did not stop the sweep, so 81 examples each failed after 32.6 s until the 45-minute job timeout.

Where the 33 minutes go (CI)

Run 34768603616, serial, fbuild 2.5.23, all 98 examples passed:

First example, from fbuild's elapsed-time output:

Remaining examples: median 13.7 s, AutoResearch 93 s, nothing else above 24 s.

Where a per-example build goes (local, all 97 examples)

bash compile esp32s3 all on a 16-core host, warm fbuild cache, parsed from the fbuild daemon log. The FastLED library stayed up to date for all 97 builds, so this is the steady-state per-example cost.

About 6 seconds of every build (a third) is fixed work that produces the same result for every sketch. Across 97 examples that is roughly 10 minutes.

Findings

1. Framework library cache misses on every FastLED header change (fbuild)

The ~300 s silent gap is fbuild recompiling the 42 Arduino built-in libraries (WiFi, SPI, etc.). Their cache key in crates/fbuild-build-esp/src/esp32/orchestrator/framework_library_cache.rs hashes every header under the project's src/, include/ and lib/:

fn hash_project_headers(hasher: &mut Sha256, project_dir: &Path) {
    for name in ["src", "include", "lib"] {
        hash_tree(hasher, &project_dir.join(name), true);
    }
}

FastLED is synced into lib/FastLED, so all 2,332 FastLED headers are part of the key for libraries that never include them. 171 of the last 425 master commits touched a header under src/, so most PRs pay the full recompile. It is also silent in CLI output, which makes it look like a hang.

2. Every example is a separate, complete fbuild project build

BoardCompiler._build_fbuild_sync stages each example into the same project and runs fbuild build once per example. Every build re-resolves packages, re-provisions esptool, hydrates and re-stores the core cache, links against the full SDK, and regenerates bootloader.bin and partitions.bin.

fbuild already ships compile-many (framework once, sketches in parallel), but FastLED gates it behind FASTLED_USE_FBUILD_CI, citing FastLED/fbuild#335. That issue was closed on 2026-06-02 and the gate was never re-evaluated.

3. The CI cache does not carry the expensive state

• No esp32s3 cache on master. The key is per board, and the restore key falls back to any board. The last sharded master run could not save: two shards lost the reservation race and the third failed with tar exit 1. Master runs therefore restore another board's cache (esp32c6, attiny4313, leonardo in recent runs). PR caches are branch-scoped and invisible to master. The serial job removes the race, but master still has no esp32s3 cache.
• The FastLED library is rebuilt in every job. Its objects live in .build/fbuild/esp32s3/.fbuild/build/, which is not cached. That is about 70 s per job on 4 cores.
• Compile cache location. The setup action exports ZCCACHE_DIR=$RUNNER_TEMP/zccache but only caches FBUILD_CACHE_DIR. Whether fbuild's embedded zccache writes under either path on CI is unverified; locally it uses ~/.fbuild/prod/zccache, which the CI cache does not include.

4. --max-failures is ignored on the fbuild path (FastLED)

compile_board_examples checks the failure count while iterating as_completed(futures), but BoardCompiler.build() runs every example synchronously before returning any future. The threshold can never stop a sweep.

Run 34768611588 on master: every fbuild call failed with daemon did not become healthy after 3 spawn attempts (10s each), 81 of 81 examples, 32.6 s each. With a working threshold the job would have stopped in about 2 minutes. Instead it ran into the 45-minute job timeout.

5. Intermittent fbuild daemon health failure on CI

The PR run and the master run used the same fbuild version, the same workflow and the same restored cache (esp32c6 key). The PR run passed; the master run's daemon never became healthy for the whole job. fbuild's daemon state lives under ~/.fbuild/prod, not in the cached directory, so stale cached daemon state is ruled out. Locally, cold daemon start took three spawn attempts and about 30 s before the compile backend was ready, right at the client's 3 × 10 s limit. A re-run of the failed job is in progress.

6. Timeout headroom

The serial sweep needs 31 to 33 minutes against a 45-minute job timeout. Any slowdown such as finding 1 combined with a cold cache, or finding 4, pushes it over.

Reproduce

bash compile esp32s3 all --no-interactive

Per-phase timings come from the daemon log (~/.fbuild/prod/daemon/daemon.log) between project lock acquired and build completed in. fbuild also has FBUILD_PERF_LOG=1 for phase timers, which would give the same breakdown from a CI run.

Related

• #3791 and #4165: the sharding this gate used, removed in #4414.
• FastLED/fbuild#335: the reason compile-many is still gated off.
• FastLED/fbuild#1411 / #1413: the SDK re-download fix that made serial viable.

Follow-up work on making this faster, with fbuild as the focus, will be tracked from this issue.

Generated with Claude Code

https://claude.ai/code/session_01M9rUkvoW2gVKYurtQJpi4i