Blueprint: ideas for further reducing CI run times

Blueprint

Related: https://github.com/osquery/osquery/pull/8865

I wanted to discuss a few ideas on how the osquery CI for PRs could maybe go faster.

1. Do we need to rebuild all three Linux build types in every PR? Debug + Release should suffice; RelWithDebInfo could be only in the nightly scheduled build?
2. Could we cache the build of third-party formulas (openssl)? Even with ccache, each job re-downloads the 54 MB OpenSSL tarball and re-runs its configure/build. Caching installed_formulas/ keyed on the formula version would skip that.
3. Could we move to an external cache backend, i.e. using sccache with S3/GCS/Azure (osquery already uses sccache on Windows). Because it looks like ccache is constrained by cache size (see below, and discussion in #8865).

• Could someone help me confirm whether ccache save and restore keys are actually matching up, in practice, after #8865? Because I'm seeing some 0% hits on some jobs (see below).

Analysis of ccache performance

We are using the default 5.0GB cache size for ccache, but at this size, it is throwing out the cache sometimes, dropping the cache hit percentage. If you open the raw logs for a recent build job, and search for the ccache -s output in hits and misses, it's around 38%. The cache is capacity-bound and evicting: look for a line like Cleanups: 3.

2026-09-09T14:27:53.9170674Z Summary:
2026-09-09T14:27:53.9173148Z   Hits:            1821 / 4756 (38.29 %)
2026-09-09T14:27:53.9175845Z     Direct:        1821 / 4756 (38.29 %)
2026-09-09T14:27:53.9178297Z     Preprocessed:     0 /    0
2026-09-09T14:27:53.9181459Z   Misses:          2935
2026-09-09T14:27:53.9185994Z     Direct:        2935
2026-09-09T14:27:53.9189645Z     Preprocessed:     0
2026-09-09T14:27:53.9192729Z Primary storage:
2026-09-09T14:27:53.9194998Z   Hits:            5954 / 6577 (90.53 %)
2026-09-09T14:27:53.9197809Z   Misses:           623
2026-09-09T14:27:53.9202328Z   Cache size (GB): 4.12 / 5.00 (82.44 %)
2026-09-09T14:27:53.9206021Z   Cleanups:           3

As mentioned in #8865 there's a compounding constraint above ccache: GitHub's cache quota is 10 GB per repo, LRU-evicted, shared across every cache key. The osquery CI workflow saves a separate ccache per OS, build type, and arch. So there's roughly 14 keys just across the macOS and Linux jobs (Windows is not using ccache). This exceeds 10 GB, so the keys evict each other at the GitHub layer too. This is why I asked whether we could use a remote cache (maybe it's cost prohibitive).

Zero-hit ccache stats?

On some other runs, I see 0% cache hits, which is suspect.

2026-09-09T14:34:14.4446589Z Summary:
2026-09-09T14:34:14.4446870Z   Hits:               0 / 4467 (0.00 %)
2026-09-09T14:34:14.4447229Z     Direct:           0 / 4467 (0.00 %)
2026-09-09T14:34:14.4447577Z     Preprocessed:     0 /    0
2026-09-09T14:34:14.4447898Z   Misses:          4467
2026-09-09T14:34:14.4448205Z     Direct:        4467
2026-09-09T14:34:14.4448479Z     Preprocessed:     0
2026-09-09T14:34:14.4448756Z Primary storage:
2026-09-09T14:34:14.4449023Z   Hits:               0 / 4467 (0.00 %)
2026-09-09T14:34:14.4449357Z   Misses:          4467
2026-09-09T14:34:14.4449638Z   Cache size (GB): 1.81 / 5.00 (36.23 %)

and

2026-09-09T14:58:28.8100500Z Cacheable calls:    3913 / 3913 (100.0%)
2026-09-09T14:58:28.8101810Z   Hits:                0 / 3913 ( 0.00%)
2026-09-09T14:58:28.8102200Z     Direct:            0
2026-09-09T14:58:28.8102540Z     Preprocessed:      0
2026-09-09T14:58:28.8102820Z   Misses:           3913 / 3913 (100.0%)
2026-09-09T14:58:28.8103170Z Local storage:
2026-09-09T14:58:28.8103430Z   Cache size (GiB):  1.9 /  5.0 (38.98%)
2026-09-09T14:58:28.8103710Z   Hits:                0 / 3913 ( 0.00%)
2026-09-09T14:58:28.8104100Z   Misses:           3913 / 3913 (100.0%)

I saw that #8865 dropped the SHA + restore-keys from the restore step, but left the SHA on the save step. The reasoning was "drop the SHA to reduce evictions", but GitHub Actions cache keys are immutable (write-once). You cannot overwrite an existing key. To keep a cache fresh, each save must use a new, unique key. The _${github.sha} suffix provides that.

But because saves must be uniquely keyed, restores must use restore-keys (prefix match) to find "the newest one." An exact key with no SHA and no restore-keys (the current state) can never match a ..._<sha> save. I think this explains the 0% cache hit seen above. I'll prepare a PR to test/prove/disprove this. This theory was ruled out based on the discussion in https://github.com/osquery/osquery/pull/9105