Parallelise index_call_sites and process_libs: ~75% of scan time is single-threaded on large apps

Background

• Brakeman version: 8.0.6 (also checked the two methods below against current main — unchanged)
• Rails version: 8.0
• Ruby version: 3.4
• Application code: closed-source (a large Rails monolith at Gusto). All figures below are rounded/approximate.

Is your feature request related to a problem? Please describe.

On a large monolith — roughly 50k Ruby files, ~2k controllers, ~3k models, ~1.5k templates — a full scan takes about 430s on an 8-core CI machine, but most of that runs on one core. Approximate phase breakdown, taken from the progress output:

So the two phases that are already parallel are ~100s of the scan, and roughly 325s is serial. The machine's load average sat around 1.75 on 8 cores for most of the run.

Worth noting why Processing libraries dominates here rather than models/controllers: FileTypeDetector#detect_type falls back to :lib, so on an app that keeps most of its logic in service objects, jobs and query objects, the large majority of files are processed as libs. I'd expect this shape on any big app that isn't controller/model-centric.

The checks themselves are ~15s. Nearly all the wall time is preparation.

Describe the solution you'd like

Two independent changes, easiest first.

1. Parallelise Tracker#index_call_sites (~90s here). This one looks like it needs no design work, because it is already accumulate-then-merge:

def index_call_sites
  finder = Brakeman::FindAllCalls.new self

  self.each_method do |definition, set_name, method_name, file|
    finder.process_source definition, :class => set_name, :method => method_name, :file => file
  end
  # ... each_class, each_template, initializers ...

  @call_index = Brakeman::CallIndex.new finder.calls
end

One FindAllCalls accumulates into finder.calls, and the CallIndex is constructed once at the end. Each worker could take a partition of each_method / each_class / each_template, run its own FindAllCalls, and return its calls; the parent concatenates them and builds a single CallIndex. That's the same "Parallel.map returning results, parent collects" shape FileParser#parse_files already uses, so there's an in-tree precedent and no new dependency.

2. Parallelise Scanner#process_libs (~160s here). Bigger win, genuinely harder:

def process_libs
  libs = file_cache.libs.sort_by { |path, _| path }
  track_progress libs do |path, lib|
    process_step_file path do
      process_lib lib
    end
  end
end

process_lib goes through LibraryProcessor.new(@tracker).process_library, which mutates the shared tracker. This needs either per-worker partial trackers merged in the parent, or thread-safe accumulation into the library index. I'd treat it as a follow-up to (1).

Describe alternatives you've considered

• Sharding the scan across parallel CI jobs by file subset. Rejected, and I'd argue against anyone doing it: Tracker and CallIndex are global, so a controller that reaches a sink through a service object in another shard loses that edge. The failure mode is a silent false negative in a security scanner, which is the worst possible direction.
• --skip-libs. Removes the single biggest phase, but on an app of this shape that drops most of the application's code from analysis.
• Rescanning support / persisting the file cache between CI runs. Plausible, but the cache is the parsed AST for ~50k files, so storing and restoring it may well cost more than the scan.

Additional context / questions

• Determinism is the constraint I'd most want to preserve. process_libs deliberately sorts by path, and warning fingerprints need to be stable across runs, so any parallel version has to produce byte-identical output including ordering. Is there order-dependence in the tracker beyond that sort that I should know about?
• Was serial execution a deliberate choice? My main suspicion is marshalling cost — Parallel in processes has to serialise results back. For (1) the returned calls should be small next to the ASTs parse_files already marshals today; for (2) it's a real question.
• How would you want it exposed — reuse the existing parallel_checks option, add a separate flag, or default on?

Happy to prototype (1) and open a PR if the approach sounds acceptable — I didn't want to send code before checking whether there's a reason this was left serial.