detectors.apikey.ApiKey hangs during detector phase with parallel_attempts=32 (CPU freezes, 61GB RSS)

Summary

The detectors.apikey.ApiKey detector hangs mid-execution during the detector phase. The progress bar freezes at 42% (108/256) and the process stops consuming CPU entirely, while RSS climbs to 61 GB. The probe generation phase completes successfully (256/256), but the subsequent detector phase never finishes.

This may be related to #1357 (parallel_attempts > 1 causing hangs) and #1588 (ReDoS in detector regexes).

Environment

• garak version: Packaged in container image garak:dev-2026-02-20T22-27-d12de5fb (Feb 2026)
• Python: 3.12 (inside container)
• OS: Linux (Slurm HPC cluster, NVIDIA H100)
• Model: Qwen3.5-4B served via vLLM (OpenAI-compatible endpoint)
• Probe: apikey.CompleteKey
• Detector: detectors.apikey.ApiKey
• parallel_attempts: 32 (set via system.parallel_attempts in garak config)
• generations: 5
• max_tokens: 32768

Steps to Reproduce

1. Serve a model via an OpenAI-compatible endpoint (e.g., vLLM)
2. Run garak with apikey.CompleteKey probe and parallel_attempts=32:
   yaml复制

   plugins:
     probe_spec: apikey.CompleteKey
     target_type: nim.NVOpenAIChat
     target_name: Qwen3.5-4B
     extended_detectors: true
   system:
     parallel_attempts: 32

3. garak completes probe generation (256/256) but hangs during detector analysis

Observed Behavior

The detector phase progress bar freezes at 42% and the process becomes unresponsive:

probes.apikey.CompleteKey: 100% 256/256 [1:41:44<00:00, 20.19s/it]
detectors.apikey.ApiKey:   42% 108/256 [27:05<00:30, 4.80it/s]  ← FROZEN HERE

Key evidence the process is hung (not slow):

The last line in garak.log before the hang:

2026-09-13 20:33:29,956  DEBUG  harness: run detector garak.detectors.apikey.ApiKey

No errors, exceptions, or tracebacks are logged — the process simply stops after the detector starts.

Expected Behavior

The detector phase should complete (256/256) and the run should finish normally. If a specific output causes a regex to hang, garak should timeout or skip that attempt rather than blocking indefinitely.

Analysis

Possible causes

1. ReDoS in ApiKey regexes: The ApiKey detector uses regex patterns from the dora project. PR #1588 already fixed a similar exponential-backtracking issue in the JavaScript detector. Some ApiKey patterns have large quantifier ranges (e.g., pypi_upload_token: ([A-Za-z0-9-_]){50,1000}) and alternation with overlapping matches (square_application_secret) that could cause catastrophic backtracking on certain model outputs.

2. parallel_attempts=32 threading issue: Issue #1357 reported garak hanging for 3+ hours with parallel_attempts > 1. Our config sets parallel_attempts: 32. The detector phase may have a similar threading/deadlock issue when processing results in parallel.

3. Memory pressure: 61 GB RSS for a regex-only detector is abnormally high. The process may be memory-leaking or swapping, causing it to block on I/O rather than compute (consistent with frozen CPU time).

4. None detector results: ApiKey.detect() returns None when output is None or output.text is None. Issue #1513 documented a TypeError from None detector results. A similar edge case in the parallel processing path could cause a deadlock.

Suggested Investigation

• Test ApiKey detector with parallel_attempts=1 to see if the hang reproduces (isolate threading vs. regex issue)
• Add a timeout to detector_instance.detect() calls in harnesses/base.py:_run_detector()
• Audit REGEX_DICTS patterns for catastrophic backtracking (similar to #1588)
• Check for memory leaks in the detector result aggregation path when parallel_attempts > 1