chore(test): validate estimate_screen_color thresholds against real-world plates

Author: alexandremendoncaalvaroCreated May 1, 2026Updated May 1, 2026

Context

PR #241 added estimate_screen_color(image_srgb, alpha_hint) in CorridorKeyModule/core/color_utils.py. It runs whenever the user passes --screen-color auto (the default) and decides whether to load the green or the blue checkpoint by comparing mean(G) vs mean(B) on the background pixels of the first frame in the batch.

It has three magic numbers, all chosen by reasoning about a typical well-lit plate:

Threshold Default What it decides
Background mask alpha_hint < 0.3 Which pixels count as "the screen" (everything else is the subject).
Minimum coverage 1% of pixels Below this we treat the sample as untrustworthy (subject fills the frame, background is occluded) and fall back to green.
Ambiguity gap 0.05 between mean(G) and mean(B) Below this we can't tell green from blue with confidence and fall back to green.

These match an intuition about "reasonable VFX footage." They are not validated against actual CorridorKey plates.

Why this matters

The auto-detection runs once per batch and applies to every clip — get it wrong and an entire render goes through the wrong checkpoint, with despill on the wrong channel. The drift warning added in PR #241 only fires when subsequent clips look different from the first; it does not catch a wrong call on the first clip.

Two specific failure modes the current thresholds may not handle well:

  1. Underexposed plates. A dim blue screen (B ≈ 0.3, G ≈ 0.25) sits inside the 0.05 ambiguity window and silently routes to green. Same plate at full exposure (B ≈ 0.85, G ≈ 0.10) routes correctly.
  2. Subject-dominant frames. A close-up shot where the subject fills 99%+ of frame triggers the "coverage too low" fallback. That fallback is green, regardless of what the actual screen color is. A blue plate's first frame happening to be a close-up would mis-route.
  3. Lit-from-the-front blue plate. Studios sometimes use blue with strong front lighting that brings the subject's face into the same brightness range as the screen. mean(G) vs mean(B) on a person's skin can confuse the heuristic.

I haven't seen these failures in the wild because there is no corpus to validate against.

Why we didn't tune it inline with PR #241

The thresholds need real footage to validate. PR #241 ships the auto-detection working correctly on the synthetic tests (well-lit green vs well-lit blue, ambiguous-defaults-to-green, no-background-defaults-to-green) — that's the floor of correctness. Threshold tuning is a tightening, not a fix, and tightening without data is just moving knobs around.

The fallback is also conservative-to-green by design: a wrong green decision degrades the despill quality on a blue plate but does not crash. A user noticing residual blue cast can re-run with --screen-color blue explicitly. The auto path is a UX shortcut, not a safety-critical decision.

Definition of done

  • A small fixture corpus (5–10 clips minimum) covering a realistic spread of plates, committed under tests/fixtures/screen_color_corpus/ (or excluded from VCS and pulled from a known location — see Open questions below).
  • A parametrized test that runs estimate_screen_color on the first frame of each fixture and asserts the verdict.
  • A short markdown doc next to the fixtures describing each plate's lighting condition (well-lit green, dim blue, close-up subject, etc.) so the corpus stays interpretable.
  • If any threshold needs to change to handle the corpus, the change is justified in the commit message with the specific case it fixes.

Suggested fixture spread

Aim for one each of:

  • Well-lit green (canonical case — should detect green easily)
  • Well-lit blue
  • Greenscreen with magenta backlight (a known despill pain point for traditional keyers; tests that the "is screen pixels" mask isolates correctly)
  • Blue with cyan reflections (subject's clothing reflects the blue, blurring the boundary)
  • Green with subject wearing predominantly blue (e.g. blue costume — subject pixels skew the mean if alpha hint is loose)
  • Blue with subject wearing predominantly green
  • Underexposed green (low B and low G — the ambiguity guard should fire)
  • Close-up plate (subject fills most of frame — coverage guard should fire)
  • Mixed-lighting plate (sodium vapor in the room, e.g. an interior practical) — actual studio reality
  • Garbage-mask test (a plate where the alpha hint is rough; should still detect the right screen)

10 plates is enough for first-pass tuning. The point isn't exhaustive coverage, it's known cases with known correct verdicts.

Suggested approach

  1. Collect the corpus. Best source is whatever plates the project authors / community Discord can share. Anonymize if needed (cropped frames, no recognizable people). 480p is enough — auto-detection only needs the broad pixel statistics.
  2. Snapshot the alpha hint per plate. A reasonable hint can come from any of the existing alpha generators (BiRefNet is fastest). Save alongside the input frame.
  3. Write a parametrized test in tests/test_color_utils.py::TestEstimateScreenColorRealWorld that loads each fixture, runs estimate_screen_color, and asserts the expected verdict. Include the actual (G_mean, B_mean, coverage) tuple in the assertion message so failures are diagnosable.
  4. If a fixture fails on the current defaults, first check whether the verdict is genuinely wrong (the keyer would key it badly) or the test expectation was wrong. Only after that, adjust thresholds — and document which case forced the change.
  5. Don't over-tune. A single threshold change per failed case. If three different plates push the ambiguity threshold in three different directions, the heuristic itself is the wrong model and we should revisit (e.g., look at the spread of B−G across the background pixels, not just the mean).

Open questions for whoever picks this up

  • Where do the fixtures live? Adding 10 small images to the repo is fine. If they need to be larger or there are licensing concerns, host them on HF and pull them in a conftest.py fixture with caching.
  • Does the test get a @pytest.mark.slow marker? Reading 10 image files is fast (<1s), but if any fixture is video, it pushes into the slow-tier and CI might want to skip it.
  • Should we expose the thresholds as parameters? estimate_screen_color(..., ambiguity_threshold=0.05) already does this. The 0.3 background cutoff and 1% coverage minimum are still hard-coded constants — promoting them keeps the heuristic configurable without code changes.

Out of scope

  • Replacing estimate_screen_color with a smarter algorithm (ML-based screen detection, color-clustering, etc.). That's a separate design conversation.
  • Sampling more than the first frame for auto-detection. Would help with the close-up edge case but adds I/O cost on every batch.
  • Per-clip auto-detection (re-detecting screen color for every clip in a batch). Currently we detect once and warn on drift; making it per-clip would change the engine cache contract.

Source: nikopueringer/CorridorKey