de-bias gate is path-separator dependent: `--check` false-fails and `--fix` churns 373 files on Windows

Author: dajiaohuangCreated Sep 10, 2026Updated Sep 10, 2026

Where

  • File: scripts/debias_quizzes.py
  • Function: seed_for() (line 38), called from debias_question() (line 63)
  • Gate: the quiz answer positions are de-biased step in .github/workflows/curriculum.yml (python3 scripts/debias_quizzes.py --check)

What's wrong

seed_for() derives the shuffle seed from the path string:

h = hashlib.sha256(f"{path}\x00{question_text}".encode("utf-8")).hexdigest()

That path comes from glob.glob("phases/*/*/quiz.json"), so it uses the platform's path separator. On Windows it is phases\02-ml-fundamentals\03-logistic-regression\quiz.json; on Linux and macOS it is phases/02-ml-fundamentals/03-logistic-regression/quiz.json. Different string, different seed, different permutation.

The de-biased arrangement committed to the repository was produced under POSIX paths, so it only reproduces where the separator is /.

The sibling script is already correct here — scripts/debias_certification_questions.py seeds off path.relative_to(ROOT).as_posix() and its gate passes on Windows. Only debias_quizzes.py was missed.

Impact

On Windows, on the same commit:

Invocation Result
--check (the CI gate, documented in the module docstring) FAIL: 2098 quiz question(s) are not de-biased, exit 1 — a false failure; the arrangement is actually correct
bare run (the documented fix) rewrites 2098 questions across 373 files into an arrangement Linux CI then rejects

So the plugin's own documented remediation, run on Windows, produces a ~373-file churn diff that breaks the gate it was meant to satisfy.

Reproduce on Windows

python scripts/debias_quizzes.py --check
# questions: 2237  files affected: 373  questions would rewrite: 2098
# FAIL: 2098 quiz question(s) are not de-biased.

Feeding seed_for POSIX-normalized paths on the same Windows machine yields files affected: 0 questions would rewrite: 0, exit 0 — confirming the separator, not the content, is the variable.

Suggested fix

Normalize before hashing, matching the certification script:

normalized = path.replace("\\", "/")
h = hashlib.sha256(f"{normalized}\x00{question_text}".encode("utf-8")).hexdigest()

POSIX paths are unaffected (no backslashes to replace), so the committed arrangement and the CI gate do not change. A PR with regression coverage is linked below.

Source: rohitg00/ai-engineering-from-scratch