de-bias gate is path-separator dependent: `--check` false-fails and `--fix` churns 373 files on Windows
Where
- File:
scripts/debias_quizzes.py - Function:
seed_for()(line 38), called fromdebias_question()(line 63) - Gate: the
quiz answer positions are de-biasedstep in.github/workflows/curriculum.yml(python3 scripts/debias_quizzes.py --check)
What's wrong
seed_for() derives the shuffle seed from the path string:
h = hashlib.sha256(f"{path}\x00{question_text}".encode("utf-8")).hexdigest()
That path comes from glob.glob("phases/*/*/quiz.json"), so it uses the platform's path separator. On Windows it is phases\02-ml-fundamentals\03-logistic-regression\quiz.json; on Linux and macOS it is phases/02-ml-fundamentals/03-logistic-regression/quiz.json. Different string, different seed, different permutation.
The de-biased arrangement committed to the repository was produced under POSIX paths, so it only reproduces where the separator is /.
The sibling script is already correct here — scripts/debias_certification_questions.py seeds off path.relative_to(ROOT).as_posix() and its gate passes on Windows. Only debias_quizzes.py was missed.
Impact
On Windows, on the same commit:
| Invocation | Result |
|---|---|
--check (the CI gate, documented in the module docstring) |
FAIL: 2098 quiz question(s) are not de-biased, exit 1 — a false failure; the arrangement is actually correct |
| bare run (the documented fix) | rewrites 2098 questions across 373 files into an arrangement Linux CI then rejects |
So the plugin's own documented remediation, run on Windows, produces a ~373-file churn diff that breaks the gate it was meant to satisfy.
Reproduce on Windows
python scripts/debias_quizzes.py --check
# questions: 2237 files affected: 373 questions would rewrite: 2098
# FAIL: 2098 quiz question(s) are not de-biased.
Feeding seed_for POSIX-normalized paths on the same Windows machine yields files affected: 0 questions would rewrite: 0, exit 0 — confirming the separator, not the content, is the variable.
Suggested fix
Normalize before hashing, matching the certification script:
normalized = path.replace("\\", "/")
h = hashlib.sha256(f"{normalized}\x00{question_text}".encode("utf-8")).hexdigest()
POSIX paths are unaffected (no backslashes to replace), so the committed arrangement and the CI gate do not change. A PR with regression coverage is linked below.
Source: rohitg00/ai-engineering-from-scratch