Daily ironclaw failure taxonomy — 2026-09-07
Author: pranavraja99Created Sep 7, 2026Updated Sep 7, 2026
Daily ironclaw failure taxonomy — 2026-09-07
Suites analyzed
- officeqa (42 non-pass) — This run's 42 officeqa failures are overwhelmingly genuine model-quality numeric errors: DeepSeek-V4-Flash navigates the OCR-digitized Treasury Bulletins with healthy, self-recovering trajectories (read_file/grep/shell all succeed) but extracts the wrong cell, picks the wrong year/quarter, or applies a wrong formula, so the final number misses the ±1% tolerance while nothing in ironclaw actually breaks. Every 'Hard' capability signature is incidental to a document-grounded task: the Network/OperationFailed/builtin.http[.save] entries are all off-document external fetches (FRED, stlouisfed.org) that correctly fail because officeqa never needs the web, the operation_failed/policy_denied/resource entries are self-recovered arg errors or a missing-numpy fallback inside otherwise model-error trajectories, and UID0012 even printed the correct 36080 in its own shell before submitting the wrong 35532. No harness defect drives any failure here (the prior run's InputEncode-mislabel and apply_patch-message issues do not surface as failure causes), so there is no ironclaw fix and the PR order is empty. Two caveats worth a separate bad_eval audit but unconfirmable from this bundle: UID0013 is an exactly-1000x unit mismatch that could be a golden-answer units convention, and UID0041's Theil index (0.0118 vs expected 0.011) may reflect a data-selection/methodology difference.
ironclaw rev(s): 0280dd1a48a4f813f123735b31558c459d1ce032
Harness fix candidates (ranked by blast radius)
No harness-attributed failure modes today.
Per-suite breakdown (every task links to its trajectory)
officeqa (42 non-pass)
- Numeric extraction/analysis error over OCR'd document tables — 38 · model quality UID0008 · UID0021 · UID0014 · UID0030 · UID0034 · UID0041 · UID0057 · UID0058 · UID0059 · UID0060 · UID0062 · UID0066 · UID0069 · UID0077 · UID0082 · UID0075 · UID0089 · UID0074 · UID0105 · UID0113 · UID0084 · UID0114 · UID0124 · UID0123 · UID0121 · UID0138 · UID0135 · UID0143 · UID0148 · UID0144 · UID0154 · UID0044 · UID0096 · UID0012 · UID0110 · UID0037 · UID0140 · UID0013
- Off-document external HTTP fetch (correctly failing) — 6 · model quality UID0096 · UID0012 · UID0110 · UID0044 · UID0027 · UID0120
- No final numeric answer emitted — 3 · model quality UID0028 · UID0029 · UID0027
Generated by the bench-taxonomy workflow. Categories are dynamic — discovered from each run's failures and root-caused via codegraph/source.
Source: nearai/ironclaw