Daily ironclaw failure taxonomy — 2026-09-10
Daily ironclaw failure taxonomy — 2026-09-10
Suites analyzed
- officeqa (42 non-pass) — This officeqa run's 42 non-pass tasks are overwhelmingly genuine model errors: DeepSeek-V4-Flash navigates the OCR-digitized Treasury Bulletins with healthy, self-recovering trajectories but extracts the wrong cell, picks the wrong year/table, or applies a wrong formula, leaving the final number outside the +/-1% tolerance (e.g. UID0041 computes a clean 0.0118 Theil index that rounds to 0.012 vs golden 0.011). Every 'Hard' capability signature is an incidental off-document FRED fetch the model used as a fallback: builtin.http returns 403 (OperationFailed) or a transport reset (Network), the model recovers via curl in builtin.shell, and still lands the wrong number. The only harness thread is that builtin.http is rejected/reset where the identical-URL curl succeeds, which localizes to the ironclaw_network reqwest client sending no default User-Agent -- real but non-causal for this document-grounded suite. Two bad_eval candidates (UID0057 needs 1970-1980 data from an Oct-1969 bulletin; UID0110 needs 2017 data with only 2016/2020 bulletins seeded) look like mis-seeded source assets but cannot be confirmed from the bundle. No harness defect drives the failures; the prior run's cosmetic apply_patch message-drop note does not recur.
ironclaw rev(s): 0280dd1a48a4f813f123735b31558c459d1ce032
Harness fix candidates (ranked by blast radius)
| # | Category | Tasks | Suites | Confidence | Files |
|---|---|---|---|---|---|
| 1 | builtin.http rejected/reset where shell curl succeeds | 4 | officeqa:4 | 0.40 | crates/substrates/ironclaw_network/src/transport.rs:107 |
1. builtin.http rejected/reset where shell curl succeeds — 4 tasks
Root cause: build_reqwest_client() constructs the outbound reqwest client with only redirect(none) + timeout and never calls .user_agent(...), so builtin.http requests carry no User-Agent header. Public hosts behind WAFs commonly 403 or reset empty-UA requests, producing the OperationFailed(403) and Network(reset) signatures seen here, while the shell's curl (default UA) is admitted. This makes the mediated HTTP capability less reliable than the raw shell for exactly the class of public endpoints it exists to serve. Confidence is bounded because the bundle exposes only the generic reason codes (ResponseError/NetworkError), not the server's rejection reason.
Fix: In build_reqwest_client(), set a default User-Agent on the reqwest::Client::builder() (e.g. .user_agent(concat!("ironclaw/", env!("CARGO_PKG_VERSION"))) or a configurable neutral browser-like UA) so builtin.http requests are not indistinguishable from empty-UA bot traffic. Verify with an integration test in crates/kernel/ironclaw_host_runtime/tests/runtime_http_egress_contract.rs asserting the outbound request carries a non-empty User-Agent header, and confirm the header is still redacted/scrubbed like other request headers. This closes the builtin.http-vs-curl parity gap; it does not change officeqa outcomes.
Files: crates/substrates/ironclaw_network/src/transport.rs:107
officeqa: UID0118 · UID0057 · UID0029 · UID0042
Per-suite breakdown (every task links to its trajectory)
officeqa (42 non-pass)
- builtin.http rejected/reset where shell curl succeeds — 4 · ironclaw (PR-able) UID0118 · UID0057 · UID0029 · UID0042
- Numeric extraction/analysis error over OCR'd document tables — 35 · model quality UID0006 · UID0012 · UID0005 · UID0027 · UID0018 · UID0034 · UID0041 · UID0053 · UID0058 · UID0059 · UID0062 · UID0037 · UID0069 · UID0077 · UID0074 · UID0084 · UID0082 · UID0085 · UID0089 · UID0101 · UID0105 · UID0096 · UID0102 · UID0113 · UID0121 · UID0123 · UID0138 · UID0140 · UID0147 · UID0148 · UID0150 · UID0174 · UID0029 · UID0110 · UID0118
- Off-document external HTTP fetch as a fallback — 8 · model quality UID0057 · UID0114 · UID0120 · UID0117 · UID0135 · UID0042 · UID0029 · UID0118
- No-progress termination with empty or give-up answer — 2 · model quality UID0030 · UID0118
1 benchmark-eval defect(s) from these runs are filed separately in nearai/benchmarks (they're eval bugs, not harness bugs).
Generated by the bench-taxonomy workflow. Categories are dynamic — discovered from each run's failures and root-caused via codegraph/source.
Source: nearai/ironclaw