Daily ironclaw failure taxonomy — 2026-09-17
Daily ironclaw failure taxonomy — 2026-09-17
Suites analyzed
- officeqa (35 non-pass) — This run's 35 non-pass tasks are overwhelmingly genuine model-quality errors: DeepSeek-V4-Flash navigates the OCR-digitized Treasury Bulletins with healthy, self-recovering trajectories (read/grep/shell all succeed) but reads the wrong cell, picks the wrong row/year, or applies a wrong formula, so the final number falls outside the tight ±1% tolerance while nothing in ironclaw fails. Every 'Hard' capability signature (Network, OperationFailed, http 404/403, auth_required) is the model attempting OFF-DOCUMENT external HTTP fetches (BLS, FRED, treasury.gov CPI/MTS files) on a document-grounded suite whose answer is already in the seeded workspace .txt — the harness correctly failed/denied those, and the model then fell back to the file and computed a wrong number anyway. The one genuine harness defect is UID0140: a transcript-write serialization error terminally fails the turn because retry_transcript_backend_write retries only Backend, not Serialization, errors (blast radius 1; the model's answer was also wrong). Two empty-answer tasks (UID0028, UID0030) have no trajectory or error and read as transient empty completions (infra). The prior run's cosmetic http.rs:492 RequestDenied→InputEncode mislabel is still live but maps to no task in this bundle (no RequestDenied signature appears), so it is noted here rather than filed as a category.
ironclaw rev(s): b0b999d96781516ee05e6ba961d6f3ead900da96
Harness fix candidates (ranked by blast radius)
| # | Category | Tasks | Suites | Confidence | Files |
|---|---|---|---|---|---|
| 1 | Terminal transcript-write serialization failure | 1 | officeqa:1 | 0.40 | crates/domains/ironclaw_threads/src/filesystem_service.rs:3029, crates/loop/ironclaw_loop_host/src/lib.rs:1239, crates/loop/ironclaw_loop_host/src/lib.rs:3548 |
1. Terminal transcript-write serialization failure — 1 tasks
Root cause: In ThreadBackedLoopTranscriptPort, retry_transcript_backend_write (crates/loop/ironclaw_loop_host/src/lib.rs:1249-1274) retries ONLY SessionThreadError::Backend(). A SessionThreadError::Serialization() — produced when serializing the appended assistant/tool-result-reference record in crates/domains/ironclaw_threads/src/filesystem_service.rs (e.g. :3029-3137) — is deterministic, so it returns immediately through transcript_write_error (lib.rs:3548) as AgentLoopHostErrorKind::TranscriptWriteFailed, which planned_driver maps to a terminal HostUnavailable. A record that cannot serialize (oversized/control-char/non-UTF8 model_observation preview or safe_summary derived from a large saved HTTP body) therefore permanently wedges the transcript instead of degrading gracefully. LLM data must never be dropped, but a serialization fault should not be a terminal, unrecoverable host failure.
Fix: Make transcript record serialization non-fatal and, better, unfailable. (1) At the source, bound and sanitize the fields fed into append_tool_result_reference / append_assistant_draft records (model_observation preview, safe_summary) so a large/control-char/non-UTF8 tool observation cannot yield a serde error — cap length and lossily scrub non-serializable content at construction, matching the existing safe-summary discipline. (2) At the port, when the append still returns SessionThreadError::Serialization, degrade to a stored placeholder tool-result-reference record (retaining the result_ref and a truncated safe summary) rather than raising TranscriptWriteFailed and terminally aborting the turn — reserve the terminal host failure for genuine Backend loss. Add a regression test appending a tool-result reference whose observation would previously fail serde, asserting the turn continues and the record (or placeholder) is persisted.
Files: crates/domains/ironclaw_threads/src/filesystem_service.rs:3029, crates/loop/ironclaw_loop_host/src/lib.rs:1239, crates/loop/ironclaw_loop_host/src/lib.rs:3548
officeqa: UID0140
Per-suite breakdown (every task links to its trajectory)
officeqa (35 non-pass)
- Terminal transcript-write serialization failure — 1 · ironclaw (PR-able) UID0140
- Numeric extraction/analysis error over OCR'd document tables — 28 · model quality UID0018 · UID0041 · UID0052 · UID0053 · UID0058 · UID0059 · UID0062 · UID0057 · UID0077 · UID0084 · UID0101 · UID0096 · UID0113 · UID0114 · UID0121 · UID0117 · UID0123 · UID0138 · UID0148 · UID0172 · UID0174 · UID0071 · UID0037 · UID0110 · UID0085 · UID0027 · UID0074 · UID0135
- Off-document external HTTP fetches on a document-grounded task — 11 · model quality UID0012 · UID0029 · UID0118 · UID0174 · UID0003 · UID0074 · UID0135 · UID0027 · UID0071 · UID0110 · UID0037
- Model refusal / off-the-rails narrative instead of an answer — 1 · model quality UID0003
- Empty final answer (transient empty completion) — 2 · infra (rerun) UID0028 · UID0030
Generated by the bench-taxonomy workflow. Categories are dynamic — discovered from each run's failures and root-caused via codegraph/source.
Source: nearai/ironclaw