signals — trajectory triage infrastructure for agentic interactions

Title

epic: signals — trajectory triage infrastructure for agentic interactions

Overview

Plano is implementing the signal-based triage framework described in Signals: Trajectory Sampling and Triage for Agentic Interactions (arXiv:2604.00356, Chen et al., Apr 2026).

The paper proposes attaching lightweight, model-free signals to live agent trajectories so that a small fraction of high-information interactions can be sampled for review, preference-data construction, and post-deployment optimization. The reported result: 82% informativeness at 1.52× efficiency over heuristic baselines on τ-bench, without affecting online agent behavior.

This epic tracks the end-to-end delivery of that framework inside brightstaff: detectors → attributes → triage sampler → feedback loop.

Signal taxonomy (paper §3)

Three coarse-grained categories, seven mid-level groups, 20 leaf signal types:

Layer	Category	Intent
Interaction	misalignment	semantic/intent mismatch between user and agent
	stagnation	discourse continues without visible progress
	disengagement	withdrawal of cooperative intent
	satisfaction	explicit stabilization and completion
Execution	failure	action attempts that don't yield usable outcomes
	loops	repetitive execution patterns without progress
Environment	exhaustion	boundary and infrastructure conditions

All detectors are computed without model calls; all are attached as structured OTel span attributes + events so they surface in existing observability pipelines without a separate sidecar.

Phases

Phase 1 — Analyzer + observability integration ✅ DONE

Delivered by #903.

• Rust port of the Python reference (katanemo/signals) into crates/brightstaff/src/signals/, aligned with the paper's three-layer taxonomy and 20 leaf SignalTypes
• Strongly-typed SignalReport exposing per-layer, per-category counts and severity (no stringly-keyed maps)
• Dual-emit OTel attributes: - New layered keys (signals.interaction.misalignment.count, …) - Legacy aggregate keys kept for backward compatibility
• Per-instance span events (signal.<type> with confidence, snippet, metadata)
• Summary string grouped by taxonomy
• Parity harness (tests/parity/signals/) — 100% agreement with Python reference on 2,000 lmsys/lmsys-chat-1m samples (seed 42)

Phase 2 — Parity, coverage, documentation IN PROGRESS

Tracked in the follow-up issue (child of this epic). Briefly:

• Documentation for the new OTel attribute set; deprecation notice for legacy keys
• Synthetic tool-trace dataset for execution.* + environment.* parity (lmsys covers only interaction — 10/20 leaf types fire naturally there)
• Pin the HF dataset revision in the parity workflow for bit-reproducibility
• Upstream the grouped-summary format to katanemo/signals (optional)
• Legacy OTel key sunset in a subsequent release

References

• Paper: arXiv:2604.00356
• Python reference implementation: katanemo/signals
• Plano Phase 1 PR: #903
• Phase 2 follow-ups:
• Related observability surface: #904 (brightstaff Prometheus metrics)

Authors / DRIs

• Paper: @nehcgs, @adilhafeez, @salmanap
• Plano implementation DRI: @syedhashmi