Proposal: supported external reliability scoring and regression-testing pattern for AgentSession

Feature Type

Would make my life easier

Feature Description

We have built a working prototype that turns a LiveKit AgentSession into a
provider-neutral reliability trace for post-call diagnosis and repeatable
regression testing. It captures final transcripts, user and agent state changes,
response latency, tool execution, interruptions, usage, provider errors and
session completion, then sends the observed facts to an external service for
server-side scoring.

The prototype currently calculates four reliability components—turn handling,
transcript integrity, tool reliability and response latency—and uses the same
normalized event model for six deterministic acceptance scenarios and historical
regression monitoring.

This is not a request to add the complete external product to livekit/agents.
Before preparing an upstream PR, I would like guidance on the smallest useful and
maintainable contribution. Would the project be interested in one of these?

1. A dependency-light example showing the supported way to observe an
   AgentSession, build a bounded post-session reliability payload and flush it
   during shutdown.
2. Documentation describing which public session events and ordering guarantees
   external reliability integrations can safely depend on.
3. A small generic exporter/reporter interface that external reliability tools
   can implement without adding a vendor dependency to the core package.
4. A reporter that connects LiveKit simulations or CI runs to an external
   regression-testing backend.

I would particularly value maintainer feedback on the following technical
questions:

• Is attaching public event listeners before session.start() an encouraged
  integration pattern, or should post-session integrations prefer
  session.history, session.usage and recorded session data?
• Which event contracts are intended to be stable for third-party consumers,
  especially tool lifecycle, interruption, transcription-timeout, usage and
  close events?
• Are event ordering and terminal delivery sufficiently defined for an adapter
  to produce one deterministic report, or should an upstream example document
  deduplication and fallback rules?
• Should an example default to metadata-only capture and require an explicit
  opt-in before exporting transcript or tool content?
• Would a LiveKit simulation/CI reporter be more useful than a runtime
  AgentSession example?

The aim is to contribute a small, tested integration pattern that other
observability and reliability tools can reuse—not to introduce a large framework
or a TurnTrace-specific dependency into LiveKit Agents. If the current approach
relies on the wrong event or lifecycle assumptions, I am happy to revise the
prototype and contribute the narrower primitive or example that maintainers
recommend.

A clear indication that this is or is not aligned with the project would be
valuable before we invest further in extracting an upstream patch.

Workarounds / Alternatives

The current prototype attaches an adapter before session.start() and registers
an async shutdown callback. It normalizes public AgentSession events, guards
against duplicate in-process flushes and sends the completed trace through an
idempotent, token-protected ingestion endpoint.

Keeping this entirely in an external repository works, but it leaves several
questions unanswered for other integration authors: which event surface is the
preferred public contract, how to handle attachment timing and fallback events,
and how to export content without creating privacy or redaction surprises.

OpenTelemetry is another possible transport, but the prototype also needs
application-level facts and deterministic acceptance results. I would welcome
guidance on whether this should instead be expressed as an OTLP consumer or a
custom observability endpoint.

Additional Context

Prototype and implementation notes:

• Source: https://github.com/niallsemple/turntrace
• LiveKit adapter: https://github.com/niallsemple/turntrace/blob/main/sdk/python/livekit_adapter.py
• Runnable agent example: https://github.com/niallsemple/turntrace/blob/main/sdk/python/livekit_example.py
• Event map, scoring model and deployment notes: https://github.com/niallsemple/turntrace/blob/main/docs/livekit-adapter.md
• Adapter test: https://github.com/niallsemple/turntrace/blob/main/sdk/python/test_livekit_adapter.py

Quick adapter check:

git clone https://github.com/niallsemple/turntrace.git
cd turntrace/sdk/python
python -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python -m unittest -v test_livekit_adapter

Current verification evidence:

• The adapter test covers event normalization and duplicate-safe shutdown flush.
• The example imports and runs against LiveKit Agents 1.6.8.
• A local worker registered successfully with LiveKit Cloud.
• The container built remotely and the production worker registered as
  turntrace-front-desk in eu-central.
• The first real microphone-session trace is still pending. If a maintainer
  reproduces a lifecycle or event-contract problem first, I am happy to turn it
  into a failing test and work from that evidence.

This proposal is intended to complement, not duplicate:

• livekit/agents#6664 / #6666, which focus on a post-call CRM and analytics report plus webhook
  delivery.
• livekit/agents#5853 / livekit/agents#6026, which focus on exporting session recordings to third-party
  observability systems.

TurnTrace's distinct use case is event-level reliability diagnosis, deterministic
failure scenarios and regression detection across repeated runs. I am happy to
test against the direction chosen for those efforts and adjust the proposed
scope accordingly.