Feature: GenAI observability — docs + optional @genai_trace extension reusing existing OTel infra
Author: arnabwithabCreated Sep 6, 2026Updated Sep 9, 2026
Context
Recursive steps (Aug 2025) enable agentic loops but Metaflow has no
LLM cost/latency observability. OTel GenAI semconv is GA May 2026
(opentelemetry.io/blog/2026/genai-observability). Prefect/Dagster/Flyte
and Airflow 3.3 already emit OTel; Metaflow's own OTel exists at
metaflow/tracing/__init__.py:57, tracing_modules.py:36 (OTEL_ENDPOINT,
OTEL_SERVICE_NAME), span_exporter.py:13, but no GenAI story.
#3307 addresses a similiar problem
Proposed lazy path
Phase 1, docs + extension (no core churn):
- Reuse existing
METAFLOW_OTEL_ENDPOINT/CONSOLE_TRACE_ENABLED— no new flag. - Provide
metaflow_extensionsexampleGenaiTraceDecorator(StepDecorator)atextension_support/__init__.py:343seam that wrapstraced()and emitsgen_ai.usage.*+gen_ai.modelonly (capture_messages=False by default, opt-in redacted events). Noinstall_requireschange,extras_require[otel]or extension requirements.txt only. Lazy import so local runs without OTel still work (tracing/__init__.py:58pattern). - Doc recipe:
with traced("gen_ai.invoke_agent", {...}): ...+mflog.decorate(...)so users get OTLP to existing Jaeger/Tempo without a decorator if they prefer. ReusesRequestsInstrumentorattracing_modules.py:46+ auto-instrumentation.
Phase 2, graduate to core @genai_trace iff >3 teams adopt:
- Promote to
plugins/__init__.py:39 STEP_DECORATORS_DESCwith maintainer sign-off, add tosetup.py:extras_require, keep PII off by default.
Scope
- Phase 1:
docs/genai-observability.md+examples/genai_trace/+ one-file extension (ormetaflow/plugins/genai_trace.pybehindif tracer_provider is None: no-op). - Tests:
test/unit/test_genai_trace.pychecks attrs without collector (mock exporter).
Questions for maintainers before coding
- Core vs extension: graduate-if-adopted OK?
- Dep policy:
extras_require[otel]or extension-only? - PII default:
capture_messagesoff? - Offline contract: silence vs buffer when
OTEL_ENDPOINTunreachable?
Source: Netflix/metaflow