#3364·metaflow

Feature: GenAI observability — docs + optional @genai_trace extension reusing existing OTel infra

Author: arnabwithabCreated Sep 6, 2026Updated Sep 9, 2026

Context

Recursive steps (Aug 2025) enable agentic loops but Metaflow has no LLM cost/latency observability. OTel GenAI semconv is GA May 2026 (opentelemetry.io/blog/2026/genai-observability). Prefect/Dagster/Flyte and Airflow 3.3 already emit OTel; Metaflow's own OTel exists at metaflow/tracing/__init__.py:57, tracing_modules.py:36 (OTEL_ENDPOINT, OTEL_SERVICE_NAME), span_exporter.py:13, but no GenAI story.

#3307 addresses a similiar problem

Proposed lazy path

Phase 1, docs + extension (no core churn):

  • Reuse existing METAFLOW_OTEL_ENDPOINT / CONSOLE_TRACE_ENABLED — no new flag.
  • Provide metaflow_extensions example GenaiTraceDecorator(StepDecorator) at extension_support/__init__.py:343 seam that wraps traced() and emits gen_ai.usage.* + gen_ai.model only (capture_messages=False by default, opt-in redacted events). No install_requires change, extras_require[otel] or extension requirements.txt only. Lazy import so local runs without OTel still work (tracing/__init__.py:58 pattern).
  • Doc recipe: with traced("gen_ai.invoke_agent", {...}): ... + mflog.decorate(...) so users get OTLP to existing Jaeger/Tempo without a decorator if they prefer. Reuses RequestsInstrumentor at tracing_modules.py:46 + auto-instrumentation.

Phase 2, graduate to core @genai_trace iff >3 teams adopt:

  • Promote to plugins/__init__.py:39 STEP_DECORATORS_DESC with maintainer sign-off, add to setup.py:extras_require, keep PII off by default.

Scope

  • Phase 1: docs/genai-observability.md + examples/genai_trace/ + one-file extension (or metaflow/plugins/genai_trace.py behind if tracer_provider is None: no-op).
  • Tests: test/unit/test_genai_trace.py checks attrs without collector (mock exporter).

Questions for maintainers before coding

  1. Core vs extension: graduate-if-adopted OK?
  2. Dep policy: extras_require[otel] or extension-only?
  3. PII default: capture_messages off?
  4. Offline contract: silence vs buffer when OTEL_ENDPOINT unreachable?