bug(sdk-python): media is uploaded before masking runs — neither mask nor mask_otel_spans can prevent it

Describe the bug

Summary

Media content is extracted and queued for upload before either masking hook can act on it, so neither mask nor mask_otel_spans can keep the bytes of an inline image, audio clip or document out of Langfuse. The masked span that ships afterwards carries only a @@@langfuseMedia:... reference, which reads as if masking worked.

The ordering itself is deliberate — _process_media_attributes() runs ahead of _apply_mask_otel_spans() in LangfuseSpanExporter.export(), and test_mask_otel_spans_receives_post_media_batch_and_applies_sparse_patch pins it. What I don't think is intended is the consequence: mask_otel_spans is documented as the export-stage hook for exactly this problem, and it is structurally unable to prevent a media upload.

The sharpest case is the failure path. The docstring says:

If the hook raises or returns an invalid batch result, Langfuse drops the whole export batch.

That is a fail-closed guarantee, and it doesn't hold for media. When the hook raises, the span batch is dropped as documented — and the media bytes are uploaded anyway.

Expected behavior

One of:

• media extraction runs after the masking hooks, so a redacted attribute never produces an upload; or
• media uploads enqueued from a batch are discarded when that batch is dropped, so "drops the whole export batch" means all of it; or
• at minimum, the mask / mask_otel_spans docs state plainly that neither hook covers media content, and that inline media must be stripped before it reaches the SDK.

Actual behavior

Span attributes are masked correctly. The media bytes behind them are queued for upload with their original content, including when the batch that produced them is dropped.

The legacy mask hook doesn't help either, though for a different reason: media is extracted at attribute-creation time, so mask is handed a LangfuseMedia object rather than the data: URI. A string-based masking function — the shape in every example I can find — sees a non-str and returns it untouched.

Reproduction

Offline: fake keys, in-memory span exporter, and the upload job intercepted instead of sent. Nothing leaves the machine. Only dependency is langfuse.

Steps in the "Steps to reproduce" section

Output on main (4.14.4):

--- mask_otel_spans redacts every attribute ---
spans exported                   : ['handles-a-document']
canary in exported attributes    : ABSENT (masked) - OK
media upload jobs enqueued       : 1
canary in uploaded media bytes   : ['0n9Nh-_jU64hx5Wwv-4I0M']

--- mask_otel_spans raises (batch is dropped) ---
spans exported                   : []
canary in exported attributes    : ABSENT (masked) - OK
media upload jobs enqueued       : 1
canary in uploaded media bytes   : ['0n9Nh-_jU64hx5Wwv-4I0M']

The interception point is MediaManager._process_upload_media_job, which is the method the upload consumer calls to POST the bytes, so a job reaching it is a job that would have been sent.

Why this matters

mask_otel_spans is what the masking docs point users to when mask isn't enough, and langfuse/langfuse#15372 is a user arriving there for compliance reasons. Someone who has done everything the docs ask — export-stage hook, fail-closed allowlist — still ships the contents of any inline image or document to Langfuse. For a self-hosted deployment that is a surprise; for Cloud it may be a compliance problem.

Steps to reproduce

"""Does `mask_otel_spans` cover media, or is the content uploaded before it runs?

Fully offline: fake keys, in-memory span exporter, and the media upload job is
intercepted rather than sent. Nothing leaves the machine.
"""

import base64

from langfuse import Langfuse
from langfuse._task_manager.media_manager import MediaManager
from langfuse.types import MaskOtelSpansResult, OtelSpanPatch
from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult

CANARY = b"PII-CANARY-INSIDE-MEDIA-789"

# A "screenshot" whose bytes carry PII. Content is never parsed by the SDK.
MEDIA_BYTES = b"\x89PNG\r\n\x1a\n" + CANARY + b"\x00trailing"
DATA_URI = "data:image/png;base64," + base64.b64encode(MEDIA_BYTES).decode()

captured_spans: list = []
upload_jobs: list = []


class InMemoryExporter(SpanExporter):
    def export(self, spans):
        captured_spans.extend(spans)
        return SpanExportResult.SUCCESS

    def shutdown(self):
        pass


# Intercept at the point the SDK would POST the bytes to the media API.
def intercept_upload(self, *, data):
    upload_jobs.append(data)


MediaManager._process_upload_media_job = intercept_upload


def redact_everything(*, params):
    """The fail-closed allowlist pattern the docs recommend: redact all attributes."""
    patches = {}
    for identifier, span in params.spans.items():
        patches[identifier] = OtelSpanPatch(
            set_attributes={key: "[REDACTED]" for key in span.attributes}
        )
    return MaskOtelSpansResult(span_patches=patches)


def raise_instead(*, params):
    raise RuntimeError("masking function blew up")


def run(label, mask_fn, public_key):
    captured_spans.clear()
    upload_jobs.clear()

    client = Langfuse(
        public_key=public_key,
        secret_key="sk-lf-0000",
        mask_otel_spans=mask_fn,
        span_exporter=InMemoryExporter(),
    )

    with client.start_as_current_observation(name="handles-a-document", input=DATA_URI):
        pass

    client.flush()
    client.shutdown()

    attr_leaks = [
        (s.name, k)
        for s in captured_spans
        for k, v in (s.attributes or {}).items()
        if CANARY.decode() in str(v)
    ]
    media_leaks = [
        job["media_id"] for job in upload_jobs if CANARY in job.get("content_bytes", b"")
    ]

    print(f"\n--- {label} ---")
    print(f"spans exported                   : {[s.name for s in captured_spans]}")
    print(f"canary in exported attributes    : {attr_leaks or 'ABSENT (masked) - OK'}")
    print(f"media upload jobs enqueued       : {len(upload_jobs)}")
    print(f"canary in uploaded media bytes   : {media_leaks or 'ABSENT - OK'}")


run("mask_otel_spans redacts every attribute", redact_everything, "pk-lf-0001")
run("mask_otel_spans raises (batch is dropped)", raise_instead, "pk-lf-0002")

Langfuse Cloud or self-hosted?

Self-hosted

If self-hosted, what version are you running?

Not applicable — the repro is SDK-only with an in-memory exporter, so the behavior is independent of the deployment.

SDK and integration versions

• langfuse (Python SDK) main at 73b5c02 (4.14.4)
• opentelemetry-sdk 1.39.x
• Python 3.12.13, macOS

Additional information

Related but distinct: langfuse/langfuse#15372 is about mask not covering third-party span attributes. This is about media content escaping both hooks, and it reproduces on a plain native observation with no third-party instrumentation involved.

Are you interested in contributing a fix for this bug?

Yes