Stereo 3D layout signalled in the stream (H.264 frame packing SEI, MP4 st3d) is ignored

mpv Information

mpv v0.41.0, libplacebo v7.360.1 — Debian 14, x86_64, libavcodec 63.1.101.

Summary

mpv detects a stereo 3D layout only from the Matroska StereoMode element. A layout signalled in the video stream — the H.264/HEVC frame_packing_arrangement SEI (payload type 45), or the MP4 st3d box — is ignored, and such files play as stereo-in=mono.

libavcodec already decodes that SEI into AVStereo3D frame side data and hands it to mpv. mpv never asks for it.

Reproduction Steps

Three files isolating each signal, built in three commands:

ffmpeg -f lavfi -i "testsrc2=size=960x1080:rate=24:duration=2" \
       -f lavfi -i "smptebars=size=960x1080:rate=24:duration=2" \
       -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" \
       -c:v libx264 -crf 20 -pix_fmt yuv420p source_sbs.mp4

ffmpeg -i source_sbs.mp4 -c:v copy -an plain.mkv                      # no signal
mkvmerge -o container_only.mkv --stereo-mode 0:1 plain.mkv            # container tag only
ffmpeg -i source_sbs.mp4 -c:v libx264 -crf 20 -pix_fmt yuv420p \
       -x264-params frame-packing=3 sei_only.mp4                      # in-band SEI only

for f in plain.mkv container_only.mkv sei_only.mp4; do
  printf '%-20s ' "$f"
  mpv --no-config --vo=null --frames=1 --no-audio \
      --term-playing-msg='${video-params}' "$f" 2>/dev/null | grep -i stereo-in
done

Actual Behavior

plain.mkv            stereo-in=mono
container_only.mkv   stereo-in=sbs2l
sei_only.mp4         stereo-in=mono     <-- expected sbs2l

Expected Behavior

sei_only.mp4 should report stereo-in=sbs2l. The two files are the same picture carrying the same layout declared two different ways.

The data is present and correct

The SEI is in the stream — trace_headers parses it:

$ ffmpeg -v trace -i sei_only.mp4 -map 0:v:0 -frames:v 2 -c copy \
      -bsf:v trace_headers -f null - 2>&1 | grep frame_packing
frame_packing_arrangement_id                 1 = 0
frame_packing_arrangement_cancel_flag        0 = 0
frame_packing_arrangement_type         0000011 = 3      # 3 = side by side
frame_packing_arrangement_repetition_period  010 = 1

And libavcodec already converts it to AVStereo3D and attaches it to the decoded frame:

$ ffmpeg -v info -i sei_only.mp4 -vf showinfo -frames:v 1 -f null -
[Parsed_showinfo_0] side data - Stereo 3D: type - side by side, view - packed, primary_eye - none

So by the time mpv receives the AVFrame, the layout is already decoded, normalised and attached.

Diagnosis

stereo3d has exactly one source in the tree. demux/demux_mkv.c:752 reads the Matroska element, and no other demuxer sets it — grep -rn 'stereo3d\|StereoMode\|stereo_mode' demux/*.c matches only demux_mkv.c.

mp_image_from_av_frame() (video/mp_image.c) imports nine kinds of frame side data from libavcodec — AV_FRAME_DATA_DISPLAYMATRIX, ICC_PROFILE, MASTERING_DISPLAY_METADATA, CONTENT_LIGHT_LEVEL, DYNAMIC_HDR_PLUS, A53_CC, DOVI_METADATA, DOVI_RPU_BUFFER, FILM_GRAIN_PARAMS — but not AV_FRAME_DATA_STEREO3D.

Additionally, fix_image_params() in filters/f_decoder_wrapper.c overwrites the layout on every frame with the container tag from the demuxer.

Why reading the side data is not enough on its own

Worth recording, because it is the non-obvious half of the bug. The in-band signal is attached only to the frames that carry it — measured on the files above:

An exact 1:1 match with keyframes, consistent with frame_packing_arrangement_repetition_period = 1 ("persists until the next IDR"). So reading the side data alone still produced mono: the keyframe was detected, the following frame carried no side data, fix_image_params() ran again and reset the format. The layout has to persist across the frames that do not repeat it.

Relation to existing issues

• #1045 ("auto-detecting 3D input format") is closed and predates AVStereo3D being wired through this path.
• #18348 concerns the removed full-width sbsl/abl choices; its reporter's file is a Matroska whose tag mpv reads correctly. That is the layout vocabulary; this is the detection source.

Why this is becoming more common

HandBrake merged support for writing this SEI (PR #8100), and FFmpeg has an open request for a lossless injector for it (#24531), so files carrying the in-band signal are going to keep appearing.

Fix

A tested two-part patch is ready and I will open a pull request referencing this issue: read AV_FRAME_DATA_STEREO3D in mp_image_from_av_frame(), and remember an in-band layout across the frames that do not repeat it in f_decoder_wrapper.c, falling back to the container tag only when the stream signals nothing.

Verified on a patched build: sei_only.mp4 goes from mono to sbs2l, three real SEI-only MP4s go from mono to sbs2l/sbs2r, and no container-tagged or unsignalled file changes behaviour. One of those files declares side by side (inverted) — right eye first — while its filename says SBS, so it is also a case where guessing from the filename would swap the viewer's eyes.

Disclosure

An AI partner helped produce the outcomes and this post.

• Harness: Claude Code (Anthropic's agentic CLI)
• Model class: Claude Opus (Anthropic frontier tier)
• Model: Claude Opus 5 (1M context), model ID claude-opus-5[1m]

Every result above was executed and measured on the machine described, not inferred: the patch was compiled, the before/after values come from running both binaries against the same files, and the side-data-per-keyframe counts were measured with ffmpeg -vf showinfo.