Linux (X11/amdgpu): fullscreen stream start applies an invalid DoubleScan HDMI mode, TV rejects it as "unsupported"

Describe the bug

As soon as an actual GameStream session starts (any app, any tested resolution/framerate: 1920x1080@30, 1920x1080@60, [email protected]), the TV immediately drops to a legacy fallback signal and displays "unsupported mode" on its own OSD. This happens only once the stream itself starts — Moonlight-Qt's own app-selection screen (before starting a stream) displays perfectly fine on the exact same cable/output.

Diagnosed by comparing DISPLAY=:0 xrandr --verbose | grep "\*" on the HDMI output right before starting a stream vs. during the "unsupported mode" state (stream still connected):

Before / normal desktop (working fine):

1920x1080 (0x74) 148.500MHz +HSync +VSync *current

Standard CEA-861 timing for 1080p60.

During the failure (stream connected):

1920x1080 (0x71) 356.375MHz -HSync +VSync DoubleScan *current

356MHz for a nominal 1920x1080 mode is far outside any real 1080p timing (real 1080p60 ≈ 148.5MHz, 1080p30 ≈ 74.25MHz), and the DoubleScan flag is a strong indicator of a badly-synthesized modeline rather than a genuine, EDID-listed CEA-861 timing. No kernel/dmesg errors accompany this transition — the amdgpu driver appears to consider the modeset successful; only the TV's own interpretation of the resulting signal rejects it.

For reference, manually forcing an existing EDID mode (reusing a mode already known to the display, not synthesizing one) via xrandr --output HDMI-1 --mode 1920x1080 --rate 30 produces a clean 74.250MHz +HSync +VSync mode and displays correctly — confirming the display/EDID/cable are not the limiting factor, and that this specific resolution/refresh is genuinely supported when applied through a normal, non-synthesized path.

I checked a fresh clone of the moonlight-qt source tree for any custom X11 modeline-creation code (cvt, XRRAddOutputMode, XRRCreateMode, etc.) and found none — on Linux, display mode switching appears to rely entirely on SDL2 (SDL_SetWindowDisplayMode / SDL_SetWindowFullscreen), not custom code in this project. So this looks like it's happening inside SDL2's X11 mode-switching path, or in the interaction between SDL2's requested mode and the amdgpu DDX/modesetting driver — but I don't have visibility into SDL2 internals to confirm further. I'm also filing this against Batocera's own repo since their Moonlight launch integration explicitly opts out of their own safe resolution-setting mechanism for this system, so I can't be 100% sure which project actually owns the bug.

Steps to reproduce

1. Set up Apollo (Sunshine fork) as GameStream host on a Windows 11 PC, and pair a Batocera v43.1 mini PC (AMD Radeon 780M iGPU) as a Moonlight-Qt client, connected via HDMI to a 4K TV.
2. From Batocera, launch the "Moonlight" system and start a stream (e.g. "Desktop") to the paired host, at any resolution/framerate.
3. As soon as the stream connects and video starts arriving, the TV immediately drops to a legacy fallback signal and shows "unsupported mode".
4. This happens on every single attempt, regardless of the windowmode setting in Moonlight.conf (tested 0/1/2 — WM_FULLSCREEN, WM_FULLSCREEN_DESKTOP, WM_WINDOWED) or the target resolution/framerate.
5. It does NOT happen just browsing Moonlight-Qt's own app-selection screen before starting a stream — only once the actual video stream begins.

Screenshots

I have photos of the TV showing "Modalità non supportata" (Italian TV firmware — "Mode not supported") taken during earlier stages of this investigation; happy to attach them if useful, or take fresh ones showing the current 4K TV/setup. [NOTE FOR STEFANO: attach here the earlier TV photos from this chat if you still want to include them, or take a fresh one next time you reproduce it — I don't have image-attach access to the GitHub form myself.]

Affected games

Primarily reproduced streaming "Desktop" (Batocera's own basic Moonlight entry, no actual game involved) — the TV rejects the signal before any game/app content is ever displayed, so this is not game-specific. After working around the underlying bug (see "Additional context"), also confirmed a real game (BioShock) streams correctly once the corrected mode is in place, with Moonlight-Qt's stats overlay showing 0.00% dropped frames and ~1ms network latency — so the encode/network/game side of the pipeline is otherwise fine; the issue is purely the HDMI mode applied at stream start.

Other Moonlight clients

Not tested — only tested via Moonlight-Qt on this Batocera (Linux/X11) setup, no iOS/Android client available to compare against.

Moonlight settings (please complete the following information)

• Yes, several settings were changed while troubleshooting, but the bug occurs identically with default settings too:
  • windowmode in Moonlight.conf: tested all three values (0/1/2) — bug occurs identically in all three, including the default (0, exclusive fullscreen).
  • HDR: tried disabling HDR entirely on the host side (Apollo → Advanced display device options → "Do not change HDR settings" instead of the default "Auto switch HDR state as client requests") — no change either way.
  • Resolution/framerate/bitrate: tested 1080p@30, 1080p@60, [email protected], various bitrates up to 90Mbps — bug occurs identically regardless.
• So: reverting every setting back to default does not resolve the issue.

Gamepad-related issues

Not applicable — this is a pure video/HDMI-mode issue, unrelated to input.

Client PC details

• OS: Batocera Linux v43.1 (Linux-based, purpose-built gaming/emulation distro)
• Moonlight Version: 6.1.0
• GPU: AMD Radeon 780M iGPU (amdgpu driver, DCN 3.1.4)
• Linux package type: not applicable — this is the moonlight-qt build bundled directly in the Batocera OS image (not Flatpak/Snap)

Server PC details

• OS: Windows 11
• Apoll Version 0.4.6
• GPU: NVIDIA RTX 4060
• GPU driver: 616.64

Moonlight Logs

I was not able to locate a Moonlight-Qt client-side log file on this Batocera build (it doesn't appear to be in an obvious ~/.local/share or ~/.cache location under the userdata home, at least not one I could find). Happy to look further or capture live stdout by launching moonlight-qt manually over SSH if a maintainer can point me to where Batocera's build writes logs (if anywhere) — the diagnosis above was done entirely via xrandr/dmesg on the Batocera side and Apollo's host-side logs, since I couldn't find a moonlight-qt-specific log.

Additional context

Extra investigation done while chasing this down, in case it's useful:

• Ruled out as causes: faulty HDMI cable (separate, earlier, already-fixed issue), a stale per-client Apollo Display Mode Override, antivirus/firewall on the host (Kaspersky fully uninstalled, no change), managed switch settings (port isolation/QoS/storm control/loop guard all checked, no change), all required Sunshine/Apollo network ports (verified open/listening both idle and during an active session), and HDR (see above).
• Batocera's own Moonlight launch integration (configgen/generators/moonlight/moonlightGenerator.py) has getResolutionMode() return the literal string 'default' for this system, meaning Batocera's generic/safe resolution-setting mechanism (batocera-resolution setMode, which just does a plain xrandr --mode X --rate Y reusing an existing EDID mode) is never invoked around a Moonlight launch. Resolution/framerate reach moonlight-qt purely as CLI flags (--1080/--4K, --fps N) derived from Batocera's own per-system settings.
• Current workaround: re-applying an existing, known-good EDID mode via batocera-resolution setMode <res>.<rate> a couple of seconds after the stream connects (while it's already active/broken) immediately fixes the picture and audio. This has been automated via a Batocera game-event script that fires on gameStart for system "moonlight". It works reliably (no visible glitch, no issue exiting the stream) but obviously isn't a real fix — it has to reapply the correction on every single launch, and the target mode is hardcoded and must be kept in sync manually with whatever resolution/framerate is configured in Moonlight-Qt.
• I'm filing essentially the same report on Batocera's repo as well, since I can't determine with confidence whether the fix belongs in Moonlight-Qt's SDL2 usage, in Batocera's launch integration, or further upstream in SDL2/mesa/amdgpu itself.