NVIDIA: alpha is lost when compositing client buffers delivered via linux-dmabuf — transparent windows render as opaque black

Environment

• niri 26.04 (single output [email protected])
• NVIDIA RTX 4090, driver 595.71.05 (open module), nvidia_drm.modeset=1
• Xwayland 24.1.13, spawned rootless by xwayland-satellite 0.8.2
• Also reproduced with a standalone xwayland-satellite :1 (see reproduction below)

Summary

On this stack, a client buffer that has an alpha channel and is delivered via linux-dmabuf is composited as if it were fully opaque: pixels with alpha = 0 are drawn in their RGB value (black), instead of letting what is behind them show through.

The same buffer delivered via wl_shm composites correctly. Only the buffer transport differs between the two runs below — same window, same content, same size, same focus state.

Practical impact here: Wine/CEF applications create transparent Depth-32 (ARGB) toplevel windows for window frames, shadows and popup menus. With the alpha dropped, each of those becomes an opaque black rectangle covering the application — e.g. the whole app window goes black, and context menus flash black.

Evidence

1. The alpha survives all the way to niri.

WAYLAND_DEBUG=1 on the xwayland-satellite process, buffer creation for the window in question (decimal + fourcc decoded):

# Xwayland -> satellite
create_immed(new id wl_buffer#25, 1897, 2130, 875713089, 0)
create_immed(new id wl_buffer#26, 1897, 2130, 875713089, 0)

# satellite -> niri
-> [email protected]_immed(wl_buffer@42, 1897, 2130, 875713089, 0)
-> [email protected]_immed(wl_buffer@43, 1897, 2130, 875713089, 0)

875713089 = 0x34325241 = AR24 = DRM_FORMAT_ARGB8888. Both hops use the same format; the wl_shm path is not used at all in this run (wl_shm_pool occurrences: 0, zwp_linux_buffer_params_v1: 22).

The surface is clean — attach / damage_buffer / frame / commit only. Nothing calls wl_surface.set_opaque_region (0 occurrences in the trace, and the satellite source contains no such call).

2. A/B: only the buffer path changes the result.

Test window: a root X window with a 32-bit TrueColor visual, content = fully transparent black plus one opaque coloured square. In both runs the window ends up tiled at 1265x1420 logical and focused; grim captures are compared before/after the window appears. "New black" = pixels that were not near-black before and are near-black after.

The 9.9 % in the shm run is not window content: it matches niri's own window shadow (layout { shadow { size … } }), i.e. the transparent content composited correctly and only niri's shadow was added. With dmabuf, the window content itself was black.

Reproduction

No Wine/CEF needed — any 32-bit-visual X window with transparent content shows it:

# 1. a second, standalone instance so the main session is untouched
xwayland-satellite :1                    # or: xwayland-satellite :1 -glamor none

# 2. create a root window on :1 with depth 32, map it, fill with 0x00000000,
#    draw one opaque marker (Xlib/XCB; the depth-32 window needs CWBorderPixel set,
#    otherwise XCreateWindow returns BadMatch)

# 3. grim before / after, compare

If that is useful I can attach the exact script (it is ~90 lines of Python using ctypes against libX11, no build dependencies).

What this is not

• Not a client bug — reading the window back with XGetImage shows correct content (transparent where it should be, opaque where it should be).
• Not Xwayland or xwayland-satellite — the trace above shows AR24 is passed through unchanged on the way to niri.
• Not an opaque-region issue — nothing sets one.

Possibly related

• smithay::backend::renderer::gles: [GL] GL_INVALID_VALUE error generated. Size and/or offset out of range. is logged by niri on this machine while such windows are on screen. I saw it only a handful of times, so I am not claiming a firm correlation — but it is the renderer path, not the X server.
• Downstream write-up with the full story: originally filed as xwayland-satellite#502 — I have since posted a correction there with the same evidence (https://github.com/Supreeeme/xwayland-satellite/issues/502#issuecomment-5718461402); the cause was initially — incorrectly — attributed to the satellite's "capture path"; that issue should be corrected to point here).

Workaround used downstream

Running Xwayland in shm mode (xwayland-satellite -glamor none, i.e. Xwayland -shm) makes the alpha composite correctly. Cost: all X11 clients go through CPU copies instead of GPU dmabuf, which is a significant performance regression for anything demanding (games, video) — so it is a workaround, not a fix.