Baike.dev
Log in
Back to tool/Back to issues
#16156·rustdesk

Linux: h264_vaapi encoding regressed in 1.4.4 on CQP-only Intel iGPUs — hwcodec never sets rc_mode for vaapi (last good: 1.4.3)

Author: NiehztogCreated Sep 10, 2026Updated Sep 12, 2026

Bug Description

On Intel iGPUs whose VA-API driver exposes H.264 encode only via the low-power entrypoint (VAEntrypointEncSliceLP, which supports VA_RC_CQP and nothing else), h264_vaapi hardware encoding regressed between 1.4.3 and 1.4.4 and is still broken in 1.4.9. The host silently falls back to software encoding.

Bisected across every release from 1.3.9 to 1.4.9 on the same machine, same OS, same driver — only the RustDesk version differs. Result was read from ram_encode in check-hwcodec-config:

Version Hardware encoding RC mode / avcodec_open2 errors
1.3.9, 1.4.0, 1.4.1, 1.4.2 works 0
1.4.3 (2025-10-17) works — last good release 0
1.4.4 (2025-11-19) broken — first bad release 2
1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9 broken 2 each

The break is clean and monotonic; it is not fixed again in any later release.

1.4.3:  ram_encode: [CodecInfo { name: "h264_vaapi", mc_name: None, format: H264, priority: 1, hwdevice: AV_HWDEVICE_TYPE_NONE }]
1.4.4:  ram_encode: []

Encoder init fails from 1.4.4 onward with:

ERROR [hwcodec/src/ffmpeg.rs:34] Driver does not support any RC mode compatible with selected options (supported modes: CQP).
ERROR [hwcodec/src/lib.rs:17] [FFMPEG_RAM_ENC] avcodec_open2 failed, ret = Invalid argument, name: h264_vaapi
DEBUG [hwcodec/src/ffmpeg_ram/encode.rs:386] Failed to create encoder h264_vaapi

Why it fails. In hwcodec, set_rate_control() (cpp/common/util.cpp) has entries for nvenc, amf and mediacodec, but none for vaapi, and low_power is never set anywhere:

std::vector<CodecOptions> codecs = {
    {"nvenc", "rc", {{RC_CBR, "cbr"}, {RC_VBR, "vbr"}}},
    {"amf", "rc", {{RC_CBR, "cbr"}, {RC_VBR, "vbr_latency"}}},
    {"mediacodec", "bitrate_mode", {{RC_CBR, "cbr"}, {RC_VBR, "vbr"}, {RC_CQ, "cq"}}},
    // {"videotoolbox", "constant_bit_rate", {{RC_CBR, "1"}}},
};

So for VAAPI only bit_rate is set and rc_mode is left at ffmpeg's auto. With a bitrate set and no AV_CODEC_FLAG_QSCALE/global_quality, ffmpeg's vaapi_encode rules out CQP and asks for VBR/CBR, which these drivers do not offer — so avcodec_open2 fails. This code is unchanged on hwcodec master, so it is latent on every version; something in the 1.4.4 build began exercising it.

What I could not identify: the trigger. The hwcodec pin moved between the two releases:

1.4.3  hwcodec @ 17c1dbb38450fe4a64aeba78fb50bec32f364a16
1.4.4  hwcodec @ 398e5a8938dd8768ade0fcdc27ea80e8b4b38738

That range is only 3 commits touching 2 files — cpp/common/platform/mac/mac.mm and src/ffmpeg_ram/encode.rs (PR rustdesk-org/hwcodec#39, "Support hardware h265 encoder detection for Intel Mac"). The encode.rs change is a retry loop around the encode test; it does not touch Encoder::new(), which is where avcodec_open2 actually fails. So that diff does not explain this regression.

The rustdesk repo diff for 1.4.3...1.4.4 shows no ffmpeg version bump either — res/vcpkg/ffmpeg/portfile.cmake only gains a macOS patch, and no vcpkg baseline change is visible in the workflows.

Update: the ffmpeg-version hypothesis is disproven. I previously suspected the ffmpeg linked into 1.4.4 was newer, with stricter RC auto-selection. It is not: res/vcpkg/ffmpeg/vcpkg.json pins ffmpeg 7.1 in 1.4.2, 1.4.3, 1.4.4, 1.4.9 and master alike. Recording that as a negative result so nobody else chases it.

So the trigger between 1.4.3 and 1.4.4 remains unidentified, with the bundled ffmpeg, Encoder::new() and the whole set_rate_control() path all unchanged across the boundary.

The nightly (1.5.0) build does not contain a fix. It pins the same hwcodec commit as 1.4.9 (778df1f), and set_rate_control() there still has no vaapi entry and no low_power anywhere. Noting this because trying the nightly is a natural suggestion.

How to Reproduce

  1. Use a Linux host whose iGPU exposes only the low-power H.264 encode entrypoint. Check with:
    vainfo --display drm --device /dev/dri/renderD128 -a
    Affected hardware shows VAEntrypointEncSliceLP with VAConfigAttribRateControl : VA_RC_CQP, and no plain VAEntrypointEncSlice.
  2. Install RustDesk 1.4.4 or newer (rustdesk-1.4.4-x86_64.deb) and enable the hardware codec.
  3. Inspect ~/.local/share/logs/RustDesk/check-hwcodec-config/rustdesk_rCURRENT.log → ram_encode: [] plus the RC-mode error above.
  4. Install 1.4.3 (rustdesk-1.4.3-x86_64.deb), restart the service, re-check → ram_encode: [h264_vaapi], no error.

Expected Behavior

h264_vaapi should stay available on these GPUs, as it was up to and including 1.4.3 — i.e. when the driver only advertises VA_RC_CQP, request CQP (and the low-power entrypoint) instead of a bitrate-based RC mode.

Operating system(s) on local (controlling) side and remote (controlled) side

Remote (controlled, where the failure happens): Debian 13 (trixie), kernel 6.12.107+deb13-amd64, XFCE on X11.

The controlling side is irrelevant here — the failure is at host-side encoder initialisation during --check-hwcodec-config and reproduces with no client connected at all.

RustDesk Version(s) on local (controlling) side and remote (controlled) side

Last working: 1.4.3. First broken: 1.4.4. Still broken as of 1.4.9. All tested with the official x86_64 .deb builds from GitHub Releases.

Screenshots

n/a — log output included above.

Additional Context

Hardware / driver stack:

  • CPU: Intel Core i3-8145U
  • GPU: Intel UHD Graphics 620, WhiskeyLake-U GT2 [8086:3ea0] (Gen9.5)
  • intel-media-va-driver (iHD) 25.2.3, libva 2.22.0

Relevant vainfo output — note the LP-only entrypoint and CQP-only rate control:

VAProfileH264Main               : VAEntrypointEncSliceLP
VAProfileH264High               : VAEntrypointEncSliceLP
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP

VAProfileH264High/VAEntrypointEncSliceLP
    VAConfigAttribRateControl : VA_RC_CQP

Hardware decoding (h264 + hevc via VAAPI) is unaffected and works on all versions tested.

LIBVA_DRIVER_NAME=i965 is not a viable workaround. The legacy i965 driver does advertise VAEntrypointEncSlice with VA_RC_CBR on this GPU, but encoder init aborts the process:

rustdesk: i965_encoder.c:1692: intel_enc_hw_context_init: Assertion `encoder_context->mfc_context' failed.
Check hwcodec config, exit with: signal: 6 (SIGABRT)

That is strictly worse than the current behaviour, because the SIGABRT kills the probe before it reaches the decoder tests, so hardware decoding is lost too.

h264_qsv is not an alternative on Linux either — it is gated behind #[cfg(windows)] in hwcodec's encoder list.

Current workaround: pin to 1.4.3 (apt-mark hold rustdesk on Debian/Ubuntu).

Suggested fix: add a vaapi entry to set_rate_control() mapping to the rc_mode option (RC_CQ → "CQP", with qp/global_quality set), ideally selected by querying the driver's supported RC modes; and set low_power=1 when only VAEntrypointEncSliceLP is available. Happy to test a build on this hardware.

Source: rustdesk/rustdesk

View original on GitHubView discussion on GitHub