VAAPI Encoder Broken using MP4 Container w/ AMD GPU (Navi48)

Problem Description

Using new VAAPI encoding (see https://github.com/HandBrake/HandBrake/pull/7467) with MP4 container target fails on AMD GPU (Navi48) while MKV container target works.

Filed by @galad87 here https://github.com/HandBrake/HandBrake/pull/7467#issuecomment-4786277177 Log test file: https://github.com/user-attachments/files/29406433/vaapi.mp4.log.txt Codecs: H264, H265 and AV1, all on vaapi.

My reproduction excerpt. Encoding fails w/ result = 4.

[mp4 @ 0x7f89d8579900] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 26610333 >= 26610333
ERROR: avformatMux: track 0, av_interleaved_write_frame failed with error 'Invalid argument'
[mp4 @ 0x7f89d8579900] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 26610333 >= 26610333
[18:38:09] worker Muxer waiting to die
...
[18:38:09] libhb: work result = 4

Encode failed (error 4).

Potentially related ffmpeg issue - but more specific to hevc_vaapi, while the issue filed here exists for all codecs.

• https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15529
• as mentioned above https://github.com/HandBrake/HandBrake/pull/7467#issuecomment-4786277177

Activity Log, Crash Log or any other details

see above and referenced log file.

What Operating System are you running?

GNU/Linux

What version of HandBrake are you running?

main branch

Where did you download HandBrake from?

github