macOS Apple VT Issue on Apple Silicon, current method resulting in Encoding Overloaded (fixing can allow M2 Max to do 2x 4K60 HEVC streams at once)

Operating System Info

macOS 26

Other OS

No response

OBS Studio Version

32.2.2

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/JfZ8VKFaREqhBxJz

OBS Studio Crash Log URL

No response

Expected Behavior

I have an M2 Max Mac Studio, they double up the Apple encoding engines compared to the base M chips, but I consistently hit a limit on what gains could be had. The device should be able to encode two 4k60 HEVC streams, it does not as standard.

Initial testing was outside of OBS (ffmpeg), two 4K60 HEVC encodes run at the same time at about 96fps each with real-time on. The same Mac Studio can also do a good 4+ 4K60 encodes in ProRes in OBS.

Current Behavior

I receive "Encoding Overloaded" when attempting to record/stream more than one output at a time. Attempting to record at 4K60 and stream at 1440p60 runs into a wall, even with the more efficient CRF option. We're looking at 56.7% frames skipped due to encoding lag.

Steps to Reproduce

1. Canvas and output resolution 3840x2160, 60fps
2. Recording: Apple VT HEVC Hardware Encoder, CRF
3. Streaming: Apple VT HEVC Hardware Encoder, Rescale Output 2560x1440, CBR
4. Start recording, streaming (at the same time) with a moving source (capture card gameplay)
5. "Encoding overloaded" appears within seconds, and the log shows the skipped frames

Anything else we should know?

With a couple of minor tweaks (code) we can get TWO 4K60 HEVC streams recording/streaming at once, with headroom for another 1080p60 output too.

Within plugins/mac-videotoolbox/encoder.c, function create_encoder()

• kVTCompressionPropertyKey_RealTime is hard-coded to false, set it to true
• kVTCompressionPropertyKey_ExpectedFrameRate is always set, stop setting it.

Both are needed, 600 frames per 10 seconds is the target, meaning a clean 60fps. stock: 427 only frame rate change: 424 only RealTime: 436 both: 601

#5840 / #5880 was seemingly the reason RealTime was switched off, however I didn't see those drops with reverting RealTime and adjusting Expected Frame Rate, even with my 80 Mbps CRF bitrate limit. It seems this change was made in 2022 to match defaults for FFmpeg and Handbrake, however HandBrake is not for live streaming and FFmpeg allows you to set this to true. Apple documentation still says to set this to true for real-time/live encoding.