Unable to provide valid hw_frames_ctx for CUDA filter chain in JavaCV GPU pipeline

Issue Description

Hello, I'm trying to implement a complete GPU pipeline using JavaCV and encountering an issue with hardware frame context initialization.

Goal

I want to implement a fully GPU-accelerated pipeline for RTSP stream processing:

1. GPU decoding: Using GPU decoders (h264_cuvid, hevc_cuvid)
2. GPU memory retention: Setting PixelFormat = AV_PIX_FMT_CUDA to keep frames in GPU memory
3. Hardware frame upload: Using hwupload_cuda to convert GPU-decoded frames to hardware frames (hw_frames) for CUDA filters
4. Scaling and color conversion: Using scale_cuda CUDA video filter for resizing and RGB conversion
5. GPU encoding: Using NVENC hardware encoder to convert RGB to JPEG

Problem

I'm getting the following error during FFmpegFrameFilter initialization:

Setting BufferSourceContext.pix_fmt to a HW format requires hw_frames_ctx to be non-NULL! Query format failed for 'in': Invalid parameter avfilter_graph_config() error -22

Current Code Approach

java // GPU decoder initialization WeakReference weakRecorder = new WeakReference<>(grabber); grabber = new FFmpegFrameGrabber(config.getRtspUrl()); grabber.setOption("rtsp_transport", "tcp"); grabber.setVideoOption("gpu", "0"); grabber.setVideoCodecName("h264_cuvid"); grabber.setPixelFormat(avutil.AV_PIX_FMT_CUDA); grabber.setVideoOption("surfaces", "64"); grabber.setVideoOption("extra_hw_frames", "5");

// hwaccel config does not work #grabber.setVideoOption("hwaccel", "cuda"); #grabber.setVideoOption("hwaccel_device", String.valueOf(gpu));

// GPU filter chain // String gpuFilter = "scale_cuda=format=rgb24"; // Also reported same error // String gpuFilter = "vpp_cuda=format=rgb24"; // Also reported same error String gpuFilter = "hwupload_cuda,scale_cuda=format=rgb24"; filter = new FFmpegFrameFilter(gpuFilter, width, height); filter.setPixelFormat(avutil.AV_PIX_FMT_CUDA); filter.start(); // Fails here avutil.av_log_set_level(avutil.AV_LOG_ERROR);

Error log

[in @ 0x7f484fe39f40] Setting BufferSourceContext.pix_fmt to a HW format requires hw_frames_ctx to be non-NULL! [in @ 0x7f484fe39f40] Query format failed for 'in': Invalid parameter Info: avformat_open_input rejected some options: Option: pixel_format, value: bgr24 Input #0, rtsp, from 'rtsp://192.168.96.136:30242/aisp-data-analysis-center/1916462708309229570': Metadata: title : Streamed by ZLMediaKit(git hash:8bf48ed/2024-12-15T11:43:31+08:00,branch:master,build time:2024-12-15T03:44:48) Duration: N/A, start: 0.023750, bitrate: N/A Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s [h264_cuvid @ 0x7f484fe50780] Invalid pkt_timebase, passing timestamps as-is. [in @ 0x7f47ec6ac800] Setting BufferSourceContext.pix_fmt to a HW format requires hw_frames_ctx to be non-NULL! [in @ 0x7f47ec6ac800] Query format failed for 'in': Invalid parameter 2025-10-15 16:56:30.049 [scheduling-1] INFO c.b.v.c.capture.executor.RtspCaptureExecutor - set h264_cuvid success 2025-10-15 16:56:30.051 [scheduling-1] ERROR c.b.v.c.capture.executor.RtspCaptureExecutor - error:avfilter_graph_config() error -22 (For more details, make sure FFmpegLogCallback.set() has been called.) org.bytedeco.javacv.FFmpegFrameFilter$Exception: avfilter_graph_config() error -22 (For more details, make sure FFmpegLogCallback.set() has been called.) at org.bytedeco.javacv.FFmpegFrameFilter.startVideoUnsafe(FFmpegFrameFilter.java:425) at org.bytedeco.javacv.FFmpegFrameFilter.startUnsafe(FFmpegFrameFilter.java:321) at org.bytedeco.javacv.FFmpegFrameFilter.start(FFmpegFrameFilter.java:289)

Specific Issue

I learned that. the scale_cuda filter requires hardware frames (hw_frames), but I don't know how to properly provide the CUDA hardware frame context (hw_frames_ctx). The hwupload_cuda filter expects the input to have a valid hardware frame context, which doesn't seem to be automatically set when using GPU decoding with JavaCV.

Questions

1. How can I properly initialize/provide the CUDA hardware frame context (hw_frames_ctx) for the filter chain?
2. Is there a specific sequence or additional configuration needed to make hwupload_cuda work with GPU-decoded frames?
3. Are there alternative approaches to achieve the same goal (full GPU pipeline with color output) while avoiding this hardware frame context issue?

Environment

• JavaCV version: 1.5.12
• GPU: NVIDIA GeForce RTX 2080
• CUDA: 12.2
• OS: Linux
• java：21

gradle： implementation "org.bytedeco:opencv:4.11.0-1.5.12:linux-x86_64-gpu" implementation "org.bytedeco:cuda:12.6-9.5-1.5.11:linux-x86_64-redist" implementation "org.bytedeco:ffmpeg:7.1.1-1.5.12:linux-x86_64-gpl" implementation "org.bytedeco:openblas:0.3.30-1.5.12:linux-x86_64" // javacv implementation "org.bytedeco:javacv:1.5.12" implementation "org.bytedeco:javacpp:1.5.12:linux-x86_64" implementation "org.bytedeco:ffmpeg:7.1.1-1.5.12:linux-x86_64"

Any guidance or working examples for implementing a complete GPU pipeline with JavaCV would be greatly appreciated!