[BUG] Intermittent Video Corruption in BigBlueButton Recordings
Describe the bug
We are experiencing intermittent video corruption in BigBlueButton recordings.
Some recorded sessions contain short periods of severe video artifacts/noise. The affected section can appear unexpectedly in the middle of a long recording and may last approximately 10 seconds to 1 minute.
The issue is intermittent: not all recordings are affected, and the rest of an affected recording is usually normal.
Most importantly, the corruption is already present in the recorded/raw video file itself. It is therefore not limited to the BigBlueButton playback interface.
The issue started appearing within the last month. We did not intentionally make any BigBlueButton configuration changes or perform a relevant software upgrade immediately before the issue started.
We have also observed the issue with newer/latest BigBlueButton versions.
To Reproduce
We have not yet identified a deterministic way to reproduce the issue.
The problem occurs intermittently during normal online classes:
- Start a BigBlueButton meeting.
- Conduct a normal online class with video/webcam and/or screen sharing.
- Record the meeting.
- End the meeting and wait for the recording to be processed.
- Open the recording.
- Some recordings may contain a corrupted/noisy video section, while the rest of the recording remains normal.
The corrupted section can occur at an apparently random point in the recording.
Expected behavior
The recorded video should remain intact throughout the entire recording without corrupted frames, severe visual artifacts, or video noise.
Actual behavior
Some portions of the recorded video contain severe visual corruption/noise.
For example, a recording may be approximately two hours long, while only a 10-second to 1-minute section in the middle is corrupted.
The corruption is present in the raw recorded video and is therefore not introduced only during playback.
Audio does not appear to be affected. The issue appears to be limited to video.
Screenshots
Attached is a screenshot showing an example of the corrupted video frames.
Server environment
- BigBlueButton:
2.6.17 (1445)or even3.0.34 - OS:
Ubuntu 20.04.6 LTS - Kernel:
5.4.0-216-generic - CPU: 16 cores
- RAM: 16 GB
- FFmpeg:
4.2.7-0ubuntu0.1
Relevant recording configuration:
recordScreenSharing: true
recordWebcams: true
codec_video_main: VP8
codec_video_content: VP8The server is running the BigBlueButton WebRTC SFU and Kurento.
Additional context
The issue started approximately within the last month.
We are not aware of any intentional configuration, infrastructure, or software changes that coincided with the beginning of the problem.
We initially suspected that this might be related to the playback pipeline, but after checking the raw recorded media, we confirmed that the corruption is already present in the recorded video itself.
This makes us suspect that the issue may occur somewhere in the recording/capture/encoding pipeline rather than in the playback interface.
The server currently has approximately 16 GB RAM and 16 CPU cores. At the time of checking, overall system load and memory usage were not unusually high.
We would appreciate guidance on which recording/WebRTC/FFmpeg logs or diagnostic information should be collected to determine where the corrupted video is being introduced.
In particular, we would like to determine whether this could be related to:
- VP8 encoding
- FFmpeg
- Kurento
- WebRTC media/packet loss
- Recording pipeline
- Storage I/O
- Dropped or corrupted frames
- Another component involved in the recording process
We can provide additional logs and diagnostic information from an affected recording if needed.
Source: bigbluebutton/bigbluebutton