System audio recording loses silent gaps, causing progressive desynchronization with microphone

What happened?

When recording both the microphone and system audio on Windows, the system-audio track appears to lose periods during which no audio is being played.

Instead of preserving those periods as silence, the next system-audio samples are appended immediately after the previous ones.

As a result, the system-audio timeline becomes shorter than the microphone timeline and progressively shifts earlier relative to the microphone.

I can reproduce this consistently in Vibe 3.2.2.

This may not be very noticeable during typical Teams, Google Meet, etc. meetings, because these applications usually keep an audio stream active throughout the meeting. It becomes much easier to reproduce when the system audio source genuinely stops producing audio for a while.

Steps to reproduce

1. Start a Vibe recording with both:
   • microphone
   • system audio / speakers
2. Start playing an audio file through the system output while also speaking into the microphone.
3. Stop playback of the audio file for a few seconds, while keeping the Vibe recording running.
4. Speak into the microphone during this period.
5. Resume playback of the audio file.
6. Repeat the stop/resume sequence a few times.
7. Stop the Vibe recording and listen to the resulting audio.

The system audio progressively moves earlier compared with the microphone.

Current behavior

Conceptually, the captured streams behave like this:

Real time:

0s          5s          10s         15s         20s
|-----------|-----------|-----------|-----------|

Microphone:
[========== continuous recording ==================]

System playback:
[ AUDIO A ][  no system audio  ][ AUDIO B ][silence][ AUDIO C ]


Current system-audio WAV produced by Vibe:

[ AUDIO A ][ AUDIO B ][ AUDIO C ]
           ^
           silent periods are missing

Resulting timelines:

Microphone:
0s ------------------------------------------------ 20s

System audio:
0s ----------------------------- 14s

After each silent period, subsequent system audio is shifted earlier.
The offset therefore accumulates over the recording.

Expected behavior

Periods without system audio should still occupy time in the recorded system-audio track.

Real time:

0s          5s          10s         15s         20s
|-----------|-----------|-----------|-----------|

Microphone:
[========== continuous recording ==================]

System playback:
[ AUDIO A ][  no system audio  ][ AUDIO B ][silence][ AUDIO C ]


Expected system-audio WAV:

[ AUDIO A ][     SILENCE       ][ AUDIO B ][SILENCE][ AUDIO C ]

Resulting timelines:

Microphone:
0s ------------------------------------------------ 20s

System audio:
0s ------------------------------------------------ 20s

Both tracks keep the same timeline and remain synchronized.

Additional observation

I performed a second test where I kept another very low-volume audio stream playing continuously through the system output during the whole recording.

With a continuous system-audio stream present, I could no longer reproduce the desynchronization: microphone and system audio remained synchronized.

This seems to suggest that the issue occurs specifically when the Windows system-audio / loopback capture temporarily stops providing audio buffers.

Possible cause

From what I understand from the current Vibe recording implementation, microphone and system audio are captured independently through CPAL and written to separate WAV files before being mixed with FFmpeg.

The audio callback writes the samples it receives directly to the WAV file, but the callback timing information does not appear to be used to preserve gaps between buffers.

Therefore, if WASAPI loopback does not provide buffers while nothing is being rendered, no samples are written during that period. When audio resumes, the new samples are appended directly after the previous samples, effectively removing the silent interval from the timeline.

The relevant callback appears to be approximately:

move |data: &[T], _: &_| {
    meter.push(buffer_peak(data));
    write_input_data::<T, T>(data, &writer)
}

The callback timing information is currently ignored.

FFmpeg's amix=...:duration=longest in Vibe 3.2.2 prevents the complete output from simply being truncated to the shortest input, but it cannot restore silent intervals that are already missing from one of the source WAV files.

Possible fix

One possible approach would be to maintain a real-time timeline for each captured stream.

When a new system-audio buffer arrives, Vibe could compare its timestamp with the expected timestamp based on the previously written samples.

If there is a gap, the corresponding number of zero-valued samples could be inserted before writing the new buffer.

Conceptually:

previous buffer
      |
      v
[AUDIO A]

      <----- elapsed time with no loopback buffers ----->

                                             new buffer
                                                  |
                                                  v
                                               [AUDIO B]


Current:
[AUDIO A][AUDIO B]

Expected:
[AUDIO A][000000000000000000000000000000000][AUDIO B]
          ^ inserted silence preserving real time

Another possible workaround would be to keep the loopback capture active with a continuous silent render stream, but reconstructing missing time from capture timestamps would probably preserve the timeline more explicitly.

Environment

• Vibe: 3.2.2
• Platform: Windows