#2911·midscene

[Bug]: Android Studio Recorder captures stale ADB screenshots instead of the current scrcpy preview frame

Author: pengalCreated Jul 30, 2026Updated Jul 30, 2026

Version

bash
Midscene Studio: [fill packaged version, if available]
Source commit: 69a7bcadfef3ad0c7a5cc917d613c26f3f24245f

Details

When recording an Android workflow in Midscene Studio, the user interacts with the live scrcpy preview. However, the Recorder screenshot associated with the interaction does not come from the currently displayed scrcpy frame.

The Studio Android runtime currently creates two independent capture paths:

  1. A standalone ScrcpyServer sidecar provides the live Studio preview.
  2. The Recorder captures screenshots through agent.interface.screenshotBase64().

The Android Agent is created using new AndroidDevice(deviceId) without enabling scrcpyConfig. Therefore, Recorder screenshots use the normal ADB takeScreenshot / screencap path instead of the scrcpy stream displayed in Studio.

For the screenshot before an interaction, Recorder first tries the last MJPEG frame and then falls back to _studioPreviewRecorderLastScreenshot. When Android scrcpy preview is active, the renderer does not consume the Playground MJPEG endpoint. As a result, this fallback may be the screenshot captured when recording started or after the previous interaction.

This produces a mismatch:

  • The recorded coordinates refer to the current scrcpy frame visible to the user.
  • The screenshot used by Recorder and describeElementAtPoint may represent an older UI state.
  • The generated YAML/Markdown script can describe the wrong target or fail to replay.

Raw Tap/Input events are generally still recorded. The primary impact is incorrect screenshots, semantic element descriptions, and generated scripts.

This is related to #2712, but it is specifically caused by Studio's Android Recorder and scrcpy preview using separate frame sources.

Reproduce link

none

Reproduce Steps

  1. Open Midscene Studio and connect an Android target.
  2. Confirm that the live scrcpy preview is connected.
  3. Open an Android screen that automatically changes from UI state A to UI state B without a Studio interaction.
  4. Start Recorder while state A is visible.
  5. Wait until the scrcpy preview changes to state B.
  6. Tap an element that exists in state B.
  7. Stop recording.
  8. Inspect the screenshot stored for the Tap event and the generated element description/script.

Actual result:

  • The Tap coordinates correspond to state B.
  • The Recorder screenshot may still show state A—the screenshot captured at Recorder startup or after the previous interaction.
  • AI element description or generated script can reference the wrong element.

Expected result:

  • Recorder should use the current scrcpy frame visible when the interaction begins.
  • Screenshot dimensions and coordinates should use the same stream coordinate space.
  • If no valid scrcpy frame is available, Recorder may fall back to screenshotBase64()/ADB.