[Bug]: Agent browser commands steal the user's keyboard focus or type into their terminal

Operating system

macOS (Apple Silicon)

Orca version

1.4.201

Details

Summary

While an agent drives the embedded browser, the user typing in a terminal pane in the same Orca window loses keyboard focus, or has the agent's keystrokes land in their terminal. Several issues describe parts of this (#8139, #16410, #12912, #19297, #16674, #20533). This one measures every browser command an agent typically uses, in a single controlled run on the current release, so the causes can be seen side by side.

The main finding: the problem has more than one code path, so a fix applied per command leaves some of them open. In particular orca mouse down steals focus exactly like orca click, but reaches the page through the bundled agent-browser binary rather than the click handler, so PR #10875 would not cover it. The closed PR #11732 handled focus at the CDP proxy that all routes pass through, which looks like the approach that would.

Measured results (1.4.201)

A human typed lowercase letters continuously (about 8 keys/s) into a terminal pane running a timestamped key logger. A script repeated one command six times per 12 second segment against a background browser tab showing a local file:// probe page, which logged its own keydown, mousedown and window focus events. The script only ever typed uppercase Z/Q, so those letters are attributable to automation.

Command	Human's typing	Detail
none (baseline)	stayed in terminal	longest pause 0.3s
eval, snapshot	stayed in terminal
fill	stayed in terminal
click	moved to the page	page got window focus on 6 of 6 presses; 23 of the human's keystrokes landed in the page
mouse move / mouse down / mouse up	moved to the page	page got window focus on 6 of 6 presses; 22 of the human's keystrokes landed in the page
type --input ZQ	stayed in terminal	the automation's text landed in the human's terminal 3 of 6 times and reached neither terminal nor page the other 3; the page input never received it
keypress --key Z	stayed in terminal	the key landed in the human's terminal 6 of 6 times, never in the page
tab switch --focus	view switched to the browser tab	expected for --focus, but the human's keystrokes after the switch reached neither the terminal nor the page until they clicked (same as #20533)
tab create	stayed in terminal	no longer steals focus on this version

All commands used --page <browserPageId>. No other agent was driving the browser.

The keypress row is the dangerous one in practice: an agent sending Escape to a page interrupts whatever agent session the user is typing into (as reported in #12912).

Where it comes from (from the 1.4.201 bundle)

• click: the main-process mouseClick handler calls webContents.focus() on the guest before dispatching Input.dispatchMouseEvent mousePressed, and nothing restores the previous focus.
• mouse down: mouseDown shells out to agent-browser (execAgentBrowser(['mouse','down'])), whose mousePressed goes through the CDP proxy unmodified. Chromium hands native focus to the guest on the press. There is no focus() call to remove on this path, which is why a handler-level fix misses it.
• type / keypress: key events only reach a <webview> guest that holds native focus; otherwise they go to whatever surface has it, which is the user's terminal.
• tab switch --focus: Page.bringToFront calls webContents.focus(), which (per #11732's measurements) does not give a guest DOM focus, consistent with keystrokes going nowhere.

Minimal repro

1. Save as probe.html and open it: orca tab create --url file:///path/to/probe.html --json.

   xml复制

   <input id="probe">
   <div id="box" style="position:absolute;left:40px;top:140px;width:160px;height:80px;background:#cde"></div>
   <script>
     window.__log = [];
     addEventListener('keydown', e => __log.push('key:' + e.key), true);
     addEventListener('mousedown', () => __log.push('mousedown'), true);
     addEventListener('focus', () => __log.push('winfocus'));
   </script>

2. Click into a different terminal pane in the same Orca window and keep typing.

3. From another shell, run any of these a few times, two seconds apart:

   bash复制

   orca mouse move --page $P --x 100 --y 180; orca mouse down --page $P; orca mouse up --page $P
   orca keypress --page $P --key Z

4. Result: after mouse down your typing stops appearing in your terminal and orca eval --page $P --expression "JSON.stringify(__log)" shows winfocus plus your keystrokes. After keypress, a Z appears in your terminal and the page log has no key:Z.

Expected

Browser automation targeted with --page should neither move the user's keyboard focus nor deliver input anywhere except that page. Where input cannot reach the page, the command should fail rather than report success.

Related

#8139, #16410, #12912, #19297, #16674, #20533, PR #10875 (open, click/drag/check only), PR #11732 (closed, proxy-level).