[Bug]: Plugins share the host event loop — one synchronous call in any plugin freezes the whole instance (no contract, no monitoring, no isolation)

QwenPaw Version

2.2.0 and 2.2.1 — reproduced on both (managed cloud runtime, qwenpaw.platform.agentscope.io, Docker container).

Description

A locally installed plugin performing synchronous I/O on the event loop thread froze the entire instance for ~40 s: every agent, every channel, and all HTTP/SSE requests stalled. The user-visible symptom was a Console page stuck on "Loading Console" forever, with no log clue at all — only restarting the process recovered it.

This is not one plugin's problem. It is structural: plugins run on the host's event loop, with no non-blocking contract, no detection, and no isolation. The same failure is available to any plugin.

Concrete case (plugin is public): chcsyf/qwenpaw-web-terminal v0.2.2 does exactly this —

• https://github.com/chcsyf/qwenpaw-web-terminal/blob/082736db5774c8646e4e616b04b34d50267aea44/plugin.py#L648 — select.select([master_fd], [], [], 0.1) called synchronously from the async def _pty_loop body, every ~0.02 s
• https://github.com/chcsyf/qwenpaw-web-terminal/blob/082736db5774c8646e4e616b04b34d50267aea44/plugin.py#L653 — os.read(master_fd, 4096) on the blocking master fd returned by pty.openpty(): a genuinely blocking read, on the event loop
• https://github.com/chcsyf/qwenpaw-web-terminal/blob/082736db5774c8646e4e616b04b34d50267aea44/plugin.py#L239 — subprocess.run(cmd, timeout=…) inside async def exec_cmd (up to 300 s)

Related PR(s): N/A

Security considerations: Not remotely exploitable — plugins are installed locally. But the blast radius is the entire instance (all agents, all channels), and plugins can come from third parties / a catalog, so under an untrusted-plugin model this becomes a security-boundary issue. The failure mode is also indistinguishable from a generic frontend hang, which sends users down the wrong debugging path for days.

Component(s) Affected

• Core / Backend (app, agents, config, providers, utils, local_models)
• Console (frontend web UI)
• Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
• Skills
• CLI
• Documentation (website)
• Tests
• CI/CD
• Scripts / Deploy

Environment

• QwenPaw version: 2.2.0 / 2.2.1
• OS: Linux (managed Docker container)
• Install method: Docker (managed platform runtime)
• Python version (if applicable): 3.11

Steps to Reproduce

A. With the real plugin (public, no private code needed)

1. Install qwenpaw-web-terminal v0.2.2 (link above) as a local backend plugin.
2. Open its Console page (…/plugin/qwenpaw-web-terminal) so the PTY read loop is running.
3. While the loop is busy, hit any other endpoint — e.g. GET /api/console/push-messages or /api/console/inbox/events.
4. The whole instance stops answering for tens of seconds; the Console shows only its boot screen; channels go silent; SIGTERM does not complete.

B. Minimal synthetic repro — one blocking call in an async handler is enough

# plugin.py
import time
from fastapi import APIRouter

router = APIRouter()

@router.get("/block")
async def block():
    time.sleep(40)          # synchronous: blocks the host event loop for 40s
    return {"ok": True}

class Plugin:
    def register(self, api) -> None:
        api.register_http_router(router, prefix="/repro-block", tags=["repro"])

Install it and curl http://127.0.0.1:8088/api/repro-block/block, then try to load the Console. os.read/os.write on a blocking fd, subprocess.run, requests.* — any of them works equally.

Actual vs Expected

• Actual: any synchronous call in a plugin stalls the whole host — all agents, all channels, all HTTP/SSE. No log entry, no attribution, no degradation, no self-recovery; only a manual restart.
• Expected: a blocking plugin should not be able to take the host down. At minimum the stall should be detectable and attributable to a plugin.

Logs / Screenshots

There is nothing to paste — that is part of the problem. No warning, no error, no traceback is emitted when this happens. Observed from outside, on 2.2.1 while the Console was stuck on the boot screen with the plugin page open:

# DevTools -> Network, requests made from the plugin page
GET  /api/console/push-messages            200  0.1 kB   140 ms    <- works
GET  /api/console/push-messages            (已取消)  0.0 kB   30.00 s  <- stops answering
GET  /api/console/inbox/events?limit=1...  (已取消)  0.0 kB   30.00 s
GET  /api/console/push-messages            (已取消)  0.0 kB   30.01 s
GET  /api/console/push-messages            (已取消)  0.0 kB   30.01 s   (x many)
GET  <plugin>/stream?session=default       (待处理)  eventsource        <- stays pending
GET  <plugin page>                         (待处理)  document

# DevTools -> Console (same moment)
Failed to load agents:
Message not found for content: null   (repeated)

Note that the stall is not limited to the plugin's own routes: unrelated Console endpoints (/api/console/*) hang and are cancelled at exactly 30.00 s. Reference request headers confirm the page in question is the plugin page (Referer: …/plugin/qwenpaw-web-terminal, Accept-Encoding: gzip, deflate, br, zstd).

Verified in 2.2.x: slow_callback_duration, loop.set_debug and PYTHONASYNCIODEBUG appear nowhere in the package. The only faulthandler.enable is in the desktop sidecar (tauri/sidecar_logging.py), not the server path.

Additional Notes

Four structural gaps (verified in 2.2.x):

1. No fault isolation — plugin routers are mounted with app.include_router (plugins/registry.py) and hooks are awaited directly: no worker-thread offload, no timeout.
2. No API contract — plugins/api.py carries many docstring constraints (e.g. instances must not be shared) but nothing forbids blocking the loop, and nothing enforces it. Note: plugins/module_isolation.py is import-namespace isolation (for #6683's bare-import collisions), not fault isolation — please don't read it as covering this.
3. No observability — see Logs above.
4. No circuit breaking — an offending plugin is neither disabled nor degraded, and nothing is reported in the UI.

Suggested fixes, cheapest first:

1. Loop-lag watchdog with a real stack dump. Use faulthandler.dump_traceback_later(seconds, repeat=True) — it runs on a separate watchdog thread and captures the actual blocking frame (e.g. plugins/<id>/plugin.py:NNN). A pure asyncio.sleep-based lag probe only notices the stall after it ends, when the blocking frame is already gone; asyncio's slow-callback warning attributes better (task + definition site) but still has no stack.
2. Attribute stalls to a plugin — wrap hook/router invocation, bucket by plugin id, show it on the plugin page.
3. Optional circuit breaker — if a dumped stack top falls inside plugin_<id>, auto-disable that plugin and warn.
4. Document the contract in plugins/api.py ("never block the event loop; use to_thread for sync I/O") and ship a helper so authors get a correct default.
5. Longer term: process isolation for plugins.

Same failure class as #7786 (NFS deployment, 5–6 min freeze), #7721, #7363, and the startup-time variant #4946. Precedent for the direction: #5074 (merged) — perf(backend): Unblock Event Loop & Parallelize Startup.