Automatically backgrounded shell commands survive cancellation

Description

Crush appears to leave automatically backgrounded shell commands running after the work that started them has been cancelled. This is particularly problematic for broad filesystem commands: a cancelled recursive search can continue (almost) indefinitely, consume CPU, and access remote or virtual filesystems without any visible indication.

This report is based on an incident in which Crush initiated recursive searches from /. The searches continued for approximately two days after the operation was cancelled, repeatedly traversing a GVFS-backed SMB mount and causing substantial CPU usage in gvfsd-smb and gvfsd-fuse, plus high load on the remote NAS.

Observations

The workstation was otherwise idle, but top showed:

612345 testuser  29   9  536424  38820  19364 S  23.5   0.1 690:25.97 gvfsd-smb
6812    testuser  29   9 3102064   2.5g   5908 S  11.8   7.9 477:49.36 gvfsd-fuse

The SMB daemon had an established connection to a private-network SMB server:

TCP workstation.example.invalid:49152->smb-server.example.invalid:microsoft-ds (ESTABLISHED)

Tracing the FUSE requests showed a continuous stream of LOOKUP requests for changing .mp4 filenames. The FUSE request payloads included the requester PID. The relevant requests identified these processes:

612901  find / -path */project/providers/backend -name *.go
612777  find / -path */project*/providers/backend* -type d

The processes were still present when inspected:

$ ps -o pid,ppid,user,etime,stat,args -p 612901,612777
    PID    PPID USER         ELAPSED STAT COMMAND
 612777 612776 testuser   1-21:48:15 SN   find / -path */project/providers/backend -type d
 612901 612900 testuser   1-21:47:28 SN   find / -path */project*/providers/backend* -name *.go

$ pstree -aps 612901
systemd,1
  └─systemd,7042 --user
      └─zsh,612900 -cl...
          └─find,612901 / -path */project*/providers/backend* -name *.go

SN indicates sleeping and niced; these were not kernel zombies; they were live, long-running processes. Their parent shell was a zsh launched in the user session, consistent with Crush's shell execution path. If the user cancels the model operation, the UI operation ends, but the command can continue running indefinitely.

The user had no obvious indication that the cancelled operation was still active. The eventual symptom appeared as unexplained CPU usage (gvfsd-smb and gvfsd-fuse) rather than as anything Crush related.

Root Cause Theory

The current bash tool starts synchronous commands with a detached context so they can be moved to the background:

// internal/agent/tools/bash.go:303-310
// Start synchronous execution with auto-background support
startTime := time.Now()

// Start with detached context so it can survive if moved to background
bgManager := shell.GetBackgroundShellManager()
bgManager.Cleanup()
bgShell, err := bgManager.Start(context.Background(), execWorkingDir, blockFuncs(), params.Command, params.Description)

The command is automatically handed off after the configured threshold:

// internal/agent/tools/bash.go:318-340
autoBackgroundAfter := cmp.Or(params.AutoBackgroundAfter, DefaultAutoBackgroundAfter)
autoBackgroundThreshold := time.Duration(autoBackgroundAfter) * time.Second
timeout := time.After(autoBackgroundThreshold)

...

case <-timeout:
    stdout, stderr, done, execErr = bgShell.GetOutput()
    break waitLoop
...
case <-ctx.Done():
    bgManager.Kill(bgShell.ID)
    return fantasy.ToolResponse{}, ctx.Err()

Once the timeout path is taken, the tool returns the job as a background job. Thus cancellation after automatic backgrounding cannot reach the detached shell through the ctx case above.

The process-group termination implementation looks fine for ordinary Unix commands:

// internal/shell/exec_unix.go:64-76
stopf := context.AfterFunc(ctx, func() {
    if killTimeout <= 0 {
        _ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
        return
    }
    _ = syscall.Kill(-cmd.Process.Pid, syscall.SIGINT)
    time.Sleep(killTimeout)
    _ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
})

This suggests the primary issue is not necessarily a failure to kill a process group during an active cancellation. Rather it's the lifecycle boundary introduced by automatic backgrounding: the process is intentionally detached from the original tool context, but there is no apparent session-level cleanup or user-visible ownership mechanism that guarantees eventual termination.

Version

v0.92.0

Environment

Pop_OS 22.04 (kernel 7.0.11-generic x86_64)