[dialog] Closing a nested Dialog dismisses its parent inside a closed shadow root

Summary

When the app is rendered inside a closed shadow root, closing a nested Dialog also dismisses its parent Dialog with reason: "outside-press".

Version: @base-ui/[email protected] (nothing in the 1.8.0 changelog looks related). Chrome 152.

Steps to reproduce

1. Render the app inside host.attachShadow({ mode: "closed" }).
2. Open a modal Dialog, and from inside its Popup open a second Dialog.
3. Click the nested dialog's Dialog.Close button (or the close button in its popup).

Expected: only the nested dialog closes. Actual: both dialogs close. The parent's onOpenChange fires with reason: "outside-press" and event.type === "click".

Escape and clicking the backdrop behave correctly — only a click that lands inside the nested dialog reproduces it.

const host = document.createElement("div")
document.body.append(host)
createRoot(host.attachShadow({ mode: "closed" })).render(<App />)

function App() {
  const [outer, setOuter] = React.useState(true)
  const [inner, setInner] = React.useState(false)
  return (
    <Dialog.Root open={outer} onOpenChange={setOuter}>
      <Dialog.Portal>
        <Dialog.Backdrop />
        <Dialog.Popup>
          <button onClick={() => setInner(true)}>open nested</button>
          <Dialog.Root open={inner} onOpenChange={setInner}>
            <Dialog.Portal>
              <Dialog.Backdrop />
              <Dialog.Popup>
                <Dialog.Close>close</Dialog.Close>
              </Dialog.Popup>
            </Dialog.Portal>
          </Dialog.Root>
        </Dialog.Popup>
      </Dialog.Portal>
    </Dialog.Root>
  )
}

Switching only mode: "closed" to mode: "open" makes the problem disappear.

What I found

useDismiss registers its outside-press listeners on ownerDocument(floatingElement). From a listener outside the shadow tree, a closed shadow root truncates event.composedPath(), so getTarget(event) (internals/shadowDom.js) returns the shadow host rather than the clicked element.

Two consequences for the parent dialog:

• isEventTargetWithin(event, floating) is composedPath().includes(node), which can never be true — the "press was inside" checks cannot fire.
• In the dialog's outsidePress predicate (dialog/root/useDialogRoot.js), the fallback contains(target, popupElement) is true, because the host is an ancestor of every popup. So the press is classified as outside.

The remaining guard is isTopmost (ownNestedOpenDialogs === 0). It does not hold here because closeOnPressOutsideCapture defers the decision via addTargetEventListenerOnce, and that listener is attached to the retargeted host. The host is an ancestor of the React root inside the shadow tree, so by the time it runs, React has already handled the click, closed the nested dialog, and reset the parent's ownNestedOpenDialogs to 0 — the parent is topmost again and dismisses itself.

A FloatingTree-based check would not help either, since isEventWithinFloatingTree also relies on composedPath().