Feature request: helper for iterating master–detail views that have no URL

The pattern

Catalogue, archive and admin-panel sites often render a list where clicking a row fills a detail area in place. The URL never changes, so the detail has no address of its own.

That rules out the usual shortcuts:

• No per-item endpoint to call — the detail isn't separately addressable
• Not in the initial HTML — it's filled after the click
• capture_xhr covers the case where each detail is its own request, but not when many rows share one endpoint, or the panel is rendered server-side

The only way to reach the data is to drive the UI: click a row, read the panel, go back, repeat.

What I write today

page_action is enough to do it, but it's a raw escape hatch — it hands over the page and steps out of the way:

# scrapling/engines/_browsers/_controllers.py
_ = params.page_action(page)

So the same loop gets rewritten for every site. Mine, reduced to its shape:

for (let i = 0; i < 5; i++) {
  // re-queried every iteration: references taken before the back-navigation are detached
  const rows = document.querySelectorAll('.list .row');
  const row = rows[i];
  if (!row) continue;

  row.click();
  await sleep(2000);                 // hope the panel filled

  const detail = document.querySelector('#detail');
  if (detail) {
    results.push([...detail.querySelectorAll('a.file')].map(a => a.href));
  }

  document.querySelector('#back')?.click();
  await sleep(2000);                 // hope the list came back
}

It works, and I've used this shape across a number of sites. But every fragile part is hand-rolled, and it's the same fragile parts each time.

Proposed API

rows = page.iterate_detail(
    list_selector='.list .row',
    detail_ready='#detail',
    back_selector='#back',
    extract=lambda d: [a.attrib['href'] for a in d.css('a.file')],
    key='data-id',
    limit=50,
    adaptive=True,
)

The parts that are actually hard

This is where a library would earn its keep, and where my hand-rolled version is either lucky or quietly wrong:

1. Detached references. The lists I work with re-render after every back-navigation — in one case the container's id is a freshly generated GUID each time — so any reference taken before the click is detached by the time it's used. Clicking it silently does nothing, because the site attaches its handler to the container and a detached node's event never reaches it. No exception, no log, just missing rows.
2. Knowing the panel is ready. Fixed sleeps are simultaneously too slow and not slow enough. Worse, when the panel is slow the container still exists holding the previous row's content, so the extract succeeds and records the wrong data under the right title. Waiting for the content to change is what you actually want. wait_selector exists at fetch level, but not for waiting inside a page_action loop where the selector is already attached.
3. Knowing the list is restored before the next click.
4. Index drift. If the list re-renders in a different order, rows[i] is a different item and nothing tells you — you get duplicates and skips. Tracking by a key attribute instead of position fixes it. My version doesn't do this; it happens to work only because the lists I hit are sorted on a stable field.
5. Partial failure. One row failing shouldn't abort the run.

Scope, honestly: this covers a fully-rendered list with an explicit back control. Virtualised lists, pagination, modals and iframes each need their own handling — which is part of why it belongs in a library rather than in everyone's one-off script.

What I checked

On dev (11e0935) there's no match in scrapling/ for iterate_detail, master-detail, click_through, or any staleness handling, and page_action is still passed straight through in _controllers.py and _stealth.py. page.evaluate is used for the shadow-DOM snapshot but isn't exposed as a general hook. If I've missed an existing way to do this, I'm happy to be pointed at it.

Disclosure

Per the AI contribution policy: I wrote this issue with AI assistance (Claude Code). The problem, the JavaScript above and the list of failure modes come from my own scraping work; the AI helped me survey the codebase and phrase the proposal.

Happy to attempt a PR against dev if this fits Scrapling's scope.