#19002·jupyterlab

Use linter rules to reduce flakiness of UI tests

Author: krassowskiCreated Jun 9, 2026Updated Sep 6, 2026
Labelsmaintenance

Problem

UI tests can be flaky if written without sufficient attention (and AI does not seem to be very good at it). Recent work e.g. in #18961 #18856 #18924 #18922 #18927 #18453 #18492 #18495 #18506 #16072 reduced the number of flaky tests down to single digits but as new tests get added it is hard to ensure they are not flaky when runner slows down.

Proposed Solution

  1. If possible, encode good practices as documented in https://jupyterlab.readthedocs.io/en/latest/developer/contributing.html#good-practices-for-integration-tests with linter rules
  2. Review rules from eslint-plugin-playwright to see which could help with flaky or hard to debug tests.

The default recommended rules of eslint-plugin-playwright seem too much, but some of them are nice; if we can enable something which requires only a few minor changes then maybe we should. Otherwise, we can reimplement things in @jupyter/eslint-plugin if it is trivial and avoid adding another dependency.

Additional context

We already encoded three of the best practices as rules:

https://github.com/jupyterlab/jupyterlab/blob/9a086de89048b9d87be7abf3b5343ed1fb2fd904/eslint.config.mjs#L494-L510

Two best practices which should be easy to encode as rules would be:

  • no waitForTimeout() (in new code, we probably don't want to change it everywhere at once)
  • no toMatchSnapshot() on entire page in new code; maybe also same for the entire notebook? not sure yet