ci: smoke job fails intermittently on link-remove and repository-info tests unrelated to the change
Summary
The smoke (linux-x86_64) job in PR Validate is failing intermittently on tests unrelated to the change under test. Over the last week, on branches from several authors, each run fails a different one or two tests out of ~1040, and a rerun on the same commit passes.
Observed (PR Validate runs, last 7 days)
| Run | Branch | Failing test(s) |
|---|---|---|
| 35215639231 | bound-branch-push-lookups (#186) | test_repository_info_local_on_fresh_repo |
| 35215974397 | fix-empty-branch-revision-info (#184) | test_link_commit_per_link_message, test_link_remove_keeps_uncommitted_local_edit |
| 35216121252 | miri-growvec-ub (#210) | test_link_remove_keeps_uncommitted_local_edit |
| 35151934781 | miri-growvec-ub (#210) | test_link_remove_keeps_ignored_file_under_mount, test_link_remove_keeps_untracked_file_under_mount |
| 35140317945 | miri-growvec-ub (#210) | test_link_remove_keeps_untracked_file_under_mount |
| 35100409458 | ap/merge-commit-authorship-metadata-main | test_link_remove_refuses_when_path_case_differs, test_link_remove_keeps_uncommitted_local_edit, test_link_remove_keeps_untracked_file_under_mount |
| 35083243074 | fix-build-rs-unwrap | test_link_remove_refuses_when_path_case_differs, test_link_remove_keeps_staged_local_edit |
None of these branches touch link removal or repository info. #184 is a 12-line change to the empty-branch RevisionInfo handler; #186 changes branch-push verification; #210 is a lore-base GrowVec fix.
Pattern
- Nearly all failures are in
test_link_remove_*(scripts/test/test_link.py); one intest_repository_info_local_on_fresh_repo(test_compact.py), whererepository info --localreturned empty output. - The smoke job runs
pytest -m smoke -n 4with--basetemp=/dev/shm/lore-smoke(tmpfs, remounted to 12G). The link tests mount a second repository under the first; every failure so far is in a test that runs a secondloreserver/repository alongside the first, which points at a per-worker isolation or timing interaction rather than a product regression. - Reruns on the same commit pass, and different tests fail each time.
Impact
Fork PRs cannot be rerun by their authors, so each flake needs a maintainer rerun before a PR can go green. Happy to help dig into the link tests if useful; pointing at the isolation problem is as far as I've got.
CC @ragnarula, since #132 is yours, and #210 just failed on this
Source: EpicGames/lore