Assertion is not done against `isRepresentativeRun` when using `median-run`
Describe the bug
When using the median-run aggregation method I'd expect that assertions are done against the run, that has the property of isRepresentativeRun in the manifest.json. This does not appear to be the case.
Manifest:
{
"url": "http://localhost:3000/",
"isRepresentativeRun": true,
"htmlPath": "frontend/.lighthouseci/localhost-_-2024_07_26_09_57_40.report.html",
"jsonPath": "frontend/.lighthouseci/localhost-_-2024_07_26_09_57_40.report.json",
"summary": {
"performance": 0.68,
"accessibility": 0.98,
"best-practices": 1,
"seo": 1
}
},However, when correlating this with the assertion-resutls.json it is apparent, that a different run was used for assertion.
[
{
"name": "minScore",
"expected": 0.7,
"actual": 0.69,
"values": [
0.57,
0.69,
0.68
],
"operator": ">=",
"passed": false,
"auditProperty": "performance",
"auditId": "categories",
"level": "error",
"url": "http://localhost:3000/"
}
]To Reproduce Steps to reproduce the behavior: Use run LHCI with filesystem target, using the following assertion:
{
matchingUrlPattern: `^${PREVIEW_URL}/$`,
assertions: {
'categories:performance': ['error', { minScore: 0.7, aggregationMethod: 'median-run' }],
'categories:accessibility': ['error', { minScore: 0.9, aggregationMethod: 'median-run' }],
'categories:best-practices': ['error', { minScore: 0.9, aggregationMethod: 'median-run' }],
'categories:seo': ['error', { minScore: 0.9, aggregationMethod: 'median-run' }],
},
},Expected behavior
Assertion should be done against the run with the isRepresentativeRun property in the manifest.json
As per the description of median-run in the Aggregation methods section of the config documentation:
Use the value of the run that was determined to be "most representative" of all runs...
Environment (please complete the following information):
- OS: macOS, ubuntu
- Version:
@lhci/[email protected]
Source: GoogleChrome/lighthouse-ci