Redesign the test reordering subsystem

Motivation

Test reordering used to be implemented by TestSuiteSorter as three orthogonal knobs:

• a main order: default, reverse, random, duration-ascending, duration-descending, size-ascending, size-descending
• a defects overlay: on/off
• a dependency resolver: on/off

Both the XML executionOrder attribute and the CLI --order-by option accept a comma-separated list of tokens, where each token flips one of these three knobs. TestSuiteSorter::sort() then applied them in a hard-coded three-phase pipeline:

1. main order
2. defects-first overlay (stable, so the main order is preserved inside each defect tier)
3. dependency resolution

Three bugs in that implementation were fixed separately (equal-weight tie-breaking inverting the preceding phase, skipped and incomplete tests being treated as defects, and a plain TestSuite always sorting as unknown size). What remained was a design problem:

• The comma-separated list looks like a pipeline but is not one. Phase order is hard-coded. A user cannot express "sort by duration, then hoist the defects", even though the token list duration-ascending,defects reads exactly like that. The tokens are a set of flags, and their written order is discarded.
• Contradictory tokens are silently accepted. duration-ascending,duration-descending resolves to last-wins with no diagnostic. depends,no-depends does the same.
• A misspelled XML token disappears. The XML loader had no default branch, so an unknown token was ignored entirely. The schema does reject it, but only as a validation warning.
• The defects notion is opaque. Which statuses count as a defect, and with what weight, was decided by TestStatus::sortWeight(), a method on a status value object that has no business knowing about ordering policy.
• There is no way to see what reordering will be applied. The Test Suite Sorted event carried the three raw knob values, which is not the same as the effective pipeline, and the test run history that drives the ordering could only be inspected by reading its JSON by hand.
• There is no single source of truth for the token semantics. The CLI builder and the XML loader each carried their own copy of the same switch, which is how they came to disagree on what to do with an unrecognized token.
• Dependency resolution is not an ordering strategy, but it is spelled like one. depends and no-depends are tokens in the same list as random and size-ascending, even though there has always been a dedicated resolveDependencies XML attribute and a dedicated pair of CLI options for exactly this.

The goal is an ordering subsystem where the configuration surface round-trips cleanly to an ordered pipeline of well-defined stages.

Implemented in PHPUnit 13.4

Everything below is either invisible to users or purely additive. No configuration that works today produces a different test order in 13.4.

The pipeline exists

A new PHPUnit\Runner\ExecutionOrder namespace holds the model the redesign is built on:

• ReorderStage, an interface with apply(list<Test> $tests, Context $context): list<Test> and name(): string
• ReorderPipeline, an ordered list of stages, with fromConfiguration(), apply(), isEmpty() and describe()
• Context, carrying the current TestSuite and the TestRunHistory
• the stages ByDuration, BySize, Reverse, Randomize, DefectsFirst and ResolveDependencies

TestSuiteSorter is now only the driver that walks the tree of test suites depth first and applies the pipeline at each level. The stage order it compiles is still the fixed main-order, defects, dependencies sequence, so behavior is unchanged. The existing end-to-end tests for execution order pin that.

DefectsFirst takes a DefectWeightPolicy. DefaultDefectWeightPolicy implements the framework rule that only errors and failures are defects. TestStatus::sortWeight(), which encoded that rule before, is gone. TestStatus is @internal, so this needed no deprecation cycle.

One parser for both configuration surfaces

ExecutionOrderParser is now the only place where the tokens are given meaning. The CLI builder and the XML loader both call it and differ only in how they report an unknown token: the CLI throws, as it always did, and the XML loader now says something instead of staying silent.

Diagnostics

Five new PHPUnit deprecations, each announcing a PHPUnit 14 change:

The defects one is the reason the deprecation cycle exists at all. See "Why the spelling has to change" below.

depends and no-depends move to their dedicated options

Whether dependencies between tests are resolved is not an ordering strategy, and it never needed to be a token in the order list. The resolveDependencies XML attribute and the --resolve-dependencies and --ignore-dependencies CLI options already express it, so depends and no-depends are now deprecated in favour of them.

This removes a whole class of confusion from the redesign. Once the tokens are gone the token list is purely about ordering, and no token in it needs a special position rule.

--help no longer lists depends and no-depends under --order-by, and the two CLI options are documented in their own right rather than as aliases for a deprecated spelling.

Event payload

PHPUnit\Event\TestSuite\Sorted gained pipeline(), which returns the stage names in the order they were applied. This is additive: the constructor is @internal, the three existing accessors are untouched, and asString() still returns Test Suite Sorted, so none of the 435 event log expectations in the test suite had to change because of it.

Schema and the configuration migrator

executionOrderType went from 42 values to 14. Every spelling that is now deprecated was removed from it, and the six reversed spellings that replace them were added:

default
defects
duration-ascending          duration-ascending,defects
duration-descending         duration-descending,defects
random                      random,defects
reverse                     reverse,defects
size-ascending              size-ascending,defects
size-descending             size-descending,defects

Removing the deprecated values from the current schema rather than leaving them in is what makes the migrator work, and it is what PHPUnit already did for duration and size in 13.2. SchemaDetector picks the newest schema a document validates against and ignores the declared schemaLocation, so a value that is still in the current schema can never route a file to the migrator. A configuration using a deprecated spelling now validates against the 13.3 schema instead, which produces the familiar prompt:

Your XML configuration validates against a deprecated schema.
Migrate your XML configuration using "--migrate-configuration"!

Two migrations registered under 13.4 do the work:

• MoveDependencyResolutionOutOfExecutionOrder takes depends and no-depends out of the attribute and writes resolveDependencies instead. It overwrites any existing resolveDependencies, because the tokens used to take precedence over it, and it drops the executionOrder attribute entirely when nothing else was in it.
• MoveDefectsAfterOrderInExecutionOrder moves defects to the end.

So executionOrder="depends,defects,duration-ascending" migrates to executionOrder="duration-ascending,defects" with resolveDependencies="true".

PHPUnit's own phpunit.xml was migrated from depends,defects,duration-ascending to duration-ascending,defects. The depends token was redundant there anyway, since resolveDependencies defaults to true.

Waiting for PHPUnit 14

Why the spelling has to change

Once the stage order follows the token order, defects,duration-ascending means "hoist the defects, then sort everything by duration", and the duration sort erases the hoist. The token becomes a no-op. The useful pipeline is the other way round: sort first, then hoist.

That is a silent, user-visible change to every configuration of that shape, and the schema currently only enumerates the shape that inverts. So 13.4 accepts both spellings and deprecates the old one, and 14 starts honoring the token order. A project that acts on the deprecation sees no change in behavior at all when it upgrades.

The rest of the 14.0 scope

• Remove the depends and no-depends tokens. Dependency resolution is then configured only through resolveDependencies, --resolve-dependencies and --ignore-dependencies, and it is applied after whatever the order list produced. Nothing in the order list needs a position rule any more.
• Honor token order for the main order and defects. With the dependency tokens gone this is the only interaction left.
• Turn the remaining deprecations into configuration errors.
• Remove the duration and size tokens, as already announced for #6075.
• Remove the old spellings where defects precedes the order.
• Decide what default resets. It currently switches dependency resolution back on as a side effect. Once dependency resolution has left the order list, default should reset the ordering only.
• Collapse the public configuration surface. Configuration::executionOrder(), executionOrderDefects() and resolveDependencies() become one accessor that returns the pipeline. The matching three accessors on the Sorted event go with them.
• Change executionOrderType from an enumeration of every legal combination to a list pattern.

Open questions, resolved

Should DefectsFirst be weight-customizable through XML or CLI? No. Only two statuses carry a non-zero weight, so the surface is much smaller than it looked, and exposing it invites bikeshedding for no concrete use case. DefectWeightPolicy exists as an internal seam with a single implementation and can be opened up later if a real request arrives.

Should the test suite comparator aggregate children by max(), sum(), or a leaf count? Keep what is there, and document it: max() for defect weight, sum() for duration. Both were settled deliberately while fixing the three bugs above. Changing them again needs a motivating bug report, not a redesign.

Can ResolveDependencies be generalized so #[Depends] always influences the order? It effectively already is: resolveDependencies defaults to true, so dependency resolution happens unless a project opts out. Keep the opt-out, because --ignore-dependencies is a documented way to expose hidden coupling between tests. What changed is where it is configured. It is no longer a token competing with ordering strategies, it is the dedicated option it always should have been, and the stage it drives is always applied last.