Guarantee that tests, test suites, and hook methods are bracketed by events

The event system does not guarantee that a test or a test suite that has been started is also finished. Test\Finished is only emitted for tests that were prepared successfully, TestSuite\Finished is not emitted for a test suite that is skipped in a "before first test" method, and some tests are reported with an outcome but without being started at all.

In addition, the *Called events for hook methods are emitted after the hook method has been invoked, not before. It is therefore impossible to tell when a hook method, or a phase of hook methods, started.

Every subscriber that needs to know when a test, a test suite, or a hook method begins and ends has to reconstruct this from other events, has to know undocumented rules for doing so, and cannot do so at all for some hook methods. PHPUnit implements the rules for tests several times itself.

I propose that PHPUnit 14 guarantees that a test or a test suite that is started is also finished, and that a hook method's *Called event is emitted before the hook method is invoked, so that subscribers can rely on a bracket of a start event and a finish event.

Current behaviour

Tests and test suites

The following sequences were observed with PHPUnit 13.3.5 using --debug, except for the two rows marked with an asterisk, which follow from the code paths listed below (the Test and Test Suite prefixes of the event names are left out):

Situation	Events	Missing
Test passes, fails, errors, is skipped, or is marked incomplete in the test method	Preparation Started, Prepared, outcome, Finished	(nothing)
Test errors in setUp() (or another "before test" method)	Preparation Started, Before Test Method Called, Before Test Method Errored, Before Test Method Finished, Preparation Errored, Errored	Finished
Test is skipped in setUp()	Preparation Started, Before Test Method Called, Before Test Method Finished, Skipped	Finished
Requirement not met (for instance #[RequiresPhp])	Preparation Started, Skipped	Finished
Test depends on a test that did not pass	Skipped	Preparation Started, Finished
Test depends on a test that does not exist*	Errored	Preparation Started, Finished
Remaining repetitions after a failed repetition*	Skipped	Preparation Started, Finished
Test suite is skipped in setUpBeforeClass()	Started, Before First Test Method Called, Skipped	Finished
setUpBeforeClass() errors	Started, Before First Test Method Errored, Before First Test Method Finished, Finished	(no events for the tests of the class)

The cases are caused by these code paths:

• PHPUnit\Framework\TestRunner\TestRunner::run() only emits Test\Finished when $test->wasPrepared() returns true.
• PHPUnit\Framework\TestCase::markSkippedForMissingDependency(), markErroredForInvalidDependency(), and markSkippedForRepeatAbort() emit an outcome for a test that was never started.
• PHPUnit\Framework\TestSuite::invokeMethodsBeforeFirstTest() emits TestSuite\Skipped and returns false without emitting TestSuite\Finished. When a "before first test" method errors or fails, the same method does emit TestSuite\Finished. This asymmetry looks unintentional.

Hook methods

PHPUnit\Framework\TestCase\HookMethodInvoker::doInvoke() invokes each hook method first and emits the *Called event for it afterwards. PHPUnit\Framework\TestSuite::invokeMethodsBeforeFirstTest() and invokeMethodsAfterLastTest() do the same for setUpBeforeClass() and tearDownAfterClass() methods. This affects BeforeFirstTestMethodCalled, BeforeTestMethodCalled, PreConditionCalled, PostConditionCalled, AfterTestMethodCalled, and AfterLastTestMethodCalled.

A *Called event therefore marks the point in time at which a hook method returned, not the point in time at which it was (about to be) called. DataProviderMethodCalled, in contrast, is emitted before the data provider method is invoked.

This has the following consequences:

• The duration of the first hook method of a phase cannot be determined, because no event marks its start.
• Issues that a hook method triggers, such as deprecations, are emitted before the *Called event for that hook method. The event stream suggests that they were triggered before the hook method was called.
• TestCase::runBare() invokes the "before test" and pre-condition methods, emits Test\Prepared, invokes the test method, and then invokes the post-condition and "after test" methods. No event marks the end of the test method. The only event that could mark the start of the post-condition phase or of the "after test" phase is the *Called event for its first hook method, and that event is emitted after that method has run. A subscriber cannot determine when these phases started: it can either leave out the first hook method or include the test method's own runtime.

For the other phases, a preceding event (TestSuite\Started, Test\PreparationStarted, Test\BeforeTestMethodFinished, or the last Test\Finished of the test class) happens to provide the start, but only by coincidence of the current order of events.

Why this is a problem

A subscriber that needs to know when a test ends has to:

• remember whether Test\Prepared was received,
• treat an outcome event as the end of a test when the test was not prepared,
• handle outcome events for tests that were never started, and
• treat TestSuite\Skipped as the end of a test suite, but only in some cases.

None of this is documented. It can only be learned by reading PHPUnit's source code or by tracing events.

PHPUnit's own subscribers implement this bookkeeping several times:

• PHPUnit\Runner\TestResult\Collector counts a test as run in testFinished(), but for tests that were not prepared it counts them in testErrored(), testFailed(), and testSkipped() instead.
• PHPUnit\Logging\JUnit\JunitXmlLogger creates the test case, and finishes it, in the outcome handlers when the test was not prepared, and ignores Test\Finished when preparation failed.
• PHPUnit\Logging\TeamCity\TeamCityLogger keeps the same $prepared and $preparationFailed state.

Third-party subscribers, such as loggers, IDE integrations, etc., need the same logic. Any subscriber that gets it wrong either leaks tests that never end or misses tests entirely.

For hook methods, there is no logic that a subscriber could implement: the information is not in the event stream.

Proposal

1. Every Test\PreparationStarted is followed by exactly one Test\Finished

Test\Finished is emitted for a test that was not prepared successfully as well. In that case, numberOfAssertionsPerformed() returns 0.

Test\Prepared continues to tell subscribers whether preparation succeeded, so no information is lost.

2. Every test that is reported with an outcome is started and finished

A test that is skipped or errored without being run, because of a missing or invalid dependency or because a repetition failed, is reported as Test\PreparationStarted, the outcome, and Test\Finished, all at the same point in time.

It is debatable whether PreparationStarted is the right event for a test that is never prepared. The alternative is to keep these outcomes outside of a bracket and document that. I prefer the bracket, because it means that subscribers do not need a special case, and making subscribers simple is the purpose of the event system.

3. Every TestSuite\Started is followed by exactly one TestSuite\Finished

TestSuite\Skipped becomes an event that is emitted before TestSuite\Finished, just as Test\Skipped is emitted before Test\Finished. This also removes the asymmetry between a "before first test" method that skips and one that errors or fails.

4. A hook method's *Called event is emitted before the hook method is invoked

HookMethodInvoker, TestSuite::invokeMethodsBeforeFirstTest(), and TestSuite::invokeMethodsAfterLastTest() emit the *Called event for a hook method before they invoke it, as is already the case for DataProviderMethodCalled. The *Errored or *Failed event for a hook method that errors or fails, and the *Finished event for a phase, remain where they are.

With this change:

• a hook method runs from its *Called event to the next *Called event of the same phase, or to the phase's *Errored, *Failed, or *Finished event,
• a phase of hook methods runs from the *Called event of its first hook method to its *Finished event,
• the test method ends where the post-condition or "after test" phase begins, whenever such a phase exists, and
• issues that a hook method triggers are emitted after the *Called event for that hook method.

Emitting an additional event before each invocation, and keeping the current *Called events, would also work.

Resulting guarantees

These guarantees would be documented as part of the event system's contract:

• A test's events are bracketed by Test\PreparationStarted and Test\Finished.
• A test suite's events are bracketed by TestSuite\Started and TestSuite\Finished.
• Test\Prepared is emitted only when preparation succeeded.
• A hook method's events are bracketed by its *Called event and the next *Called, *Errored, *Failed, or *Finished event of the same phase.

Backward compatibility

This changes the sequence of events that subscribers observe, so it belongs in a major version.

Subscribers that already compensate for the missing events will double-count or finish a test twice unless they are adjusted. PHPUnit\Runner\TestResult\Collector is an example: it would count a test that errored in setUp() twice, once in testErrored() and once in testFinished().

In PHPUnit itself, the adjustment consists of removing the compensating code from Collector, JunitXmlLogger, TeamCityLogger, and the other loggers, so the change should reduce the amount of code overall.

Emitting *Called before a hook method is invoked changes the point in time and the position in the event stream of these events, but not which events are emitted. Subscribers that only count or list the hook methods that were called are not affected. Subscribers that attribute issues to hook methods by their position in the event stream see issues after the *Called event instead of before it.

Out of scope: tests of a class whose setUpBeforeClass() errors

When setUpBeforeClass() errors, no events are emitted for the tests of the class, although TestSuite\Started reports how many tests the test suite contains.

Emitting events for these tests, for instance Test\Errored for each of them, would change the numbers in the test result, not only the shape of the event stream.