Provide a reliable way to report output from multithreaded and parallel tests
TL;DR: I propose to:
- Add an API to report stdout and stderr from tests using the
TestReporter, different frompublishEntry("stdout", "..."). For example, providePrintStreams likeTestReporter.outandTestReporter.errso the test authors can doTestReporter.out.println("...")instead ofSystem.out.println(). - Add a callback to
TestExecutionListenerto get streaming output coming from these new streams (as proposed in https://github.com/junit-team/junit5/issues/4317), or maybe some other reporting keys (different fromstdout/stderr) to get this output inreportingEntryPublished.
At the moment, JUnit doesn't recognize output from other threads than the original test thread, or at least it doesn't attribute them to the test it originates from. As we can see in the docs:
Please note that the captured output will only contain output emitted by the thread that was used to execute a container or test. Any output by other threads will be omitted because particularly when executing tests in parallel it would be impossible to attribute it to a specific test or container.
Properly dealing with output from System.out.println() on JUnit side would require something like Kotlin's CoroutineContext, and I'm not sure there is an equivalent in pure Java. So I get that it might be effectively impossible for JUnit to attribute regular output to a test (when using parallel execution).
Nevertheless, test authors need a reliable way to print things from multithreaded tests while also ensuring it will be attributed to the proper test in reports and in listeners if run in parallel. While using System.out and System.err is not possible, nothing prevents JUnit from providing a TestReporter.out / TestReporter.err mechanism, as mentioned in this comment: https://github.com/junit-team/junit5/issues/780#issuecomment-303146311
Alternatives that don't seem appropriate:
TestReporter.publishEntry(String)just publishes regular metadata under thevaluekey. Tools cannot really decide whether this should just be printed to the console, or added to the metadata of the test with other key-value pairs.TestReporter.publishEntry("stdout", "...")seems like the perfect candidate at first glance, because it's exactly what is used by the stream capture mechanism to report the captured output. One problem with this is that it's unclear whether the same key can be used repeatedly for each line of output (a key-value store probably accepts only one value per key?). More importantly, theStreamInterceptingTestExecutionListenerreports the whole captured output in bulk at the end of the tests, which means that the output from background threads would come first, and then the output from the test thread will appear at the end, giving a very confusing ordering. See https://github.com/junit-team/junit5/issues/4317- Also, more generally, the
publish*APIs seem mostly meant for reports / report entries (semantically), and don't feel appropriate to stream output to test listeners
With a special way to print output via the TestReporter, the TestExecutionListener could have a new (pair of) callback(s) to receive stdout/stderr events that come either from the captured System.out/System.err from the main test thread, or from these new print streams. See https://github.com/junit-team/junit5/issues/4317.
Alternatively, this could be achieved by piggy-backing on the existing reportingEntryPublished and using some new keys like stdout-stream and stderr-stream.
Source: junit-team/junit-framework