Native support for property-based testing

I want PHPUnit to support property-based testing natively, without depending on Eris or any other library. A property should be an ordinary test method whose parameters PHPUnit generates from their types, and every generated example should run as a real test, with its own before-test and after-test methods, its own test doubles, and its own expectations. A library cannot provide this from the outside, because only PHPUnit controls the lifecycle of a test.

Part 1: Why PHPUnit should do this itself

Properties find the inputs we did not think of

When we write tests the usual way, we pick example inputs and specify the expected output for each of them, for instance with a data provider. We only test what we think of, and the bugs are where we did not think to look.

A property describes an invariant that must hold for all valid inputs instead: reversing a list twice yields the original list, adding two amounts of money is commutative. A generator produces many random inputs, and the property is checked for each of them. When an input is found for which the property does not hold, shrinking simplifies it step by step until a minimal counterexample remains: [-1] instead of [-12, 5, -7, 3, 19, -2].

I have written about this in more detail: Property-Based Testing introduces the idea, Smaller Input, Greater Insight explains shrinking, and Data Provider or Properties? discusses when to use which.

Eris works from outside PHPUnit, and that is where its limits come from

Today, property-based testing with PHPUnit means using Eris. You use its trait in your test case class, declare generators with forAll() inside a test method, and pass the property to then() as a closure:

<?php declare(strict_types=1);
use Eris\TestTrait;
use PHPUnit\Framework\TestCase;
use function Eris\Generator\int;
use function Eris\Generator\seq;

final class ReverseTest extends TestCase
{
    use TestTrait;

    public function testReversingTwiceIsIdentity(): void
    {
        $this->forAll(seq(int()))->then(
            function (array $input): void
            {
                $this->assertSame($input, array_reverse(array_reverse($input)));
            },
        );
    }
}

I am grateful for Eris. @giorgiosironi started it in 2014 and has made property-based testing available to PHP developers for more than a decade, most recently by adding support for PHPUnit 11, 12, and 13. The articles linked above use Eris. I am not proposing this to compete with Eris, and certainly not to spite its maintainer and his contributors.

I am proposing it because Eris has to integrate with PHPUnit through a trait and the few extension points PHPUnit offers, and some of the resulting limitations can only be removed inside PHPUnit.

The whole property runs inside a single execution of the test method. before-test and after-test methods run once for all examples, so state leaks from one example into the next, and into shrinking. A test that creates an empty stack in a before-test method, pushes the generated value onto it, and asserts that the stack now holds one element fails with a message such as Failed asserting that 80 is identical to 1. Expectations for test doubles such as once() are verified after all examples have run. And expectException() applies to the property as a whole: the first example that throws the expected exception ends the test, and the test passes, even when the property does not hold for other inputs.

Only assertion failures are shrunk. When the code under test throws an exception or raises an error, the property ends without a minimal counterexample, and the report does not show the input that caused it. The same is true for expectations for test doubles that are verified after the property has run. When an example triggers a deprecation, notice, or warning, PHPUnit reports it for the test but cannot say which input triggered it.

The seed that reproduces a run comes from an environment variable or from the current time and is echoed to the console, so it is missing from Open Test Reporting and other logfiles. The random values come from PHP's global Mersenne Twister, which the code under test shares.

Finally, generators are declared apart from the parameters of the closure that receives the values, so static analysis cannot check that the closure accepts what the generators produce.

A property should be a test method with generated parameters

A property is a test method with the #[Property] attribute. Its parameters are the inputs of the property. By default, PHPUnit derives a generator for each parameter from its type, using the same PHPDoc type syntax that PHPStan and Psalm understand:

<?php declare(strict_types=1);
use PHPUnit\Framework\Attributes\Property;
use PHPUnit\Framework\TestCase;

final class ReverseTest extends TestCase
{
    /**
     * @param list<int> $input
     */
    #[Property]
    public function testReversingTwiceIsIdentity(array $input): void
    {
        $this->assertSame($input, array_reverse(array_reverse($input)));
    }
}

Refined types such as integer ranges, non-empty-string, and array shapes work as well:

/**
 * @param array{name: non-empty-string, age: int<0, 150>} $person
 */
#[Property]
public function testEveryPersonCanBeGreeted(array $person): void
{
    $this->assertStringStartsWith('Hello', (new Greeter)->greet($person));
}

Enumerations are generated from their cases. Objects are generated through their constructor, recursively, using the types of the constructor's parameters:

enum Currency: string
{
    case EUR = 'EUR';
    case USD = 'USD';
}

final readonly class Money
{
    /**
     * @param non-negative-int $amount
     */
    public function __construct(public int $amount, public Currency $currency)
    {
    }

    // ...
}

#[Property]
public function testAdditionIsCommutative(Money $a, Money $b): void
{
    $this->assertEquals($a->add($b), $b->add($a));
}

/**
 * @param int<1, 10> $parts
 */
#[Property]
public function testAllocationPreservesTheAmount(Money $money, int $parts): void
{
    $this->assertSame($money->amount, array_sum(array_map(
        static fn (Money $share): int => $share->amount,
        $money->allocate($parts),
    )));
}

When a type does not say enough, the #[ForAll] attribute specifies the generator for a parameter explicitly. The generator is built with composable functions such as Generators::lists(), Generators::tuples(), map(), and filter(). It can come from a static method of the test case class:

use PHPUnit\Framework\Attributes\ForAll;
use PHPUnit\Framework\Attributes\Property;
use PHPUnit\Framework\PropertyBased\Generators;
use PHPUnit\Framework\PropertyBased\ValueGenerator;

/**
 * @return ValueGenerator<non-empty-string>
 */
public static function bitStringsWithoutLeadingZeros(): ValueGenerator
{
    return Generators::strings(alphabet: '01', maximumLength: 62)->map(
        static function (string $bits): string
        {
            return '1' . $bits;
        },
    );
}

#[Property]
public function testConversionToDecimalAndBackIsLossless(#[ForAll('bitStringsWithoutLeadingZeros')] string $bits): void
{
    $this->assertSame($bits, decbin(bindec($bits)));
}

With PHP 8.5, which allows static closures in constant expressions, the generator can be declared right in the attribute:

#[Property(examples: 1000)]
public function testEmailAddressCanBeParsed(
    #[ForAll(static function (): ValueGenerator
    {
        return Generators::tuples(
            Generators::strings(alphabet: 'abcdefghijklmnopqrstuvwxyz', minimumLength: 1, maximumLength: 16),
            Generators::elements('example.com', 'example.org'),
        )->map(static function (array $parts): string
        {
            return $parts[0] . '@' . $parts[1];
        });
    })]
    string $address,
): void
{
    $this->assertSame($address, EmailAddress::fromString($address)->asString());
}

For domain objects that must not be created through their constructor, a test case class can provide a generator that is used wherever a value of that type is needed, including in constructor parameters of other objects:

use PHPUnit\Framework\Attributes\GeneratorFor;

/**
 * @return ValueGenerator<EmailAddress>
 */
#[GeneratorFor(EmailAddress::class)]
public static function emailAddresses(): ValueGenerator
{
    return Generators::strings(alphabet: 'abcdefghijklmnopqrstuvwxyz', minimumLength: 1, maximumLength: 16)
        ->map(static function (string $localPart): EmailAddress
        {
            return EmailAddress::fromString($localPart . '@example.com');
        });
}

#[Property]
public function testCustomerCanChangeTheirEmailAddress(Customer $customer, EmailAddress $newAddress): void
{
    $customer->changeEmailAddress($newAddress);

    $this->assertSame($newAddress, $customer->emailAddress());
}

When a property does not hold, PHPUnit reports the minimal counterexample as PHP code that can be copied into a data provider for a regression test:

Random Seed: 1759741221

There was 1 failure:

1) ShopTest::testAllocationPreservesTheAmount
Failed asserting that 0 is identical to 1.

Property falsified after 2 examples, counterexample shrunk in 17 steps:

    $money = new Money(amount: 1, currency: Currency::EUR);
    $parts = 2;

/path/to/tests/ShopTest.php:31

Only PHPUnit can make every example a real test

PHPUnit runs every example through the complete lifecycle of a test: before-test and after-test methods, creation and verification of test doubles, expectException() and the other expectations, output buffering, error handling, and the backup of global state. Properties get the isolation that data providers have always had. Any failure, be it a failed assertion, an exception, an error, or an unmet expectation for a test double, falsifies the property and is shrunk to a minimal counterexample.

This isolation has a price, and I think it is worth paying. An example costs as much as a data set of a data provider, whereas Eris calls a closure inside a single test. Shrinking makes it more expensive still, because every attempt to find a smaller counterexample is a complete test, so shrinking has to get by with few attempts. That is why I want generators to record the random choices from which they build a value, and shrinking to work on that record instead of each generator implementing its own shrinking. This is the approach that Hypothesis introduced. Generators composed with map(), filter(), and flatMap() then shrink as well as the built-in ones, and small counterexamples are found with far fewer runs of the property.

Deriving generators from types means that the type is written once. The PHPDoc type that PHPStan and Psalm use to analyse the body of the property is the type from which its arguments are generated. For PHP code, the generators have to include the values for which PHP behaves in surprising ways: PHP_INT_MAX + 1 is a float, strlen() counts bytes and not characters, and -0.0 becomes the string "-0".

Reproducing a failure should not depend on anything outside PHPUnit. One seed per test run is printed with the other runtime information. Each property derives its own seed from it and from its test ID, so --filter, execution order, and process isolation do not change the examples a property sees. The minimal counterexample of a falsified property is remembered in the test run history and tried first the next time. Because PHPUnit knows about properties, counterexamples and seeds appear in the default output, in Open Test Reporting logs etc., and a property is one test for --filter, --list-tests, --stop-on-failure, test sizes, process isolation, code coverage, and the baseline.

The other price is that PHPUnit takes on the maintenance of generators and a shrinker, work that Eris has done for PHPUnit users so far. I think this is the right trade, because the most important of the limitations described above, the missing isolation between examples, cannot be removed any other way.

Eris will keep working with PHPUnit, and I will not deliberately break its integration. If you prefer Eris, or need a generator that Eris has and PHPUnit does not, keep using it. I would welcome feedback on this proposal from the maintainers of Eris and from its users.

Part 2: Specification

1. Scope

• Property-based testing is implemented in PHPUnit itself. It does not depend on Eris or any other property-based testing library.
• The public API consists of attributes, the Generators factory and the ValueGenerator interface, two new events, and configuration options. Everything else is internal.
• Stateful (model-based) testing, explicit examples, statistics about generated values, and targeted property-based testing are out of scope for the first iteration; see section 12. The design must not prevent them.

2. Terminology

• Property: a test method with the #[Property] attribute.
• Example: one execution of a property with generated arguments.
• Generator: an object of type ValueGenerator<T> that generates values of type T.
• Choice sequence: the list of non-negative integers from which a generator constructs a value (see section 6).
• Counterexample: arguments for which the property does not hold. The minimal counterexample is the result of shrinking.
• Rejected example: an example that is discarded because a generator filter could not be satisfied or because assume() was called with false.

3. Declaring properties

3.1 #[Property]

• PHPUnit\Framework\Attributes\Property targets methods. A method with this attribute is a test method; neither the test prefix nor #[Test] is required.
• Optional argument examples (positive integer): the number of examples to check. Without it, the configured default is used (section 9).
• The method should have a void return type. A return value is ignored and is not passed to dependent tests.

3.2 Parameters

Each parameter of a property receives a generated value. The generator for a parameter is determined in this order:

1. the #[ForAll] attribute of the parameter (section 5.1);
2. a generator registered with #[GeneratorFor] for the parameter's type (section 5.2);
3. the generator derived from the parameter's type (section 4).

Variadic parameters are not supported. Parameters with default values receive generated values like all other parameters.

If no generator can be determined for a parameter, the test is not run and an error is reported for it, just as for an invalid data provider. The message names the parameter and the type that could not be handled.

3.3 Combination with other attributes

• #[DataProvider], #[DataProviderExternal], #[TestWith], and #[TestWithJson] cannot be combined with #[Property]. Such a test is reported as an error.
• #[Depends] and its variants may be used on a property for ordering and for skipping when a dependency did not pass. The return values of dependencies are not passed to the property.
• Other tests may depend on a property. A property passes when all its examples pass. Its return value is null.
• #[Repeat] and --repeat check the whole property repeatedly. Each repetition uses a different seed (section 7.2).
• #[Retry] and --retry do not apply to properties: a falsified property is deterministic for a given seed, and flaky properties are detected separately (section 6.5). A warning is emitted when #[Retry] is used on a property, as for other methods that are not eligible for retrying.

3.4 Rejecting examples inside the property

TestCase::assume(bool $condition): void discards the current example if $condition is false. A rejected example neither passes nor fails, and it does not count towards the number of examples.

4. Generators derived from types

4.1 Source of the type

• If the method's doc comment has a @param tag (including @phpstan-param and @psalm-param) for the parameter, its type is used. It is parsed with phpstan/phpdoc-parser.
• Otherwise, the native type declaration is used.
• Class names in PHPDoc types are resolved against the namespace and the use statements of the file that declares the method, the same way PHPStan does.

4.2 Supported types

Not supported, and requiring #[ForAll] or #[GeneratorFor]: interfaces (other than DateTimeInterface), abstract classes, classes without a public constructor, object, mixed, callable, Closure, iterable, resource, and template types.

Classes whose construction is recursive, for example a tree node that takes child nodes, are generated up to a maximum depth.

5. Explicitly specified generators

5.1 #[ForAll]

PHPUnit\Framework\Attributes\ForAll targets parameters. Its argument is either

• a string: the name of a public static method of the test case class that returns a ValueGenerator, or
• a static closure that returns a ValueGenerator (requires PHP 8.5).

The attribute's argument is evaluated once per property, before the first example is generated.

5.2 #[GeneratorFor]

PHPUnit\Framework\Attributes\GeneratorFor(class-string $type) targets public static methods of a test case class (including inherited methods and methods declared in traits) that return a ValueGenerator. The generator is used for every value of exactly that type that a property of this test case class needs, including values needed to derive other objects from their constructor. Two methods registered for the same type are reported as an error.

5.3 The generator API

PHPUnit\Framework\PropertyBased\ValueGenerator<T> is an interface with these methods:

• map(Closure(T): U $mapper): ValueGenerator<U>
• filter(Closure(T): bool $predicate): ValueGenerator<T>: values that do not satisfy the predicate are regenerated. If this fails too often, the example is rejected.
• flatMap(Closure(T): ValueGenerator<U> $mapper): ValueGenerator<U>

PHPUnit\Framework\PropertyBased\Generators is a class with static factory methods:

• integers(int $minimum = PHP_INT_MIN, int $maximum = PHP_INT_MAX)
• `floats(floa