[meta] Support PHPUnit 10 in Drupal 10

Problem/Motivation

PHPUnit 10 was released on Febryary 3, 2023. Drupal will need to make changes to support it in future release.

Proposed resolution

There are a number of changes that Drupal will have to address to support PHPUnit 10.

• withConsecutiveRemoval is deprecated in 9.6 and removed in 10. #3306554: InvocationMocker::withConsecutive() is deprecated in PHPUnit 9.6 and removed from PHPUnit 10
• Error handling (deprecation, warning, error, etc) does not work the same.
  • expect* for errors are deprecated in 9.6 and removed in 10. #3322785: Several expect*() methods have been deprecated in PHPUnit 9.6
  • Error exception classes are also removed which breaks some tests.
  • Symfony Bridge largely relied on this. #3272779: [Symfony 6.1] Replace deprecationListenerTrait with PHPUnitBridge ignoreFile should work around this.
  • Consider https://github.com/sebastianbergmann/phpunit/issues/5201 [Add TestCase::registerFailureInterface() to register interface that marks exceptions to be treated as failures, not errors] as a solution to convert Mink errors to PHUnit failures
• Non-static data providers are deprecated in PHPUnit 10, and need to be converted to be static methods before PHPUnit 11. This can be done now in 9.
  • Most tests can be converted with minimal effort #3353210: [PHPUnit 10] @dataProvider methods must be declared static and public
  • At least two tests do elaborate mocking in the data provider that require more effort to fix. #3368713: [PHPUnit 10] Provide a static alternative to @dataproviders using PHPUnit mocks in CommentLinkBuilderTest and MailHandlerTest
• TestCase::getName renamed. getName is an internal method that was renamed to name. Our use seems valuable, so we can quickly address this with Drupal's forward compatibility trait. Issue #3405353: Convert use of TestCase::getName to forward compatible TestCase::name.
• TestClass constructor arguments changed. We generally follow the best practice of not using the constructor but there are exceptions. A handful of tests use TestClass or the Test as a mocks instantiating the class directly. Issue #3405360: Remove use of TestClass constructor.
• Listeners are out, Extensions and events are in.
  • HTML Output printer is broken. Convert to extension. Document steps to use going forward.
  • #3125809: Remove usages of \PHPUnit\Framework\TestListener once PHPUnit 9 support is dropped
  • This breaks Symfony Bridge.
• TestSuite::static logic is gone. #2499239: Use test suite classes to discover different test types under phpunit, allow contrib harmony with run-tests is broken. There is no direct replacement, so we need an upgrade path. #3402444: Deprecate Test Suites, no longer available in PHPUnit 10
  • PHPUnit does not want you interacting with discovery. Between object cloning, readonly properties, and immutable objects it goes to great length to block extensions from modifying the test list even with reflection or other deep hacks.
  • Options:
    • We go back to the out-of-the-box directory discovery process. Contrib is using a different process in gitlab which doesn't use core's discovery so the great consolidation is only supporting a handful of submodule directories in core and legacy drupalci so this might be doable.
      • Pros:
        • We can remove a lot of complexity in our test runners and rely on PHPUnit more.
      • Cons:
        • Drupal's directory rules are complicated and we risk tests being added that aren't being run. One of the goals of the Great Consolidation was to make sure we didn't miss tests in subdirectories.
    • We implement our own runner with our own discovery.
      • Pros:
        • We have the discovery
        • We have run-tests.sh
      • Cons:
        • maintenance
        • Blocks tool adoption like coverage and paratest
        • If enforced on contrib, could block additional features adoption in contrib.
• Comparator signature changes. Typed return breaks our single custom comparator but with an easy BC compatible fix. Issue #3405351: MarkupInterfaceComparator function signatures not compatible with PHPUnit 10.
• CLI options have changed.
  • We spawn phpunit directly in some places to test our ability to run tests and such. Some of this may need version specific handling or adjustment.
  • Out-of-the-box phpunit runner output is much more verbose. Triggered deprecations and Risky tests are always shown out of the box. Don't know if this is breaking but worth noting.
• phpunit.xml.dist has migrations. Mostly because of the removal of listeners.

Related library changes:

• symfony/phpunit-bridge
  • https://github.com/symfony/symfony/issues/45223 and https://github.com/symfony/symfony/pull/45226
  • https://github.com/symfony/symfony/issues/49069
• phpspec/prophecy-phpunit
  • phpspec/prophecy-phpunit 2.1.0 should be compatible.
  • https://github.com/phpspec/prophecy-phpunit/pull/41

PHPUnit 9.6 deprecations issues: https://www.drupal.org/project/issues/search?issue_tags=PHPUnit%209.6

Remaining tasks

• Ability in Gitlab to run run different phpunit.xml.dist depending on the PHPUnit version being run. Allow testing the upgrade path for listeners and run both versions during the upgrade process.
• Decision on test suite discovery

Follow ups

• #3125809: Remove usages of \PHPUnit\Framework\TestListener once PHPUnit 9 support is dropped