Problem/Motivation
We want to quit using Simpletest: #2866082: [plan] Roadmap for Simpletest
As part of that, we need to isolate our testing infrastructure away from the simpletest module: #2863044: [plan] Remove simpletest dependencies from PHPUnit-based tests
Drupal\simpletest\TestDiscovery lives at the center of this situation, daring us to make a move in any direction.
It acts like a service provided by the simpletest module, which leads to many dependencies: #2863055: Move TestDiscovery out of simpletest module, minimize dependencies
It's also used by code which should not have dependencies on core modules. Specifically, run-tests.sh and the PHPUnit test suite discovery classes. run-tests.sh only has a dependency on an installed Drupal with simpletest enabled so that TestDiscovery can consume the module_handler service.
What Does TestDiscovery Do?
It does the following:
It registers test class names for autoloading, in TestDiscovery::registerTestNamespaces(). This is generally bad practice. It has the benefit of being able to autoload fixture classes, but that's maybe not worth it.
It can scan a directory for test files in TestDiscovery::scanDirectory(). This method makes a classmap between class names and the files that contain them. It does this by requiring a namespace prefix as a parameter, which is then prepended to the names of test files it finds.
It can find all the tests, using TestDiscovery::findAllClassFiles(), which can optionally filter by a single extension. It returns a classmap of what it finds.
It can find out metadata about a test, using TestDiscovery::getTestInfo(). Given a test class name, this method can parse out @group and other annotation, as well as assigning the test a 'type,' which tells the runner whether it's a PHPUnit-based test or a WebTestBase test. The way it does this is different enough from the way the phpunit tool does it to make false-positives a risk: #2878269: Modify TestDiscovery so the testbot runs all the tests We can also discover the @group associated with a test, but only the first one: #2296615: Accurately support multiple @groups per test class
It has a soft dependency on a cache backend. If it finds tests, it will store them to a cache and then re-use the cache without a way to invalidate. #1667822: Remove caching of test classes in TestDiscovery
Why Does TestDiscovery Do These Things?
Most of what TestDiscovery does is for the benefit of the Simpletest UI. For instance, most of what we get from getTestInfo() is there so that we can display the test information in a UI form and will be unused in other contexts.
In run-tests.sh we take the information from findAllClassFiles() and re-sort it so it's organized by group using getTestInfo() for a test listing. This ignores that many tests have multiple groups: #2296615: Accurately support multiple @groups per test class
TestDiscovery allows for the autoloading of tests so that it can run them in the batch processor associated with the Simpletest UI form. This only matters for WTB tests.
TestDiscovery caches its results so that it can improve performance of the Simpletest UI form. This strategy was used before Drupal had sophisticated form caching but has always been a problem because the cache does not invalidate when you modify tests. This has been fixed and was then reverted again in: #2893117: Improve HTML caching of Simpletest UI test form
What Else Is There?
When you run the phpunit tool, discovery is performed by test suites. These are classes that know how to discover tests in Drupal. They only perform the part of the discovery that's needed to get an array of file names.
These test suite classes are located in core/tests/TestSuites/ and they rely on the Drupal\Tests\TestSuites\TestSuiteBase::addTestsBySuiteNamespace() method. This method wraps around TestDiscovery::scanDirectory() for reasons which are mysterious. It assumes a namespace for the directory being scanned, and then only uses the class name to filter out a specific listener directory, which it doesn't actually need to do.
The test suite classes exist so that we can specify different types of tests to run from the phpunit command line tool.
The test suite classes are unused by TestDiscovery.
What Goals Should We Try To Accomplish?
We should try to build a test discovery system that does not rely on an installed Drupal site, or any code in the simpletest module. This allows us to move away from simpletest in Drupal 9 with relative ease.
For consistency, we should use the same code to discover all test types for all of the test runners.
We should fully use the frameworks we've adopted.
Proposed resolution
Normalize on PHPUnit test suite classes for discovery and test running.
Create a TestDiscoveryTwoPointOh class that can perform whatever discovery is required for a given test suite.
Convert run-tests to use this new class for discovery and for supporting the --types argument.
Add a PHPUnit test suite class that knows how to discover WTB tests, similar to the existing test suite classes. PHPUnit_Framework_TestSuite is an iterator, so it discovers tests and then we can loop through them without using PHPUnit to run them.
Convert the now-old-hat TestDiscovery to consume the results of this new discovery class and provide BC for the simpletest module.
This is similar to #2863055: Move TestDiscovery out of simpletest module, minimize dependencies but a bit more radical in design.