Simpletest is a legacy testing framework which comes out of the Drupal 6 days, was in core in Drupal 7, and is supported in Drupal 8 alongside PHPUnit-based frameworks.
Moving forward to Drupal 9, simpletest should be phased out in a reasonable way.
We currently have an initiative to convert core's tests to use PHPUnit-based frameworks, and not use Simpletest-based ones. This will happen sooner, rather than later. #2807237: PHPUnit initiative
This still leaves us with many legacy tests in contrib for Drupal 8, which we must still support, but which we'd rather not maintain.
The Testbot
DrupalCI testbot currently runs all the tests by invoking the run-tests.sh script provided by the simpletest framework and module.
The obvious reason for this is that there should be as few points of contact as possible between the test runner and the system under test. But there are other reasons as well:
- PHPUnit has an undefined behavior if the test process has a fatal error. That is, if a test crashes, the
phpunittool does not continue running other tests, and in fact does not generate a report of the tests that did run. - PHPUnit does not support concurrent test runs. This means that all tests must be run sequentially, leading to long CI times.
These factors mean that we should keep the behavior of run-tests.sh. It provides us with features we can't easily get from other libraries, and will be consistent for backwards-compatibility.
The Users
Users want to be able to run tests from the UI. #2750461: Remove Simpletest UI because we don't want to maintain a graphical test runner seeks to remove the Simpletest test runner UI from the module, but many of the comments on that issue are very much opposed to the idea.
This, and general principles of backwards-compatibility mean that we should still provide this UI for users.
The Maintainers
Core maintainers and others who wish to work on the testing subsystem are frequently frustrated with the fragile nature of the simpletest testing system.
It is difficult to test without refactoring, and is difficult to refactor without testing, and the quality of the rest of the code depends on the code that is demonstrated to work. Thus it is difficult to work on the testing subsystem from both the old-crusty-code perspective, as well as the reluctance-to-change perspective.
Note that most of the simpletest module is not marked as deprecated. There are two categories of things which are:
TestBaseand its children. These are the test types we wish to quit supporting.- Global-level functions which have been replaced by the
test_discoveryservice.
This means a deprecation process will be needed if any modernizing changes are to happen in simpletest, or if it is to be removed once and for all.
The Internals
Let's demonstrate some of the fragility of the current testing system. Here is what happens when the testbot runs your PHPUnit-based test:
run-tests.shcallsexec()in order to runphpunitfor a single test.phpunitgenerates a report for this single test in the JUnit XML format.run-tests.shparses this JUnit XML file and turns it into an array suitable for the {simpletest} database schema.- The array is added to the database.
run-tests.shdoes this for all the tests.- At the end,
run-tests.shgathers all the test info from the database. - This data is then converted to non-JUnit XML by
run-tests.sh. - Since
run-tests.shdoes not output JUnit, the XML it outputs is converted into JUnit by the testbot. - The JUnit is then consumed by Jenkins and by drupal.org for display.
As you can see, this is a lot of steps. Now, it might be that all of them are necessary. It might also be that they are not.
This illustrates that the testing system was not designed, but rather accrued, making it unnecessarily complex to maintain.
run-tests.sh and the simpletest module contain many of these shims that convert one thing to another in order to provide output that is known to be necessary.
The path of execution for these processes extend between global-level functions in run-tests.sh and simpletest.module. Note that neither of these files has a .php extension. That aside, this creates a dependency between the script and the module, which makes it increasingly unmaintainable.
The Pitch
So those are some of the problems in this space.
The solution seems to look like the following:
Consistency in Drupal 8
For the rest of the Drupal 8 lifecycle, we should provide the same behaviors from simpletest and run-tests.sh as we have now. We should provide the normal BC layer that we'd expect from any deprecated Drupal 8 code.
Drupal 8 should still be able to run Simpletest-based tests. While there is an initiative to phase out these tests in core, we're still promising to be able to run them. We need to fulfill this promise so that the testbot can still run contrib tests.
Drupal 8 should still have the Simpletest test runner UI. This form should be present and should be useable as it is now. It might be enhanced in some ways, but this should not be our focus. Whether to keep this UI in Drupal 9 can be an open conversation.
- #2750461: Remove Simpletest UI because we don't want to maintain a graphical test runner
- #2301481: Mark test groups as belonging to modules in UI
- #2858652: Support multiple @group test annotations in Simpletest UI
For Drupal 8, top-level functions in simpletest.module and run-tests.sh should be deprecated. Their functionality should be moved to helper classes which can be unit-tested with relative ease.
- #2863055: Move TestDiscovery out of simpletest module, minimize dependencies
- #2641632: Refactor simpletest's *_phpunit_*() functions etc. to a class
- #2800267: Turn simpletest_clean*() functions into a helper class/service
- #2798273: simpletest_phpunit_configuration_filepath() is dead code. Deprecate it.
Drupal 8's simpletest hook APIs should be deprecated. They are only currently supported for the Simpletest UI at this point, which is an inconsistency that will need to be maintained for the D8 life cycle. We can examine whether these hooks should be implemented within run-tests.sh, but since we're trying to move away from simpletest-isms, and since PHPUnit-based test listeners and alternate test suites are better available workarounds, this seems unlikely.
- #2460521: Deprecate hook_simpletest_alter: it isn't working from CLI
- #2234479: Deprecate hook_test_* hooks in simpletest
Radical changes to Drupal 8, with BC
In the near future, Drupal 8's simpletest module should only be the UI form. All test-running behavior should be factored into a core namespace.
All of run-tests.sh's behavior would also be refactored into this namespace, except the script itself which would then use these classes to run the tests.
This namespace should not be autoloaded unless it's test-running time. This means the code should be refactored into the core/tests/ directory. The simpletest module can autoload test-running code from this directory if it is enabled and running tests.
Once the code is moved in a consistent process into the core/tests/ directory, it could be turned into something like a component.
Drupal 8 core has a Component namespace where code lives that is completely isolated from the rest of core, and could become external libraries. The testing system would be similar to this, but should not live in the autoloaded core/src/Drupal/Component directory.
This testing component would at some point be isolated enough that it could even be an external library, marked as require-dev for drupal/core in composer.json.
This is the ideal goal for the simpletest refactor: That one could remove it from a production codebase using Composer, by issuing this command: composer install --no-dev Any point along the line between where things stand now, and this complete removal would be a step in the right direction.
Drupal 9
The Drupal 9 process would simply remove the deprecated top-level functions from the module itself.
The conversation over whether to keep the test runner UI would hopefully happen before the 9.0.0 release.
Since the other refactoring would have already occurred, the decision to remove the UI would be the same as deciding to remove the module from core. All other testing infrastructure provided by core would remain.