Problem/Motivation
We agreed on using nightwatch for JS browser testing, see below for more information.
This has been worked on by the JavaScript Initiative and is ready for a review here on d.o 🎉
It requires the patch from #2926633: Provide a script to install a Drupal testsite to be applied first.
Here is a branch which contains both this work and the work from #2926633: Provide a script to install a Drupal testsite:
https://github.com/jsdrupal/nightwatch/pull/1, see more about that on #2869825-48: Leverage JS for JS testing (using nightwatch)
Involved people:
justafish, dawehner, alexpott, drpal, mglaman, and mixologic
How to use it
Follow core/tests/README.md in the patch.
Background information
Since we java javascript test coverage backed upon ower PHPUnit based testing framework we hopefully gained some stability in our JS.
As multiple people discussed though, testing javascript in PHP might not be ideal:
Problems of testing JS in PHP/Advantages of testing JS in JS
- JS developers who want to participate into Drupal don't have to learn PHP / learn the phpunit testing framework as they are already familiar with nightwatch
- The JS world is much bigger than the PHP world now, so they should provide better testing frameworks for their needs
- Nightwatch makes it easy to open up a real browser to see what the test is doing
- By splitting up the programming languages, we make it clear that Drupal is API first. There is no way of communication possible, which traditionally lead to A LOT of insane JS bugs.
Advantages of testing JS in PHP/ Disadvantages of testing JS in JS
- One testing syntax to rule them all
- People are aware of phpunit now already
- We force the current test writers to learn another testing framework and we're seriously lacking people who are interested in writing tests already.
- Jumping from BrowserTestBase to JavascriptTestBase is quite easy right now.
- We split the functional testing into 2 separate frameworks. Upgrading an test to a JavaScript test will be more difficult and I really don't like the split in functional testing.
Will 'the people' really come if we have another testing framework? - Is it really possible to work on JavaScript alone in Drupal core or is it more of a mix-n-match with the php backend?
- It is potentially harder to setup a test in javascript. We don't have the APIs to enable modules in JS ...
This was discussed on a BOF/meeting during Drupalcon Baltimore.
What we will use
We first agreed that testing JS in PHP was a good first step, but to really get serious about it, we have to leverage JS based tools.
As a tool we agreed on using nightwatch for the following reasons: It has a wide usage, it has good documentation. It can not only support end to end testing but also unit testing. It uses the webdriver protocol, which is an open w3c standard, so it is supported by every browser vendor potentially. On top of that it could support multiple sessions, aka. parallel testing.
How it will work
Each test will reinstall Drupal, just like in PHP.
Testbot side of things
After the agreement we talked about what we need on the testbot side of things. We agreed on the following approach:
- Provide a package.json script which runs all tests
- Let the testbot execute it and parse the JSON and call it a day
- Hope that this is really enough ;)
TL;DR
- It doesn't make sense to use PHP to test JS
- Nightwatch is a good tool to do that given its wide usage, support for multiple browsers, its unit testing integration as well as good documentation.
Remaining tasks
- Provide a more detailed example involving forms, an actual login process ..., fiddling with some ajaxy form. Ideally the follow tests:
- Drag-n-drop: \Drupal\Tests\book\FunctionalJavascript\BookJavascriptTest.php
- Dialog: \Drupal\Tests\views_ui\FunctionalJavascript\FilterCriteriaTest
- Dialog:
- Complex test: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest
- Provide package.json integration so the testbot just calls this and doesn't care about how to run the test
- Make the setup process for a test as simple as possible
- Provide some basic documentation on drupal.org which points to the right places on the nightwatch documentation
- Maybe do some research around parallel test running (something the current ecosystem can't provide)
API changes
A total new API for writing tests