Problem/Motivation
Nightwatch testing is extremely unreliable and the cause of many random pipeine test failures.
The js world in general, and Drupal outside of core, is using either Cypress or Playwright for similar kinds of tests, and very few projects at all use Nightwatch these days (see links in issue).
For accessibility testing, we can also use Functional JavaScript tests with Axe enabled.
Steps to reproduce
Look at related issues that were or are problematic
- #3371963: Update Nightwatch to 3.x
- #3387927: Failing Nightwatch jobs result in a passed pipeline job
- #3389763: Impossible to run only Nightwatch tests in a given directory (f.e. for contrib modules)
- #3421202: Enable W3C-compliant webdriver testing
- ...
Proposed resolution
Pick Cypress or Playwright to replace Nightwatch- there seems to be a slight but growing preference for Playwright both anecdotally on the issue and via npm download stats.
Also convert some Nightwatch tests to Functional JavaScript + Axe where appropriate.