As outlined in #2538462: Contrib Testing Test discovery not working, the old PIFR test runners used to scan module directories to identify tests. The new test runner was built to use the --module argument, which was seen as an improvement over passing multiple dozens of individual test classes along on the run-tests.sh command line.
However, this approach is not working for many contrib projects ... ubercart is one example where passing --module ubercart will result in 'no tests found', due to the chosen module naming scheme not playing nicely with the test discovery assumptions. The suggested approach to get around this is to scan for all tests under the specified directory.
However, instead of building this scanning capability into the new test runner, the maintainers assert that test discovery should be the responsibility of the test framework itself, not the individual runner ... i.e. the scanning for files should be the responsibility of run-tests.sh, not PIFR or DrupalCI.
Proposed Solution
Add a --directory option to run-tests.sh that, when present, will scan for all test files within the specified directory; using the original logic from PIFR to perform the actual test discovery.
Marking major, as this blocks the preferred solution to resolve contrib test discovery within DrupalCI (which in turn blocks the turndown of PIFR).