BrowserTestBase creates a fresh Drupal installation for each test. This takes a considerable part of test execution time and has a negative impact on developer experience.
Another performance problem is enabling modules that a test depends on. It is quite expensive operation and in some cases it may take even more time than installing Drupal.
The possible solution could be dumping a database into SQL file just before the test case get executed. So that on the next test run Drupal can be installed using predefined SQL dump which is much faster the normal installation process.