Right now if an exception is thrown during a test's setUp() or tearDown() method, the testbot shows a "CI error". See discussion at #2189345-192: run-tests.sh should exit with a failure code if any tests failed for additional details.
Overall it would be better if that exception were handled by Drupal itself.
Drupal 8 looks like it fixed this in #2170023: Use exceptions when something goes wrong in test setup.