Slow glob() calls in ConfigStorageController::buildQuery() with many yml files

I was testing a big upgraded with lots of fields (290 fields and 500 instances or so, 1600 yml files in total with entity displays, image styles and other things like that).

One major performance issue that I noticed (apart from 300+ loaded config files, duplicated to 600 by locale.module) was that glob() in buildQuery() gets really slow with that amount of configuration.

I displayed a simple view on the frontpage and glob() took up almost 10% of the whole request time, with 100 calls to buildQuery().