Create a 'Profiling' install profile, for testbot-powered simple profiling and easier local profiling

Problem/Motivation

We've been working for many months now to improve Drupal 8 performance. But not only is measuring (i.e. "profiling") it hard, it's also hard — even for the most prolific contributors — to not accidentally introduce performance regressions. Currently, it's trivial to introduce performance regressions that go unnoticed: additional services being added for one purpose may in fact be initialized always (even when unnecessary — see #1973618-27: DIC: Lazy instantiation of service dependencies (ProxyManager for "proxy services")), additional JS assets that are loaded accidentally (e.g. #2160555: Disable Edit on admin pages), and so on.

The only way (currently) to see if a patch negatively impacts performance, is by performing many manual steps:

1. XHProf-based profiling requires a lot of labor, hence very few people use it
2. alternatively: webprofiler.module, which is a lot simpler, but very few people use it and it's often broken due to changes in HEAD
3. generating content is very cumbersome: either you have to do it manually, or you have to wait for devel_generate to be fixed (it, too, is often broken due to changes in HEAD)

Not to mention that actually making proper comparisons is extremely hard.

It's nigh impossible to do it with XHProf unless you have a computer with a constant, controlled environment — results vary wildly with the environment (hardware, PHP version, system load, etc.)! On top of that: if we're comparing a specific, approximately-real-world-scenario, you need generated content, and it's currently very painful to generate the exact same content in different versions of Drupal 8 (due to changes in HEAD).

Some profiling is better than none: increasing awareness and accessibility

In order for Drupal 8's performance to only move in the general right direction performance-wise from now on, we need to:

1. increase awareness and accessibility: when people post a patch that affects performance (in the positive or negative sense), they should be notified
2. provide a default set of profiling scenarios in core, so that it's easy for anybody to profile the same scenarios on their systems — this also helps developers onboard to more advanced profiling

Ideally, we'd have an interface to install Drupal (#2115533: Add InstallerInterface to provide a stable non-interactive installer) as well as an API to generate content. Then, anyone could write scripts for the scenarios they care about. But, that's a massive undertaking (which msonnabaum eventually had to give up on) and it still doesn't help with the above points.

Also ideally, we'd be able to automatically run XHProf (but alas, it's never installed by default and in fact for a few months you've needed a custom build of XHProf for it to work with Drupal 8 — none of the packaged XHProf versions work with PHP 5.5 + Drupal 8!), and even more ideally we'd be able to get the number of CPU instructions it took to render a page. Neither of those are achievable. And even if they were, we'd still not have apples-to-apples comparable numbers, because the results depend on the environment (hardware, PHP version, etc.)

In an überideal world, we'd have a PerfBot running in tandem with TestBot. But creating the infrastructure for this is non-trivial, especially because there are more strict environment requirements for a PerfBot, if we want its results to actually be correct.

What is achievable, is measuring numbers that don't depend on the environment and still are good performance indicators: number of CSS/JS assets, cache/config/state gets, DB queries, initialized services, rendered blocks … And to measure those things, we can simply build on top of what we already have in core: http://api.symfony.com/2.4/Symfony/Component/HttpKernel/Profiler/Profiler.htmlSymfony's Profiler! (This is also what Web Profiler uses.)

Yes, those numbers should be interpreted with care as well. There is nothing inherently bad about the number of services or the number of cache gets for a certain response. But it usually is bad when those numbers increase, and all too often that happens without us noticing. So that is what this issue wants to provide. And at the same time, we get profiling scenarios in core that we can

Proposed resolution

Therefore, this issue proposes a simpler set of steps:

1. create a 'Profiling' install profile that reuses the 'Standard' profile but adds generated content to it
2. this new install profile would include a ProfilingTest (much like the 'Standard' profile has StandardTest), but instead of testing whether things work as expected (which is already covered by the current body of tests), it enables a hidden'Simple Profiler' module (part of the 'Profiling' profile), which compares the actual number of cache gets, config gets, initialized services … with the expected numbers for various scenarios
3. this 'Simple Profiler' module would be a very barebones, no-UI version of Web Profiler, and it would leverage Symfony's Profiler and DataCollector (just like Web Profiler)
4. the test would mark every status quo as a "pass" and would mark each improvement and regression as an exception (to indicate there's no hard failure, but just a possible unwanted change — just changing the expectations in ProfilingTest will make it pass again). This is how every patch author would be notified of (potential) performance improvements/regressions.

This is what that looks like:

Remaining tasks

1. Discuss the general idea.
2. If we want to go forward with this, we'll also want to discuss what the specific profiling scenarios should be. We can always add more though!
3. Get to RTBC.

User interface changes

None.

API changes

None.