[meta] Finalize the cache contexts API & DX/usage, enable a leap forward in performance

Problem/Motivation

The Big Picture of what we've already done

In order for Drupal 8 sites to have great performance, we've done a huge amount of work. A big part of that is making Drupal 8 the first release to no longer break render caching all over the place.
E.g. by no longer doing highly personalized things (such as quick edit links, "new" and "updated" comment markers …) or highly dynamic things (adding a class to the "active" links) without thinking about consequences. Those things have been replaced either with #post_render_cache placeholders/callbacks or client-side rendering.

We've already done this work for all of the obvious violators. So, we're mostly there. But we're not yet 100% there. We're ~95% there in terms of cache tags. But we're nowhere near 100% when it comes to cache contexts.

Cache Contexts: current status & potential

So far only applied cache contexts to 2 things: blocks (see BlockBase) and access results (see AccessResult).

If we'd bring the usage of cache contexts to 100%, then we could truly leverage all that performance work… and bring a massive leap in performance. Because we would have complete knowledge about the contexts by which every rendered thing is varied by, which combined with bubbling of cache contexts, would allow us to reliably cache ever larger pieces of the page, hence needing to invoke ever fewer objects to build ever fewer render arrays, to do ever fewer rendering… to also initialize ever fewer services.

Why?

It is precisely that metadata that was missing in prior versions of Drupal… which required us to generate pretty much everything on the entire page for every single request. But in Drupal 8, we have a Holy Cache Metadata Trinity:

1. Cache keys identify a thing. E.g. ['node', 5]
2. Cache contexts identify the different variations (representations) of that thing. (The contexts it depends upon.) E.g. ['cache_context.user'].
3. Cache tags identify the reasons for (a variation of) a thing to be invalidated. (The other things it depends upon.) E.g. ['node:5', 'taxonomy_term:3']

Together, they give us exactly what we need to cache the majority of a page, instead of the minority (like in the past).

Finally: this allows us to add mindblowing alternative render pipelines, which e.g. automatically convert pieces of content that are varied by a "high frequency cache context" (like the "per user" cache context — there usually are many users on a site) into placeholders. That would mean we can still cache the entire page's HTML, but just have some placeholders in there for the highly personalized (high frequency cache contexts) things, which would be the only things that'd still need to be calculated. Those placeholders could even be rendered BigPipe-style.

For more info about the big performance improvements that this allows, see:

• 3-step plan, with algorithms etc.: https://docs.google.com/document/d/1Gw7ohBOUKu38t4kMbN9zj6cX-4-_2ZNXGotv...
• Step 1 in that plan: #2429617: Make D8 2x as fast: Dynamic Page Cache: context-dependent page caching (for *all* users!).

Proposed resolution

We need it to be applied to everything that can have different representations:

1. field formatters (output may vary depending on the permissions of the user)
2. text format filters (output may vary depending on e.g. language)
3. …

Remaining tasks

Must-haves (in order of importance, most important first)

All done!

Should-haves that we should be able to do after 8.0, because all blockers are critical

• Actionable: #2473873: Views entity operations lack cacheability support, resulting in incorrect dropbuttons
• Actionable (in its own child issues): #2469431: BigPipe for auth users: first send+render the cheap parts of the page, then the expensive parts
• #2526470: Audit core for things missing cacheable metadata (for authenticated users)

Nice-to-haves that we may not be able to do after 8.0 (in order of importance, most important first)

• #2495001: [meta] Add a replacement for renderRoot() that returns a cacheable render array instead of a string
• #2465377: Long-standing architecture flaw: Directly printing a nested element skips important processing (including #access) of parent elements

Can happen after 8.0, because no API changes:

• #2541344: BlockBase subclasses should merge their cache tags/contexts with the parent's (BlockBase's)
• Actionable: #2453945: Use multiple get for #pre_render / #cache where possible
• Actionable: #2480077: Using the URL alias UI to change aliases doesn't do necessary invalidations: path aliases don't have cache tags
• Not started yet: #2453835: Allow non-intrinsic (implementation-dependent) cache context services to specify their parents via DynamicParentContextInterface::getParents()
• #2443077: Add cache_context.main_request to force execution of the main route
• Not started yet: #2495779: Make #theme => links take cacheability metadata as an argument
• #2443077: Add cache_context.main_request to force execution of the main route
• #2352009: [pp-3] Bubbling of elements' max-age to the page's headers and the page cache

Done (in chronological commit order, first listed was committed first):

1. #2318437: Replace the hardcoded langcode key on blocks with the 'language' cache context
2. #2329101: CacheableInterface only has a getCacheKeys() method, no getCacheContexts(), leads to awkward implementations
3. #2429261: Replace the hardcoded cache key on the book navigation block with a 'book navigation' cache context
4. #2428563: Introduce parameter-dependent cache contexts
5. #2430341: Comment pager-dependent cache key should be a cache context
6. #2429257: Bubble cache contexts
7. #2396333: BlockContentBlock ignores cache contexts required by the block_content entity
8. #2445761: Add a X-Drupal-Cache-Contexts header to aid in debugging and testing
9. #2433599: Ensure every (non-views) pager automatically associates a matching cache context
10. #2445743: Allow views base tables and entity types to define additional cache contexts
11. #2443073: Add #cache[max-age] to disable caching and bubble the max-age
12. #2453891: Renderer::getCacheableRenderArray() does not include max-age
13. #2444211: Document cacheability of render arrays, and the considerations to use when generating render arrays
14. #2432837: Make cache contexts hierarchical (e.g. 'user' is more specific than 'user.roles')
15. #2448823: Add languages:<type> cache contexts
16. #2458413: BlockViewBuilder should specify cache contexts even for uncacheable blocks
17. #2458993: #cache[expire] is undocumented, unused, untested: remove it, use #cache[max-age] instead
18. #2428805: Remove the ability to configure a block's cache contexts
19. #2459003: #cache[cid] breaks bubbling
20. #2428703: Add a 'user.permissions' cache context (was: "Should cache contexts be able to associate a cache tag?")
21. #2451679: Validate cache contexts (+ cache contexts in some views plugins wrong)
22. #2453059: Set default render cache contexts: 'theme' + 'languages:' . LanguageInterface::TYPE_INTERFACE
23. #2444231: Fix CacheableInterface so it makes sense (was: "Make Config objects & Entities implement CacheableInterface + add BubbleableMetadata::createFromCacheableObject()")
24. #2458349: Route's access result's cacheability not applied to the response's cacheability
25. #2460013: Uninstalling modules containing cache contexts or #post_render_cache callbacks may break Drupal if they are cached somewhere
26. #2462851: Improve Views entity row renderer plugins' cache contexts
27. #2452317: Let views result cache use cache contexts
28. #2459819: Remove CacheableInterface (and no longer let block plugins implement it)
29. #2464877: Update RendererInterface::addDependency() to accept *any* object, not only CachableDependencyInterface objects
30. #2099137: Entity/field access and node grants not taken into account with core cache contexts
31. #2469277: Changing #cache keys during #pre_render or anywhere else leads to cache redirect corruption
32. #2468151: Rename the CacheContexts service to CacheContextsManager
33. #2463009: Introduce CacheableResponseInterface: consolidate ways of setting X-Drupal-Cache-Tags/Contexts headers
34. #2466585: Decouple cache implementation from the renderer and expose as renderCache service
35. #2335661: Outbound path & route processors must specify cacheability metadata
36. #2489966: The Views table style plugin does not specify cache contexts for click sorting
37. #2433591: Views using pagers should specify a cache context
38. #2487099: Set cache contexts for exposed sorts / items_per_page / offset.
39. #2493091: Installing block module should invalidate the 'rendered' cache tag
40. #2493047: Cache redirects should be stored in the same cache bin
41. #2470715: cacheGet-case: #post_render_callback's that result from other #post_render_calback are not processed
42. #2483781: Move cache contexts classes from \Drupal\Core\Cache to \Drupal\Core\Cache\Context
43. #2407195: Move attachment processing to services and per-type response subclasses
44. #2500443: Cache API topic says nothing about cache context, add something
45. #2511472: Refactor all usages of drupal_render()/Renderer::render() that break #2450993
46. #2449459: [PP-1] Make URLs cache context aware
47. #2487600: #access should support AccessResultInterface objects or better has to always use it
48. #2463581: #cache_redirect cache items should have an 'expire' timestamp that matches the merged max-age
49. #2495171: Block access results' cacheability metadata is not applied to the render arrays
50. #2375695: Condition plugins should provide cache contexts AND cacheability metadata needs to be exposed
51. #2443323: New convention: CacheContextInterface implementations should mention their ID in their class-level docblock
52. #2450993: Rendered Cache Metadata created during the main controller request gets lost
53. #2512718: EntityManager::getTranslationFromContext() should add the content language cache context to the entity
54. #2512866: CacheContextsManager::optimizeTokens() optimizes ['user', 'user.permissions'] to ['user'] without adding cache tags to invalidate that when the user's roles are modified
55. #2349679: Support registration of global context
56. #2217985: Replace the custom menu caching strategy in Toolbar with Core's standard caching.
57. #2351015: URL generation does not bubble cache contexts
58. #2525910: Ensure token replacements have cacheability + attachments metadata and that it is bubbled in any case
59. #2524082: Config overrides should provide cacheability metadata
60. #2430397: When mapping cache contexts to cache keys, include the cache context ID for easier debugging
61. #2526472: Ensure forms are marked max-age=0, but have a way to opt-in to being cacheable
62. #2559011: Ensure form tokens are marked max-age=0
63. #2504139: Blocks containing a form include the form action in the cache, so they always submit to the first URL the form was viewed at
64. #2429617: Make D8 2x as fast: Dynamic Page Cache: context-dependent page caching (for *all* users!)
65. #2458763: Remove the ability to configure a block's cache max-age
66. #2464427: Replace CacheablePluginInterface with CacheableDependencyInterface
67. #2483887: Mark RenderCache as internal