Cacheability information from route access checker access results are ignored by dynamic_page_cache

Problem/Motivation

Dynamic page cache currently ignores cacheability information from route access results which could lead to unexpected edge cases. Imagine a page where the access is decided per user bases but the content does not vary per user. Without adding user context to the render array/response of the page, the dynamic page cache module is not going to cache it per user.

Having this fixed is currently a blocker of another issue and the original fix for this problem were invented there: #3278759: [PP-1] Access cacheability is not correct when "view own unpublished content" is in use See also: https://git.drupalcode.org/project/drupal/-/merge_requests/8198#note_317834

Original description

Discovered in #2869426: EntityResource should add _entity_access requirement to REST routes.

See #2869426-28: EntityResource should add _entity_access requirement to REST routes, #2869426-29: EntityResource should add _entity_access requirement to REST routes and #2869426-83: EntityResource should add _entity_access requirement to REST routes in particular.

#29:

I could be wrong, but I remember that we once discussed that it is by design that route access cacheability is *not* considered by dynamic page cache because dynamic page cache runs *after* route access checking.

But maybe I misunderstood something here.

To my surprise 😲 and fear 😱, this was NEVER DISCUSSED! Zero mentions of route access. #2429617-219: Make D8 2x as fast: Dynamic Page Cache: context-dependent page caching (for *all* users!) mentions it. But doesn't dive in deeper. So I instead looked at every occurrence of access. Again nothing mentioned.

But I think @Berdir is right. It's just unfortunate that despite Dynamic Page Cache having such detailed test coverage, this is not something that is explicitly tested. (If it were, then the change made to \Drupal\Core\EventSubscriber\RouteAccessResponseSubscriber::getSubscribedEvents() should trigger a test failure.)

And I think the reason this is only coming up now and not months or years ago, is that the REST module's responses are atypical. Unlike pretty much all other responses, the controller in question (\Drupal\rest\Plugin\rest\resource\EntityResource::get()) has been:

1. Doing its own route/response-level access checking.
2. Significantly varying the contents of the response based on that access checking.

(The closest analogies are block and entity rendering, but those are not route/response-level, they're intra-response-level.)

Proposed resolution

TBD

Remaining tasks

1. Fix #3452426: Insufficient cacheability information bubbled up by UserAccessControlHandler that was identified in comment 26 by kristiaanvandeneynde OCD/spidy-sense
2. Get the #3452181: VariationCache needs to be more defensive about cache context manipulation to avoid broken redirects so it can spot potential issues revealed by this fix
1. Fix #3454346: JsonApiRequestValidator does not set cacheable metadata when the filter allows the request that was discovered in the VariationCache hardening issue
2. Fix #2719721: BreadcrumbBuilder::applies() mismatch with cacheability metadata that was discovered in the VariationCache hardening issue

User interface changes

None.

API changes

None.

Data model changes

Yes: Dynamic Page Cache would vary not just by the Response's cacheability, but also by the route access result's cacheability.