drupal_render_collect_attached() loses data that is not accessible via the render tree, puts in hidden data instead

Problem/Motivation

drupal_render_collect_attached() was added to collect all assets that need to be render cached.

However that function has several problems, all assuming a standard #pre_render / #cache pattern here, data shown

1. Hidden or non-accessible data is displayed

$build['non-accessible']['#access'] = FALSE;
$build['non-accessible']['#attached']['js'][] = ['do-not-access.js'];

Before the drupal_render_cache_set(), the JS would not be loaded. After it is cached, it would be loaded.

2. Data that is run via the theme layer or early rendered is not displayed

$build['#my_render_element']['#markup'] = 'I am displayed by a theme function';
$build['#my_render_element']['#attached']['js'][] = ['some-js.js'];
$build['#theme'] = 'theme_my_render_element';

Because theme() is called, but '#my_render_element' is not a part of element_children() - nor should it be (as it could be an object encapsulating some data), it is not accessible.

3. If two #pre_render / #cache calls are chained, the recursively attached data cannot be stored at all - as in the cache hit case, it does not exist (as its processed directly by render_cache_get) but in the cache miss case the data was stored leading to very strange bugs when one item expired and the other not.

That means:

Recursive render caching was almost impossible in Drupal 7.

Proposed resolution

• - Introduce a render context like in Drupal 8
• - This render_context is entered when drupal_render() encounters an element with '#cache' that is not cached,
• - The old render context is stored automatically via PHP's function call stack
• - It is exited when an old render context exists and is not FALSE
• - Ensure that all data that is processed by drupal_process_attached() is also added to the render_context
• - Add a new parameter to drupal_render_cache_set to take this render_context into account to avoid yet another static.
• - When the render_context isset, use that instead of drupal_render_collect_attached()
• - Make the whole thing configurable with a variable 'drupal_render_cache_use_render_context' that defaults to TRUE. (But for BC reasons new behavior could also be FALSE and opt-in.)

This is the MVP for good render caching in Drupal 7. In a follow-up drupal_add_js / drupal_add_css / drupal_add_library could optionally be tackled to be put into #attached as well.

Remaining tasks

• - Add patch
• - Add tests

User interface changes

• - None

API changes

API addition:

• - Add 3rd parameter $render_context to drupal_render_cache_set()

Data model changes

• - New $conf key: 'drupal_render_cache_use_render_context'