Updated: Comment #4
Problem/Motivation
Much of the code in our controller pipeline was written with the assumption that it would be replaced "later", after the system had matured a bit and SCOTCH figured out what it wanted to do to it. Unfortunately, that hasn't happened yet and for-reals SCOTCH is not happening in Drupal 8 core. That means we have a number of pieces of code that are not in a releaseable state. Most notably, the same render logic appears in HtmlPageController and in ViewSubscriber, the latter I think twice at least, and some of it is probably never even called. I've lost track of which of those code paths are even called.
Additionally, with the lack of global state (yay!) some modules that did wonky conditional logic in strange places no longer can do so. Some of that wonky logic is completely valid, though. For example: #1969270: 404 pages: drupal_get_http_header('Status') returns no status code at all . (I'm marking this critical due to such regressions.)
We still need to eliminate the subrequests in the wrapping controllers, as they're causing more trouble than they're worth. See #1945024: Remove subrequests from central controllers and use the controller resolver directly. .
Putting arbitrary pages into a dialog is clunky.
And of course there's the problem that using an anonymous undocumentable array as a universal data structure is just a bad idea in general, even if that's what we're doing now.
So, let's clean up this pipeline to be more consistent and structured in ways that can address the above issues.
Proposed resolution
The conerstone of this solution is to introduce a new HtmlPage object, which is strictly a data object representing a page. By that point, the "body" of the page is already string-ified, ie, the render array is already rendered. The rest of the page is really just the body and html classes, plus the contents of the HEAD element, but using structured objects rather than arrays.
Additionally, there was a lot of overlap and duplication between HtmlPageController and HtmlFormController, and some unpleasantness to make the latter work through a dialog. After some discussion with dawehner, we came up with a solution that greatly simplifies the concepts around controller handling.
Rather than _form in a route triggering a different wrapping controller, the main wrapping _controller is determined exclusively by the Accept header. The FormEnhancer now fills in only the _content property. Since forms need extra processing after the form object is called, that is handled by setting _content to an interstitial object that calls out to a dedicated interstitial FormController, which replaces HtmlFormController entirely. This is actually closer to what EntityRouteEnhancer was already doing, but it needed a little updating just the same. (This was originally done using a closure, which was far far simpler, but Drupal kept trying to serialize it.)
What that means is that any wrapping _controller can rely on _content being "the thing that gives you a render array that is your body". That thing may be a service, or a controller string, or a closure, doesn't matter. It returns the body, and the wrapping _controller does with it whatever is appropriate.
"Appropriate" in this case is to either return a Response (for the Ajax API-using controllers) or to return an HtmlPage object. It is never, ever to return a render array further down the pipeline. Just an HtmlPage object. (The legacy controller's closure has a lot of duplicated code at the moment, which I don't care about since that's slated for execution anyway. I decided it wasn't worth the effort to convert that to a dedicated interstitial class like Form, although it's nominally possible.)
It is then the responsibility of a view listener to, only, render HtmlPage into an HTML string. At present that's still done using the same tools (render arrays and html.html.twig), but we've therefore separated the body from the wrapper. The Twig folks, according to Jen Lampton, have bee considering not using the full render pipeline for the HTML tag, and this splits the problem space in two so that they could do that if desired.
This pattern of "return structured object from controller / mutate to Response in the view listener" is one I've seen and used elsewhere in the Symfony ecosystem. I actually built a project on it recently at work in Silex and it worked out extremely well.
The net result is that there is a list of data structures of increasing levels of abstraction: String -> render array -> HtmlFragment -> HtmlPage -> Response. A _content callable can return any of those it wants. A _controller can return any of the last 3, which are at the level of granularity that makes sense to be coming back from the controller layer. View listeners then convert any of those objects to the next higher object, until we get to a Response. Contrib listeners may then intercept that process at any level they want to modify the object or take over the entire process. That's a very low-complexity but high-power pipeline that gives contrib developers an enormous amount of flexibility.
Remaining tasks
There's some odd margin issues on pages rendered through the new system, where the page doesn't make it all the way to the edge of the screen. I suspect a class on the html element is missing along the way.- FIXED!The drupal_render_page() calls in overlay_render_region() and the Views Page plugin still need to be converted to use HtmlPage, or possibly HtmlFragment in the former case. That lets us remove drupal_render_page() entirely and remove the split-render for html/html_page and page/page_body- FIXED!- ExceptionController is still horrid. We should probably not try to un-horrify it here, but do that in a follow-up that we'd need anyway. (That is, this is not new technical debt.) ViewSubscriber could also still use work.
The handling of HtmlFormController right now has probably broken forms-in-dialogs. I'm still talking through how to fix that with dawehner. Suggestions welcome.- FIXED!- We probably need to do some trickery to allow an Exception controller to still call a View listener, so that we can expose non-200 HtmlPage objects to view subscribers. That's needed for #1969270: 404 pages: drupal_get_http_header('Status') returns no status code at all . That may or may not be in this issue. If we don't do it here, though, it's still not a new bug as that issue shows contrib can't do what it needs now anyway.
- This does not do anything interesting with assets (CSS/JS). There's some stubs in there from SCOTCH for doing so, but that is a follow-up. (Also not new technical debt.)
- HtmlEntityFormController is now misnamed, since there's nothing page-centric about it. EntityFormController is already taken as a class name, however. Suggestions for an alternate name are welcome. :-)
Note: To be clear, this is *not* SCOTCH. We don't get any new layout capabilities with this. Some of the code was borrowed from SCOTCH, but not much. Really all this does is move around the code that SCOTCH would be replacing in contrib anyway. (Mainly HtmlPageController, as it would no longer be concerned with view listeners. How SCOTCH/PanelsTNG goes about creating an HtmlPage object is its business, as long as it returns one.)
Jen Lampton also mentioned that RenderWrapper may go away later. This issue does not make that any easier or harder.
User interface changes
N/A
API changes
For most module developers, minimal if any. The main change is an addition: The HtmlPage object is available in view listeners, which should let them modify HTML head information in a more structured way.
For internals folks, the page is no longer rendered as one massive array from HTML tag to HTML tag. Instead, it is rendered as a slightly less-massive array for the contents of the body tag, and then separately for the HTML page itself, treating the body as an opaque string. The number of people affected by that can probably be counted on one hand and are all core developers working on Twig. Jen Lampton confirmed at MWDS that this is consistent with the direction they're going anyway.
Related Issues
#1839338: [meta] Remove drupal_set_*() drupal_add_*() in favour of #attached/response for out-of-band stuff
#1969270: 404 pages: drupal_get_http_header('Status') returns no status code at all
#1871596: Create a PartialResponse class to encapsulate html-level data [Moving to sandbox]
#1945024: Remove subrequests from central controllers and use the controller resolver directly.
#2028749: Explore addressable block rendering
(blocked by this issue)
Notes
Development is happening in the WSCCI sandbox, in the htmlpage branch here: https://drupal.org/node/1260830/git-instructions/2068471-htmlpage
Please don't just post patches; at least include interdiffs, or better yet let's collaborate in the sandbox.