[meta] Menu Links - New Plan for the Homestretch

Original report by @pwolanin & dawehner

Problem/Motivation

Since Drupalcon Portland many core/WSCCI contributors have been working (struggling) to bring a reasonable DX and coherence to the parts of hook_menu that were separate from routing but did not have a clear roadmap in the WSCCI initiative. This effort lead to developing local tasks, local actions, and contextual links as plugins using YAML discovery (after a detour through annotation-based discovery).

Menu links have remained the odd character out. A plan to convert them all to content entities has been in progress for a long while. pwolanin spear-headed the move of the "default content" that creates these entities from hook_menu (allowing use to remove hook_menu) to hook_menu_link_defaults, but it remained the only core system creating such default content. In addition, there have been unresolved performance problems moving these to "NG" entities. Of course, there were important motivations for using content entities, especially for user-facing links where one might add additional fields to the entity to expand the possible rendering options.

A much bigger problem than this being a one-off system is that Drupal 8 core does not have (and is very likely not to get before 8.0.x) a system for providing default content translations and keeping them in sync from localize.drupal.org. Since menu links were being implemented as content entities, this meant that 8.0.x was likely to have a major regression from 7.x in that a most of the admin interface would not be localizable upon install.

Brainstorming at DevDays Szeged, a possible solution became apparent. We could use a plugin type as an over-arching framework to build both static menu links that could be localized (for the admin sections) and also, in analogy to the custom_block entity type that is mapped to a block plugin instance, we could make a custom_menu_link content entity. This would be mapped to a menu link plugin instance and provide for the more user-facing sections of the site all the features that originally motivated the conversion of menu links to content entities.

Some people will ask "Why plugins?".
Some arguments in favor:

• local task, local action, and contextual links are plugins using YAML discovery - thus making menu links plugin brings greater coherence to the new system
• The existing plugin code can be leveraged to avoid writing a lot of new one-off code
• It helps enforce a clean abstraction of the plugin definition as totally distinct from the manager & storage that knows how to efficiently build and manipulate trees
• The exact behavior of any individual link can be controlled by setting/altering its class - doing away with hacks (that pwolanin originally wrote) like hook_translated_menu_link_alter()

Some against:

• Menu links don't really fit the notion of plugins as mostly being distinct implementations/classes (but neither do local tasks, etc., so this is water under the bridge)
• We are creating and updating definitions via the plugin manager without doing a full discovery (rebuild). This is really a scaling measure - if we were willing to assume sites would have relatively few menu links, it could all be managed via discovery. But considering sites with 1000s of nodes each with a menu link, this would fail. Some plugin manager in core already use such a mix of discovery and static addition of definitions.

Proposed resolution

Based on this plugin-based plan we (mostly dawehner) executed a first step to move from the hook to YAML files so that at least the storage format of menu links would immediately be consistent with other hook_menu-derived plugins and so upgraded modules would not need additional changes to declare their admin links even as the back-end was changed.

From this first step (#2226903), we contemplated 2 additional steps, including an incremental refactoring of the tree storage out from the entity (#2227179), and then a conversion to plugins (#2227441). However, as dawehner and pwolanin started working on the 2nd step, it became apparent that much of the work would be wasted if we intended to attempt the final step, and that the work to go from current to an initial version (phase 1 below) was roughly equal in scope. Thus, we pivoted to focus on the final step instead.

Based on the progress to date and limited discussions with core committers, we propose that we proceed to this final step. To avoid endless HEAD conflicts, build momentum, and get this new code in use, we also propose that the new system be committed in phases with the full knowledge that there will be some apparent regressions relative to 7.x until the full plan is completed.

Proposed phases (on top of existing YAML conversion):

• Phase 1: review new APi and classes, but commit with phase 2
  1. New Features
     • Define an extensible framework for different types of links
• Phase 2: Admin links and views, editable via menu_ui module, menu_link_content module and content entity
  
  1. New Features
     • Manage links defined in YAML and links defined by views
     • Full l10n support for localizing the D8 admin interface
     • i18n support via config translation for translating the links provided by Views.
     • An NG content entity whose base table stores the plugin definition
     • The custom menu link entity would be fieldable
     • Link title is the (translatable) entity label
     • Menu link content entity can be added/edited via the node form (substituting for existing functionality)
  2. Regressions relative to 7.x: none
• Phase 3: menu link field
  
  1. New Features
     • Define a field that provides a menu link for an entity
     • Field API goodness: multiple values or fields per bundle
     • Add the field to any fieldable entity
     • Link title (translated) always taken from entity label
     • Link from the field is specifically connected to the entity, unlike D6/7/HEAD where the 1st link that matched the node path was used in the node edit form.
  2. Regressions relative to 7.x: none

The menu link field in Phase 3 was not part of our original plan in Szeged, but has emerged as a logical solution from discussion of how to generically provide a menu link while editing any entity. This field would store a very limited amount of data - basically weight, menu_name, and parent values, and when the link is rendered would load the entity to get the current label. The corresponding definition stored via the manager would include those field values and a few more like the route name and parameters.

Architecture and scaling:

It would not really be possible to register all the proposed custom menu links and menu link fields (node menu links) using the normal strategy of plugin discovery. You could end up trying to build an array in memory with as many elements as there are pieces of content on the site - potentially 10's to 100's of thousands or more. We propose to extend the plugin manager to allow plugin definitions to be added or updated one by one. In the case of plugins that do participate in discovery (current static menu links and Views menu links), they also need to persist the update so that it is not overwritten the next time discover happens. For example, by also updating the View config entity.

The pattern of a plugin manager that uses a combination of "normal" plugin discovery with additional definitions that are directly added is already present in core. For example, in \Drupal\Core\Validation\ConstraintManager. This makes use of a class extending \Drupal\Component\Plugin\Discovery\StaticDiscovery while has a setDefinition() method to add a definition directly.

The architecture supporting menu link plugin manager is that it utilizes a tree storage service which knows how to add a definition to the tree hierarchy, and how to retrieve all the definitions that are part of of the request tree. In this way, the tree storage replaces the use of a cache to store discovered or statically added plugin definitions, while also providing optimized retrieval. This forces the tree storage and rendering be totally decoupled from the plugin registration/discovery side which helps enforce a clean separation of concerns.

The default tree storage implementation in the patch is simply an adaptation of the materialized path schema and code that's backed menu links since Drupal 6. However, the storage service now has a clean interface, so the implementation could be replaced with a different SQL algorithm, or a noSQL store. The plugin manager only gets the plugin definition back, not the value of the mlid, p1...p9, or other internal details of the tree implementation.

User interface changes

Overall, the final site building experience will be similar or better than 7.x. Significant user-facing changes relative to 7.x:

• Users can only enable/disable and edit menu, parent, and weight for static (YAML-defined) links.
• A field providing a menu link means that configuration is via field/widget settings, not via node-type settings.

Timing and remaining effort

Phase 1 is already significantly complete (> 50%) in approx 10 days of part-time effort by pwolanin and dawehner. We have the plugin manager and storage working, the admin interface and toolbar works, main and secondary menus work, and you can use the menu_ui interface to re-order and re-parent links. You can create links in Views and have them appear together with other links, and editing the link updates the related view. Especially if we can get additional assistance, it's possible this could be done in another < 2 weeks.

Phase 2 could be subdivided if we want to accelerate it. Phase 2a could save the data to a table using a form provided by the plugin but nothing from the entity API. This table that would later become the entity base table, but we could make custom links work without implementing the full entity. Phase 2b would building the full entity. This work could begin in parallel with polishing Phase 1 for commit, and could follow by 2-3 weeks.

Phase 3 will build on the code and data model already defined, so the work really just involves defining a custom field and its widget, and integrating the plugin definition save to the field save code. This could start in parallel with Phase 1 or 2. Since almost all the pieces will be reusing existing code, 2-3 weeks after Phase 2 should be possible.

Follow-up work: using a different theme function or updating theme_links and related functions to work with a #type => link render array would be needed allow us to fully leverage the power of using entities and fields. It's possible this could be incorporated into Phase 2 or 3.

API changes

For modules already using YAML discovery for admin links: none

For modules interacting more directly with the menu link system, they would need to be re-written to interact with the new menu.link_tree service (the plugin manager).