Make the entity storage system handle changes in the entity and field schema definitions

Updated: Comment #113

• This is being coded in the D8MI - Entity Translation sandbox.
• Test issue: #2298525: Test issue for Make ContentEntityDatabaseStorage handle changes in the entity schema definition.

Problem/Motivation

In #2183231: Make ContentEntityDatabaseStorage generate static database schemas for content entities we introduced the capability for the entity storage to automatically generate the SQL schema for any entity type. This is currently limited to static generation at install time and concerns only the tables described in the official entity storage SQL table layout standard. That is the ones storing values for base fields. At the moment dynamically handling changes in the entity type and field definitions, and reflecting them in the SQL table layout, is not supported.

This is a critical missing piece towards achieving a whole set of improvements:

• Modules are no longer required to inject random data into the $entity object during the load phase, since all the data retrieval is handled by the storage itself. This will motivate module authors to define their data through field definitions, instead of using random properties, with all the related benefits, such as automatic Rules integration, typed-data validation and so on.
• Any entity type leveraging the core entity storage API, natively supports revision and translation without requiring module authors any additional effort.
• Translation and revision support can be independently enabled/disabled by just altering the entity type definition. This makes it easy for site developers to adopt the table layout that is the most performant with respect to their business requirements.
• Additional base fields can easily be added to the native entity table layout, making it straightforward to implement highly-efficient denormalization strategies.
• The API is designed without having a particular storage backend in mind. This means module authors can leverage a storage-agnostic API without any additional effort required.

Proposed resolution

Part 1: unified API

• Extend the content entity schema handler implemented in #2183231: Make ContentEntityDatabaseStorage generate static database schemas for content entities to be able to handle the addition/removal/update of field storage definitions, transparently handling differences between single-valued base fields, which are stored in the entity tables, and bundle fields or multiple-value base fields, which instead are stored in dedicated tables.
• Make the content entity schema handler able to handle changes in the entity type definition for the revisionable and translatable properties.
• Move the Field SQL Storage (private) API to the (default) table mapping implementation, which is the SQL specific part of the core entity storage API, responsible for representing how fields are mapped to tables.

Part 2: UI and data handling

• Implement a new EntitySchemaManager service responsible for spotting differences between the installed schema and the one generated from the current versions of the entity type and field storage definitions. Probably leveraging state to track changes.
• Based on the current Field Storage API limitations, implement a set of validation rules so that only changes that do not imply a data migration are applied to the schema when data exists, that is:
  • adding a field table
  • adding a custom field column to a base/data/revision table
  • switching from revisionable to non-revisionable
  • switching from translatable to non-translatable
• Any change not (yet) reflected in the installed schema is reported as an error in the status report, which offers a link to the update.php script to fix the database schema. The update.php script calls the entity schema manager to trigger any allowed schema change.

See #48 for a detailed description of the implementation plan: it's a bit outdated but many details are still valid.

Remaining tasks

• Implement part 1
• Implement part 2
• Reviews
• Party!

Follow ups

• #2282119: Make the Entity Field API handle field purging
• #2248977: Support multi-value base fields in ContentEntitySchemaHandler
• #1740492: Implement an entity views data controller
• #2079019: Make Views use SqlEntityStorageInterface

User interface changes

• Entries added to the status report page.
• Minor tweaks to the update.php UI.

API changes

• All the Entity Field CRUD API has moved to the entity schema handler.
• All the Field SQL storage (private) API has been moved to the default table mapping.