Configuration schema & required keys

Problem/Motivation

Quoting @bircher, maintainer of https://www.drupal.org/project/config_split, https://www.drupal.org/project/config_filter, etc., after I asked him What concrete things have gone wrong due to bugs in configuration, especially if they were caused by overriding/splitting/… — any of the advanced Configuration Management things?

what can go wrong, well sorting mostly... oh and to know if a key is required or not. So sometimes when a value is null or the array empty we can omit the key and sometimes not

👉 This issue aims to address the "sometimes […] we can omit the key and sometimes not" part. See #3364109: Configuration schema & required values: add test coverage for `nullable: true` validation support for the other part.

When talking about "keys" here, we're referring to the keys in a type: mapping. The root of every configuration object is a mapping — so the top-level keys you see in any MODULENAME.settings.yml file for example is a key.

For configuration management purposes it's critical to know which keys are required and which ones can be omitted. This can be entirely arbitrary: it depends on the PHP code that interacts with the stored configuration whether the key is required or optional.

In other words: today requiredness is implicit: it depends on the module's code. It's impossible to know based on the config schema. Only with trial & error can you figure this out … and even then, a single new code path could possibly cause a key that seemed optional to suddenly be required.

In general, most keys are implicitly required, with the exception of third_party_settings and subkeys of dependencies.

This leads to configuration management modules needing to add hardcoded hacks such as

              // @todo find a better way to know which elements are required.
              if ($type->getDataDefinition()->getDataType() === 'config_dependencies') {
                // Except for sub keys of dependencies.
                unset($diff[$key]);
              }

Steps to reproduce

Use modules like https://www.drupal.org/project/config_split and https://www.drupal.org/project/config_filter.

Proposed resolution

#3324150: Add validation constraints to config_entity.dependencies introduced the ValidKeys constraint. It can be configured in one of two ways:

1. ValidKeys: ['foo', 'bar']→ allows these two keys
2. ValidKeys: '<infer>'→ automatically infers the allowed keys based on the keys defined for this mapping in the config schema — this removes the need to sprinkle the entire config schema with statements like the above

But that only determines which keys are allowed, not which ones are required.

1. So let's add a new constraint, similar to ValidKeys, but doing the opposite: not just allowing keys, but requiring keys: RequiredKeys. This too should support
   

     constraints:
       RequiredKeys: '<infer>'

   i.e. by default all keys should be required.

2. Only keys that have nullable: true specified should be allowed to be omitted, and that way we reuse that existing config schema property! 👍 I realized (after a LOT of debugging 🙈) that this cannot work, because we still need to be able to distinguish between optional keys vs optional values. There are plenty of examples where some piece of configuration (some key-value pair) may be required but have an optional value — in fact, those already exist!

   For example: editor.editor.*:image_upload.max_dimensions.width MUST be present in all Text Editor config entities, but it may be set to null to indicate there is no maximum image width.

   So we need a new key. This would be specific to type: mapping. I think requiredKey: false would make sense: the absence of this in the config schema would imply requiredKey: true👍

3. Do this for ALL configuration: ALL configuration that has a non-optional value that is missing should trigger a validation error.
4. BUT:
   1. In this issue, only fix invalid config_test.* configuration, to first get the infrastructure in place.
   2. Use the \Drupal\Core\Config\Schema\SchemaCheckTrait::$ignoredPropertyPaths infrastructure that #3364109: Configuration schema & required values: add test coverage for `nullable: true` validation support added to ensure we know which configuration in core is invalid, which also makes it easy to create follow-up issues per pattern, module or config entity type — just like we did for #3361534: KernelTestBase::$strictConfigSchema = TRUE and BrowserTestBase::$strictConfigSchema = TRUE do not actually strictly validate (see #15).
   3. … which also makes it possible to reduce disruption: #3361423: [meta] Make config schema checking something that can be ignored when testing contrib modules
5. In #18, the need for conditionally required keys was surfaced and explained in detail. Rather than finding all the places where this is applicable, this does it only for a single concrete case where it is clearly needed: editor.editor.*:image_upload.status. See #18 for details. To support this, a new ConditionallyRequiredKeys validation constraint was added, which allows specifying conditionality within a single type: mapping:
   • @bircher had concerns (#25) that it would be complex for config_split, config_filter etc. to determine conditionally required keys.
   • @phenaproxima raised similar concerns on the MR.
   • Thanks to @bircher in #30, I was able to implement in #32 his proposal: no more need for an additional validation constraint that must be explicitly associated whenever conditionally required keys exist, but instead using config schema's existing dynamic typing to achieve the very same!

   See #32 for details.

   The best part: this led to the pleasant surprise that we actually already have a bunch of significant conditional config structure differences specified in our schema, for example type: core_date_format_pattern.[%parent.locked] in HEAD resolves to either core_date_format_pattern.0 or core_date_format_pattern.1 already too — this uses that exact same mechanism. 👍

This has zero effect on running a Drupal site: nor on the rest of Drupal core, nor on contrib. Because config validation does not yet run in production, only on tests, since #3361534: KernelTestBase::$strictConfigSchema = TRUE and BrowserTestBase::$strictConfigSchema = TRUE do not actually strictly validate.

Remaining tasks

Review.

User interface changes

None.

API changes

None. The actual API changes would happen in #3361423: [meta] Make config schema checking something that can be ignored when testing contrib modules.

Additions:

1. Mapping::getValidKeys()
2. Mapping::getRequiredKeys()
3. Mapping::getOptionalKeys()
4. Mapping::getConditionallyValidKeys()
5. A new RequiredKeys constraint + validator, which is used by type: mapping by default.

Data model changes

None.

Release notes snippet

N/A