Add Alpha level Experimental Package Manager module

Please DO NOT push to this branch. See How to Help for instructions for working in the contrib module.

Overview

Package Manager is an API-only module which provides the scaffolding and functionality needed for Drupal to make changes to its own running code base via Composer. It doesn't have a user interface.

The easy questions to answer: Why?

Why build this if not everyone can use it?

• Two of the current strategic initiatives, Automatic Updates and Project Browser, are creating user interfaces to run Composer commands. Package Manager was created to facilitate this common need.
• These two initiatives both have known restrictions, in that they will not be usable on all sites because of the system requirements. The main system requirements are that the codebase must be writable and the Composer executable must be available.

System requirements:

All of the following system requirements have corresponding Package Manager validators (see section TBD) which help determine if the system is currently compatible with Package Manager. Modules that use Package Manager may display any validation errors in their respective user interface.

1. The codebase must be writable by the web server. Because this is intrinsic to the purpose of this module and the two modules that depend on it, this requirement is unlikely to change in the future. Although this will prevent Package Manager from working on some hosting environments, some users may use Package Manager in local or cloud environments where the file system is writable, even if their production environment is not. Project Browser is the most obvious example of this, because installing modules is common when building a site and is often done locally. In Automatic Updates, updating manually via the UI (rather than automatically during cron) may be done by some users in development environments, which is beneficial even if Automatic Updates cannot be used directly in the production environment.

   To provide Automatic Updates to site even if the web server cannot write to the file system we are working on #3351895: Add Drush command to allow running cron updates via console and by a separate user, for defense-in-depth. In this case a outside cron job would be set up to run the console command perform and Automatic Updates. The codebase would have to writable by the system user setup to run the cron job but not the user running the web server.

2. The executable composer.phar must be available somewhere on the system and runnable by the web server using proc_open (i.e., through the Symfony Process component). It's used to inspect the codebase and to install or update packages.
3. The web server must be able to write to a temporary directory (i.e., the one returned by \Drupal\Core\File\FileSystemInterface::getTempDirectory), which must be outside of the web root. (Because Composer commands may fail, file system operations may fail, package downloads may fail, and so on, we run Composer commands not on the live site, but a copy of it.)
4. Certain types of filesystem links cannot be present in the codebase → for example, hard links would prevent running Composer commands in isolation. See the NoUnsupportedLinksExist precondition documentation for details. The most common type of link is the symlink, which is supported, with one exception due to a PHP limitation: see also Symlinks to directories don't work with PhpFileSyncer.
5. The Drupal site cannot be part of a multisite, due to the danger involved with changing the codebase of several sites at once.

Biggest risk: breaking the live site

Destructive Composer operations are never done in the live codebase - only in the staged copy created by Composer Stager. If a Composer operation fails there, the live site will be unaffected.

While unlikely, it is possible that while copying the files from the staged version of the codebase back into the live site, an unexpected error or failure could occur. This is hard to avoid because any filesystem operation will always has some chance of failing.

Package Manager takes several steps to ensure that users are informed about failed operations.

At the last possible moment before Package Manager asks Composer Stager to copy staged changes into the live site (thus overwriting the files of the live site), it writes a "failure marker" file, which it deletes after the copy succeeds. If Composer Stager raises an error during the copy, the "failure marker" file is NOT deleted. The presence of this marker file, then, indicates that the staged changes were only partially copied, and the site's codebase is corrupted. In this situation, the site's codebase should be restored from a backup; Package Manager will also flag a requirements error about this.

The "failure marker" file method is used instead of tracking state somewhere in the database, because it's an atomic operating system operation completely independent of Drupal's state.

Biggest challenge: testing

Testing Package Manager presents different challenges than everything else in Drupal core.

Among these challenges are:

• Testing actual Composer commands for both non-destructive and destructive operations. To fully test the system and its many parts we needed to be able to test all those parts as realistically as possible, with actual, unsimulated Composer commands...without testing the internet.
• Testing codebase replacement (i.e., the "apply" phase) without replacing the actual codebase that is running the test.

We tried several approaches to testing that we found did not work with the above challenges:

• Relying on vfsStream in kernel tests: Composer commands don't work with VFS. It was also much harder to debug tests, because determining the state of the virtual "active" and "staged" directories was nearly impossible. Our kernel tests now use the real filesystem.
• Altering Composer metadata files directly: In an effort to avoid bad performance from running many Composer commands in many tests, we were first altering the Composer metadata files (like composer.lock, installed.json, and even installed.php) directly to simulate changes made by Composer. This worked for a while, but during #3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility we realized that our alternations were not sufficient to be usable by real Composer commands, and therefore made the tests unrealistic and hard to maintain.

Key components of our current testing infrastructure are:

1. MockPathLocator: This class allows tests to have a directory other than the real Drupal codebase used as the "live" directory. This lets us freely alter the "live" directory without altering the codebase that is running the test.
2. Valid Composer testing fixture: All of our kernel and functional tests that deal with the stage life cycle start with a "live" directory that is created by cloning a single test fixture. This test fixture is a valid Composer project that can be created using composer install by a development script that will be shipped with core. A test that runs composer validate on this fixture is also included to ensure any future updates to the fixture are still valid as far as Composer is concerned.
3. FixtureManipulator: This class provides the ability to alter the active and staged codebases with real Composer commands. After each change, composer validatei s executed to make sure the changes did not have any damaging side effects.
4. Functional tests: Because Package Manager is an API-only module, it only has one functional browser test. The testing infrastructure described above works with functional browser tests as well, though, as demonstrated by the Automatic Updates module's functional tests. We believe this proves that it is flexible enough to cover the future functional test requirements for when both Automatic Updates and Project Browser are added to Drupal core.
5. Build tests: The test fixture used by kernel and functional tests is a valid Composer project, but is not a fully bootable Drupal site. For this reason, our build tests create a fully functional Drupal project, using the core templates, which updates its own code via Package Manager.

How Does it work?

At the center of Package Manager is the concept of a stage directory. This is a complete copy of the active Drupal code base, created in a temporary directory that isn't publicly accessible.

Package Manager's interaction with the stage directory happens in 4 phases during the stage life cycle:

1. Create: A new stage directory is created and the codebase that is managed by Composer is copied into it. Any site-specific assets that aren't managed by Composer, such as settings.php, uploaded files, or SQLite databases, are omitted.
2. Require: One or more packages are added or updated by Composer in the stage directory.
3. Apply: The staged codebase is copied back into the live site.
4. Destroy: The stage directory is deleted.

External dependencies

• php-tuf/composer-stager: #3331078: Add php-tuf/composer-stager to core dependencies and governance — for experimental Automatic Updates & Project Browser modules This library allows Package Manager to run Composer commands in an isolated copy of the codebase. This is important because:
• Running Composer commands directly on the live site would require the site to be offline for the entire time the Composer command is running. Using Composer Stager allows modules that use it to only put the site in maintenance mode during the copying of the files back to the live site (the "apply" phase).
• Running Composer commands on a staged copy of the codebase allows us to inspect/analyze/validate the changes that have been made before they are copied into the live site. (For example, Package Manager ensures that any contrib Drupal projects that are changed in the staged codebase are secure and supported; see SupportedReleaseValidator.)
• Composer Stager is owned and maintained by the Drupal community, but is developed on GitHub, outside of the Drupal namespace, to enable other PHP projects to also use it.
• colinodell/psr-test-logger: Makes it much easier to test code that files log entries (mainly unattended updates during cron, which has no other way to report errors). See #3321905: Add colinodell/psr-test-logger to core's dev dependencies→ already committed! (Note that the functionality of this package was previously available in psr/log, which is an existing dependency of Drupal core → that package removed this functionality in its latest major version, which was adopted by Drupal 10.)

Security: d.o's Composer package signing

• Package Manager ensures that the site requires The Update Framework (TUF) to download packages from drupal.org's Composer endpoint. This means two things:
• The PHP-TUF Composer integration plugin is installed and enabled: https://github.com/php-tuf/composer-integration
• The https://packages.drupal.org/8 Composer repository is opted into TUF protection
• The PHP-TUF Composer integration plugin itself is completely independent of Drupal and, if enabled, alwaysenforces TUF protection for everything in opted-in repositories. That means if the site administrator runs Composer commands at the terminal, they get the same protection! It's important to note that it does NOT protect anything that belongs to a repository that is not opted in to TUF protection. So it will effectively only protect drupal/* packages, at least initially. (With the notable, current exception of the core packages, like drupal/core and drupal/core-composer-scaffold, which are part of the main Packagist repository that doesn't currently have TUF protection.)
• See #3325040: [Packaging Pipeline] Securely sign packages hosted on Drupal.org using the TUF framework and Rugged for the ongoing work to deploy TUF signing for drupal.org-hosted packages.

Public API

The API of Package Manager can be broken down into these areas:

1. Stage life cycle events: Modules can subscribe to the events dispatched during the stage life cycle. There are Pre- and Post- events for all the phases of the stage life cycle. Any subscriber to the Pre- events can flag validation errors that will stop the life cycle from proceeding until the errors are resolved.
2. Package Manager provides some useful services for analyzing and comparing the state of the live and staged codebases; in particular,PathLocator and ComposerInspector.
3. The Stage class creates the stage directory and performs the stage life cycle phases, Create, Require, Apply and Destroy as described above. Modules that want to perform other, specialized Composer operations should extend the Stage class.

For a more detailed explanation of the API see package_manager.api.php in the merge request.

Dependency evaluation

• php-tuf/composer-stager: #3331078: Add php-tuf/composer-stager to core dependencies and governance — for experimental Automatic Updates & Project Browser modules
• colinodell/psr-test-logger: #3321905: Add colinodell/psr-test-logger to core's dev dependencies

How to help

Please DO NOT push to this branch!! Because this merge request is still being automatically converted from the 3.0.x version of the Automatic Update contib module where Package Manager is sub-module. Any changes made directly to this MR will likely be lost in automatic conversion process.

Feel free to leave feedback on this merge request. If you would like to help address the feedback please search the 3.0.x issue queue for the contrib module to see if any existing issue exists and if not create one in that project.