Add php-tuf/composer-stager to core dependencies and governance — for experimental Automatic Updates & Project Browser modules

Problem/Motivation

The package_manager module that will be added to Drupal core for both https://www.drupal.org/project/automatic_updates and https://www.drupal.org/project/project_browser has a dependency on:

php-tuf/composer-stager":"^1.2"

— see the composer.json

When Automatic Updates or Project Browser needs to run Composer commands that will modify any of the site's code (e.g., composer require, composer update, or composer remove), the Composer Stager library is what copies the site's entire codebase to a separate directory, runs the composer commands in that separate directory, and then syncs the changes back to the site's real codebase. In other words, this library is what ends up modifying the site's actual code files, so it's essential that we have confidence in its reliability and security, both when we initially add it to Drupal core, and ongoing after that.

So far, 99% of Composer Stager has been written by @TravisCarden, a Drupal contributor working for Acquia as part of Acquia's investment in the Automatic Updates initiative. The Composer Stager library does not make Drupal-specific assumptions and was written with the hope that other Composer-managed PHP CMSes and applications could also some day benefit from it and contribute to it.

Proposed resolution

Place the Composer Stager library into core governance. This would mean, for example, all Drupal core committers having commit access to it, following Drupal core review processes for committing to it, and having a security policy in the repo that states that security issues should be reported according to Drupal's security reporting process.

Dependency evaluation

Per https://www.drupal.org/about/core/policies/core-dependency-policies/depe... :

Repository

https://github.com/php-tuf/composer-stager

Code quality

The project has extensive unit and functional test coverage, strict static analysis, and automated coding standards.

Maintainership of the package

Currently by @TravisCarden, but the Proposed Resolution is to place this under Drupal core maintenance.

Security policies of the package

The current github repo does not have a security policy, but the Proposed Resolution is to place this under Drupal core maintenance and therefore under Drupal core security policies.

Expected release and support cycles

TBD

Other dependencies it'd add, if any

Composer Stager has a runtime dependency on symfony/filesystem and symfony/process. symfony/process is already a production dependency of Drupal core, but symfony/filesystem is currently only a dev dependency of Drupal core, so this would promote that to a production dependency.

👉 We plan to remove the dependency on Symfony/filesystem. (See Remaining tasks.)

Composer Stager also has a long list of dev dependencies.

Remaining tasks

• Remove the dependency on symfony/filesystem (https://github.com/php-tuf/composer-stager/issues/60).
• Since the repo will be under core governance, perform a dependency evaluation of all dev dependecies, as we would if we were adding that as a core dev dependency (https://github.com/php-tuf/composer-stager/issues/78).
• Review the code that's in Composer Stager for core quality standards.
• Add needed governance documents (security policy, links to Drupal core governance documents, etc.) to the repo. (https://github.com/php-tuf/composer-stager/issues/79)

Code overview for reviewers

Composer Stager's README and Wiki, which includes a glossary, are a good place to start. In addition to what's written there, the following is a quick orientation to the project's codebase. Read at least either that README or that glossary first, in order to understand the purpose of Composer Stager's four core operations (Begin, Stage, Commit, and Clean), before reading the following.

Onion architecture: Domain and Infrastructure

The repository has two top-level directories within /src: Domain and Infrastructure. These terms have the meaning that's described in The Onion Architecture: the Domain is the inner layer that depends only on itself and describes and connects together the scope of Composer Stager's responsibilities, but doesn't actually do anything (as in, doesn't copy files, execute Composer commands, etc.), and Infrastructure is the outer layer that implements all of the concrete classes that connect Composer Stager to the outside world (the file system, Composer, etc.). More specifically:

• The Domain layer contains the interfaces, some abstract classes, the exception classes, and the concrete classes for Composer Stager's core operations: Begin, Stage, Commit, and Clean.
• The Infrastructure layer contains all of the concrete classes other than the exception classes and the Beginner, Stager, Cleaner, and Committer classes mentioned above.

With respect to semantic versioning, all of the interfaces and classes in the Domain directory are the public API, and the classes in the Infrastructure directory are @internal.

Preconditions

For each of the 4 core operations (begin, stage, commit, clean), there are preconditions that must all pass. For example, for the begin operation, there's a precondition that the staging directory must not already exist. See the constructors of BeginnerPreconditions, StagerPreconditions, CommitterPreconditions, and CleanerPreconditions, to see the required preconditions of the corresponding operation. These may each include other aggregates (implementations, interfaces) of preconditions and/or individual (implementations, interfaces) preconditions.

Several preconditions include checking for the presence of hard links or unsupported symlinks. The reason for checking for these is that the premise of Composer Stager is that once the active codebase is copied to staging, then Composer operations in staging won't affect the live site. However, if the codebase includes, for example, symlinks to absolute file paths, then copying that to staging will result in the staging copy pointing to the same place that the active codebase points to, which would open the risk that something that writes to that could end up affecting the live site. However, relative symlinks that don't traverse outside of the project root are allowed (except on Windows, which adds other challenges to working with symlinks that we decided to punt to later), because having those is common for codebases that include Drush, local path repositories, or other use cases.

Several preconditions, such as the ones that check for hard links and symlinks per above, require iterating every directory and file in the codebase being operated on. This iteration is performed by AbstractFileIteratingPrecondition, which uses RecursiveFileFinder, which in turn uses PHP's RecursiveDirectoryIterator. Because the codebase could be quite large (e.g., a site with a lot of contrib modules) and iterating each file in PHP could be slow, there's an open issue to evaluate and document the performance of it.

File syncers: Rsync and PHP

The Begin operation copies the codebase (the entire Drupal application) from its live location (the "active" directory) to a temporary location (the "staging" directory). The Commit operation (which is called after all of the desired Composer commands have finished operating on the staging directory) copies back. On systems that have rsync available, RsyncFileSyncer is used. Otherwise, PhpFileSyncer is used. We expect RsyncFileSyncer to be significantly faster than PhpFileSyncer, especially for the Commit operation which only needs to sync back the files that are different (i.e., the ones that changed as a result of the Composer commands that ran in the staging directory), which rsync is very efficient at doing. The speed of the Commit operation matters, since Package Manager puts the Drupal site into maintenance mode during that time so as not to serve requests during the time that files are being copied from staging to active (i.e., when the live codebase is in a partially updated state).

Exclusions

One of the parameters to Beginner::begin() and Committer::commit() is $excusions. Composer Stager is un-opinionated about what codebase it's being used on, so it defaults this to NULL, but Package Manager passes several directories to it, including sites/default/files. This is the set of paths to exclude from copying. This is so that the Drupal site can still serve requests (and content editors can still be uploading files) while Composer commands are running in the staging directory. Without excluding these, the Commit operation would delete/undo changes that were made on the live site since the time that the Begin operation ran.

Release notes snippet

php-tuf/composer-stager is now a dependency, which enables the experimental Automatic Updates and Project Browser functionality.