Path aliases can be "forgotten" after a cache rebuild due to race condition in CacheCollector

I just a launched a D8 site this morning which receives ~3 requests per second to the web servers. I did a code deploy a few hours later which involves a few things:

1. Changes the value of $settings['deployment_identifier'] (as a workaround for #2752961: No reliable method exists for clearing the Twig cache)
2. Runs a config-import
3. Runs a cache-rebuild
4. Run a "cache warmer" which just crawls a bunch of pages.

Immediately after this deploy, all path aliases on the site were seemingly lost. They still existed in the database, but all links throughout the site for nodes were using the node/[nid] path instead of their path alias.

This happened once before to us during beta testing of the site, but was unable to come up with a definitive reason and wrote it off as a fluke. When it happened, I spent some time trying to debug it, including cloning the database down to my local environment, but I couldn't reproduce it locally. It had to have something to do with the cache. I couldn't easily inspect the cache because we were using Memcache on the web servers. So, I just decided to run `drush cr` and that resolved it. All links were using the proper aliases like they should.

This time around, I began my investigation based on behavior I noticed in my cache warmer script. It works by first loading all nodes of a certain type and getting their alias like so:

  // Now add all nodes of types we care about. We leave out news and events.
  $alias_manager = \Drupal::service('path.alias_manager');
  $node_storage = \Drupal::entityTypeManager()->getStorage('node');
  $paths = [];
  $nodes = $node_storage->loadByProperties(['type' => ['landing_page','chaptered',
    ],'status' => Node::PUBLISHED,
  ]);
  foreach ($nodes as $node) {
    $paths[] = $alias_manager->getAliasByPath('/node/' . $node->id());
  }

It then makes an HTTP request to that path and outputs each path as it's doing it. All of the paths it collected were just the /node/[nid] instead of the alias. So the alias manager was NOT returning their alias, even though their alias certainly exists and were not deleted.

So, it has something to do with the AliasManager service, caching, and probably some race condition. Note that we've done several other deploys today with the same amount of traffic hitting the site and did not experience this issue.

This is quite scary for us. It caused several infinite redirect loops due to caching in Varnish and the Redirect module which was redirecting users to the "canonical" path.

I'm going to keep investigating and update the issue with any findings.