Race conditions with lock/cache, session storage - add a non-transactional database connection

Contact person responsible and ready to help

@Fabianx

Problem/Motivation

Drupal promises you can use different cache and lock backends like Memcache, Redis, etc. instead of the Database backend but that is essentially not true:

- Because Drupal uses transactions, and the same database connection is used for the cache and lock backends, there is an implicit coupling of the transaction to the cache or lock consistency.

Examples:

function x() {
  lock_acquire();
  $transaction = db_transaction();
  lock_release();
}

Looks right doesn't it? And it will always work correctly, because the lock will get released at the same time as the transaction, BUT as soon as you use another backend, you have introduced a race condition, where the lock is released, before the new data was written.

Correct would be:

function x() {
  lock_acquire();

  $transaction = db_transaction();
  $transaction->commit();

  lock_release();
}

@see #356399: Optimize the route rebuilding process to rebuild on write

Another example is the recent usage of drupal.org having to use the cache_consistent module.

Here the race condition was more like:

  node_save($node);
  // This invalidates the field cache -> with DB: okay, with memcache: BOOM
  // A new request comes in and loads the old node
  // Repopulates the field cache.
  // Now the transaction is committed.

All of this happens, because core takes an implicit coupling of cache, lock and database for granted.

Proposed resolution

- Use a different DB connection for cache and lock backends, so that writes / reads to / from those systems are no longer coupled.
- Obviously there is a drawback to several connections, so lets make that configurable.

BUT the default at least for the test suite should be to use one DB connection per backend.

OR: if we never use transactions on lock() or cache() backends, we could have one DB connection that is transactional and one that is not.

Remaining tasks

User interface changes

API changes