Problem
Variable caching has a number of inter-related issues.
Memory and cache object size
All variables are loaded from the {variables} table regardless of whether they're in use or not, and some older Drupal sites can have over 2,000 variables in the table since many dynamic variables (and etc.) are not properly tracked between upgrades and module uninstalls. This leads to a large cache item, and since it is assigned to a global, also high memory usage in the critical path. The cache item size ranges from around 100k on a brand new site to 1.4mb or more on a site with lots of contrib modules that has been around for a while.
Locking
When variable_set() is called (which some modules and even core doa lot, the variable cache is cleared entirely. It then has to be rebuilt in full by the next request. In Drupal 7 we added a lock so that only one request does this at a time, which prevents a stampede on the variable cache itself, but can also mean processes hanging around waiting until the single process that acquired the lock is done - since it is a global read lock on every single process that hits the Drupal install.
If you have an operation that involves a large number of variable_set() - for example rebuilding css and js aggregation with a large number of bundles - #886488: Add stampede protection for css and js aggregation, or a failing drupal_http_request() due to a third party service going down, #959620: drupal_http_request() can lead to variable_set() stampede, then this happens:
* Process 1 calls variable_set(), clears the cache.
* Process 2 acquires a lock, starts to rebuild the cache.
* Processes 3-200 all get a cache miss and fail to acquire the lock - since this is variable_initialize() that's for every single request to the site - cached pages, imagecache generation, you name it.
* Process 2 frees the lock after 100ms or so.
* Process 4 acquires the lock since it wants to do a variable_set() again.
* process 5-200 are still waiting.
* Despite load being low, apache runs out of spare threads, connections get backed up etc.
Proposed solution
Since the variable cache is a single array that includes lots of stuff that is rarely used, it is a good candidate for DrupalCacheArray. This has the following benefits:
- variables that are stale in the database or only requested very rarely don't get put into the cache item.
- there is no longer a need for a global read lock on the site (however we add a write lock for writes to reduce race condition windows - something that does not exist in core now).
- The variable cache is converted to write-through - so there is no need to do a full rebuild each time there is a cache clear.
- Because DrupalCacheArray builds up the cache object over time, the first few requests can issue a dozen or three small database queries instead of waiting for a massive db_query() and unserialization of hundreds or thousands of items before they're able to complete, which should reduce performance spikes on cold starts (with the trade off of some extra writes to the cache as it gets built).