Problem/Motivation
The original problem was that SQLite has a variable limit of 999 which is not configurable and which makes this database driver unusable for medium sized sites. The solution that we came up with is combining multiple inserts and updates in a single query, which makes SQLite a lot faster. The newer minimum required version of SQLite has a much higher variable limit and therfor it is no longer a problem in combination with Drupal.
See #2028713: Menu router does "too many SQL variables" on SQLite resulting in silent nondisplay of menu items but this can bite us in any other place too.
A first patch that set PDO::ATTR_EMULATE_PREPARES on the connection (proposed by Damien Tournoud) did not work out probably due to a bug in the underlying driver. This can be reproduced as outlined in #2028713-18: Menu router does "too many SQL variables" on SQLite resulting in silent nondisplay of menu items:
php > $options = array(
php ( PDO::ATTR_EMULATE_PREPARES => true,
php ( PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
php ( );
php > $dbh=new PDO('sqlite:dummy.sqlite', NULL, NULL, $options);
php > $dbh->setAttribute(PDO::ATTR_EMULATE_PREPARES, true); // Give it another try.
php > $values = range(1,1111);
php > $placeholders = implode(',', array_fill_keys(array_keys($values), '?'));
php > $query = "select v from (select 0 v) where v not in ($placeholders);";
php > $stmt = $dbh->prepare($query);
PHP Warning: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables' in php shell code:1
Proposed resolution
PDO::ATTR_EMULATE_PREPARES tells the driver to not use prepared statements and instead fill in all placeholder values themself.
As the driver seems not to support this, we do it ourself in userland. The driver already contains code that does this for numeric values (to workaround some other problem) and we can simply extend that code.
Related Issues
Original report by @axel.rutz
Combining multiple inserts and updates in a single query
Tests for this issue
The patch from comment #54 should be used to test if any fix will solve the problem from this issue.
