Updated: Comment #6
Problem/Motivation
SQLite has a variable limit of 999 which is not configurable and which makes this database driver unusable for medium sized sites.
See #2028713: Menu router does "too many SQL variables" on SQLite resulting in silent nondisplay of menu items but this can bite us in any other place too.
A first patch that set PDO::ATTR_EMULATE_PREPARES on the connection (proposed by Damien Tournoud) did not work out probably due to a bug in the underlying driver. This can be reproduced as outlined in #2028713-18: Menu router does "too many SQL variables" on SQLite resulting in silent nondisplay of menu items:
php > $options = array(
php ( PDO::ATTR_EMULATE_PREPARES => true,
php ( PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
php ( );
php > $dbh=new PDO('sqlite:dummy.sqlite', NULL, NULL, $options);
php > $dbh->setAttribute(PDO::ATTR_EMULATE_PREPARES, true); // Give it another try.
php > $values = range(1,1111);
php > $placeholders = implode(',', array_fill_keys(array_keys($values), '?'));
php > $query = "select v from (select 0 v) where v not in ($placeholders);";
php > $stmt = $dbh->prepare($query);
PHP Warning: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 too many SQL variables' in php shell code:1
Proposed resolution
PDO::ATTR_EMULATE_PREPARES tells the driver to not use prepared statements and instead fill in all placeholder values themself.
As the driver seems not to support this, we do it ourself in userland. The driver already contains code that does this for numeric values (to workaround some other problem) and we can simply extend that code.
Related Issues
Original report by @axel.rutz
SQLite performance Optimization: PDO::ATTR_EMULATE_PREPARES
Like we've done on Mysql and Postgres we should also do on Sqlite.
See #827554: PostgreSQL performance Optimization: PDO::ATTR_EMULATE_PREPARES.
(As suggested by Damien Tournoud in #2028713-11: Menu router does "too many SQL variables" on SQLite resulting in silent nondisplay of menu items)
Tests for this issue
The patch from comment #54 should be used to test if any fix will solve the problem from this issue.
Workaround
If you are experiencing this problem there is a workaround. You can recompile SQLite with an increased value for SQLITE_MAX_VARIABLE_NUMBER. The default setting for this value is 999. Versions of SQLite that are compiled for ac OSX have the default value set to 500000 for SQLITE_MAX_VARIABLE_NUMBER. Ubuntu has compiled SQLite with a default value of 250000 for SQLITE_MAX_VARIABLE_NUMBER.