Problem/Motivation
Drupal\migrate\Plugin\migrate\id_map\Sql::ensureTables() creates a secondary index that includes all id fields for a migration. If too many fields are included (e.g. more than three VARCHAR(255) columns), the data size may exceed the maximum allowed in an index and result in an exception.
Uncaught PHP Exception Drupal\\Core\\Database\\DatabaseExceptionWrapper: "SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes: CREATE TABLE {migrate_map_migrationname} (\n`source_ids_hash` VARCHAR(64) NOT NULL COMMENT 'Hash of source ids. Used as primary key', \n`sourceid1` VARCHAR(255) NOT NULL, \n`sourceid2` VARCHAR(255) NOT NULL, \n`sourceid3` VARCHAR(255) NOT NULL, \n`sourceid4` VARCHAR(255) NOT NULL, \n`sourceid5` VARCHAR(255) NOT NULL, \n`destid1` INT unsigned NULL DEFAULT NULL, \n`source_row_status` TINYINT unsigned NOT NULL DEFAULT 0 COMMENT 'Indicates current status of the source row', \n`rollback_action` TINYINT unsigned NOT NULL DEFAULT 0 COMMENT 'Flag indicating what to do for this item on rollback', \n`last_imported` INT unsigned NOT NULL DEFAULT 0 COMMENT 'UNIX timestamp of the last time this row was imported', \n`hash` VARCHAR(64) NULL DEFAULT NULL COMMENT 'Hash of source row data, for detecting changes', \nPRIMARY KEY (`source_ids_hash`), \nINDEX `source` (`sourceid1`(191), `sourceid2`(191), `sourceid3`(191), `sourceid4`(191), `sourceid5`(191))\n) ENGINE = InnoDB DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci COMMENT 'Mappings from source identifier value(s) to destination\xe2\x80\xa6'; Array\n(\n)\n" at /docroot/core/lib/Drupal/Core/Database/Connection.php line 686
I encountered this error using a Migrate Plus entity - whenever the entity was loaded it would attempt to re-create the table and throw an exception, making some site paths inaccessible. Resolution required deleting the configuration entity directly in the config db table.
Proposed resolutions
Three options were proposed.
- Catch the exception, log an error, and prevent other migration actions from executing if table could not be created.
- Limit the number of fields included in the secondary index so that the maximum index size is not exceeded.
- Create multiple secondary indexes, split to fit within the the index size limits (e.g. ['source1' => ['sourceid1','sourceid2','sourceid3'], 'source2' => ['sourceid4','sourceid5']])
Of those, the final option, creating multiple indexes is implemented. A while loop is used to continually increase the number of indexes until the map table is created.