Hello all, it’s time for the weekly migration initiative meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to here: https://www.drupal.org/project/drupal/issues/3178489. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
| damienmckenna | We usually put our inflatable dragon decorations outside, along with a bunch of other things, but this year they're in boxes as we're moving. We also usually do trick 'r treating, but covid. |
| benjifisher | @kevinquillen, please reply in threads. |
| benjifisher | My family has some decorations: we aim for cute rather than scary. |
| benjifisher | We will carve a pumpkin tomorrow or Saturday. |
| megachriz | Youri, from the Netherlands. I do not plan to celebrate Halloween. I think Halloween is not so big in the Netherlands, though I heard that it grew a bit in a popularity the last few years. |
| john.ouellet | John Ouellet from Florida. Looking to get more involved with the migration community. Have a ton of exp with migrations now.As far as Halloween goes, here in FL they usually have a centralized location where kids grab all the candy. |
| Michel Gryp | Hey, I'm Michel, living in Belgium. My girlfriend is a cosplayer so she will drag me into Halloween whether I want it or not. :slightly_smiling_face: But It's only big here in West Flanders where (pre-covid) there are parades and Halloween walks.I am also working on a big migration project to move the content from a D7 platform with over 300 sites over to D8. |
| dinarcon | :wave: |
| quietone | No, I don't. |
| benjifisher | From @kevinquillen :Has anyone ran batched migrations from hook_update_n? I was trying with 15 items per run, and after a few hundred it just seemed to hang. No indication or log as to why. |
| benjifisher | I have never tried that. Can you show us the code? |
| kevinquillen | let me see if ic an dig this up from history |
| kevinquillen | /** * Copy user IDs with grant entries to a backup table. */function mymodule_migration_update_8005(&$sandbox) { $db = \Drupal::database(); $db->query('CREATE TABLE {user__grants_backup} SELECT DISTINCT(entity_id) FROM {user__field_grants}'); $db->query('ALTER TABLE {user__grants_backup} ADD COLUMN migrated tinyint(4) DEFAULT 0 AFTER entity_id');}/** * Run an update migration for users from update 806. */function mymodule_migration_update_8006(&$sandbox) { $batch_size = 15; $db = \Drupal::database(); $messenger = \Drupal::messenger(); if (!isset($sandbox['total'])) { $uids = $db->query('SELECT entity_id FROM {user__grants_backup} WHERE migrated = :status', [':status' => 0])->fetchAll(); $sandbox['total'] = count($uids); $sandbox['current'] = 0; } $records = \Drupal::database() ->select('user__grants_backup', 'u') ->fields('u', ['entity_id']) ->condition('u.migrated', 0) ->range($sandbox['current'], $sandbox['current'] + $batch_size) ->execute(); $endpoint = \Drupal::service('mymodule.endpoint'); $source_ids = []; $urls = []; $migration = \Drupal::service('plugin.manager.migration') ->createInstance('mymodule_users'); // Reset status if necessary. if ($migration->getStatus() != MigrationInterface::STATUS_IDLE) { $migration->setStatus(MigrationInterface::STATUS_IDLE); } foreach ($records as $record) { $account = User::load($record->entity_id); $source_id = $migration->getIdMap()->lookupSourceId(['uid' => $account->id()]); if (empty($source_id)) { throw new \Exception("No source id provided or user id could not be found in migrate map table for {$record->entity_id}."); } $source_ids[] = end($source_id); $urls[] = $endpoint->buildUrl($account, $source_id)->toString(); $db->update('user__grants_backup') ->fields(['migrated' => 1]) ->condition('entity_id', $record->entity_id) ->execute(); $sandbox['current']++; } $source = $migration->getSourceConfiguration(); $source['data_fetcher_plugin'] = 'http'; $source['urls'] = $urls; $source['authentication'] = [ 'plugin' => 'basic', 'username' => \Drupal::service('mymodule.secrets')->getSecret('apiUsername'), 'password' => \Drupal::service('mymodule.secrets')->getSecret('apiPassword'), ]; $migration->set('source', $source); $migration->getIdMap()->prepareUpdate(); $migration->getIdMap()->setUpdate($source_ids); $executable = new MigrateExecutable($migration, new MigrateMessage()); $executable->import(); if ($sandbox['total'] == 0) { $sandbox['#finished'] = 1; } else { $sandbox['#finished'] = ($sandbox['current'] / $sandbox['total']); }} |
| kevinquillen | Basically, data was migrated into a paragraph on a user entity - requirements have changed and now it is a basic entity reference field. Instead of having to run the migration for 30k users again to update it, I tried to copy the values into a new table to know what users had data on that old field, then run an update migration for them |
| kevinquillen | Every couple of hundred passes it would just hang |
| kevinquillen | Wound up removing this and just running it for all, but it seemed like a valid approach. It was 2000 users vs 30,000. |
| benjifisher | Maybe log a message before throwing an exception:if (empty($source_id)) { throw new \Exception("No source id provided or user id could not be found in migrate map table for {$record->entity_id}."); } |
| benjifisher | It is a little hard to follow the logic, with the combination of migration and direct database queries. |
| kevinquillen | Each run just grabs the next 15 user ids from that table, then flags them (so they cant be selected again). The migration should set those 15 to be updated and then execute the migration. It all appeared to work, just hung up after a while. |
| kevinquillen | That same migration code chunk I have as a tab on entities that does the same thing, just for that one item. |
| benjifisher | I still think you might be hitting that line where you throw an exception.To debug, you could run the same code from a Drush script. If there are some user IDs that trigger a problem, it will still fail and you should be able to figure out which IDs are causing the problem. If it completed smoothly, then it might be an OOM error or a timeout, which would be harder to diagnose/fix. |
| benjifisher | 5️⃣ Documentation |
5️⃣ Documentation
| benjifisher | Pending from last week: I think it would be good to expand https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/ch... more-or-less complete YAML for both the User migration and the Node migration that depends on it.Give some example database queries to illustrate how you can troubleshoot.Mention what happens for more complicated migrations: multiple source and/or destination IDs. |
| dinarcon | Adding examples for the DOM plugins would also be useful. Beyond what is documented in the class docblocks I mean. |
| benjifisher | More than what we already have in the CR? Or would it be enough to copy that to the regular documentation? |
| dinarcon | You mean this change record https://www.drupal.org/node/3062058 ? Yes, I would say that that is not enough. There are snippets like https://benjifisher.gitlab.io/slide-decks/html-migrate-dcnj-2020.html#/t... which might raise questions as well. I think many would use these plugins to extract data from the DOM to populate fields. That is currently not possible unless you apply the patch for dom_extract or write a custom process plugin yourself. (edited) |
| dinarcon | And a working example for dom_migration_lookup would be super useful. That could use the embedded_data plugin. |
| benjifisher | This core issue is a blocker: #2852463: Create a trait and base class to implement \Drupal\Component\Plugin\ConfigurableInterface (edited) |
| benjifisher | @mikelutz (he/him) updated the title, but not the Proposed Resolution section. I am not sure how the current patch is different from the one that was committed and reverted. @heddn, can you review this issue? |
| megachriz | #3002362: Foundation: Feeds migrate Processors/Tamper UI is ready for testing. The specific thing to test is the mapping UI + the process plugin UI. And the question is: when you play around with this UI, do you experience any fatal errors? If not, I think it’s ready for commit. |
| megachriz | We do know already that editing existing migrations can lead to fatal errors. That isn’t the focus of that issue though. Focus only on configuring new migrations. |
| benjifisher | Yes, limit the scope. Other known errors can be handled in separate issues.I will add this to my to-do list. |
| mikelutz (he/him) | @benjifisher I don’t believe there was anything to update in the issue summary. The original patch didn’t include a base class. There was a trait, and we tried to do the constructor work in the main plugin base class if we detected the correct interface, but this cause problems in contrib with configurable classes that expected the configurations to be merged at a specific point in the constructor. The new patch adds a separate base class that loads the configuration in the constructor, so that contrib has to opt in to it rather than forcing the configuration loading on everybody. |
| benjifisher | #3080256: Add alter event to alter generated configuration |
| benjifisher | Some comments were added to the thread on last week's meeting after the transcript was posted to the issue, so let me repeat them here (slightly edited). |
| benjifisher | @damienmckenna updated the issue summary. :+1: |
| benjifisher | If that updated summary is accurate, then providing an alter hook or dispatching an event seems like a work-around. If that is the best we can do, then we should do it. But it would be much better if migrate_upgrade could rename the dependencies, since it is responsible for renaming the migrations. Other modules should not have to be aware of migrate_upgrade. |
| damienmckenna | I haven't had the time to follow up on the feedback. |
| benjifisher | @quietone suggested --prefix=''. I have used that on migration projects, where I use the migrations generated by migrate_upgrade as a starting point for my custom migrations. For that purpose, it works fine. I have not tried that in this context, where the plan should be to go ahead and run the migrations as generated. It certainly seems worth a try. If it works, then we can add that as a work-around. |
| damienmckenna | I wonder if part of the problem is related to the order of operations? Migrate Upgrade has logic to change dependencies, but it doesn't seem to always catch everything. I wonder if its logic is running too soon and other module hooks run later, leading to this problem? |
| benjifisher | That sounds plausible. If so, then the solution might be to replace hooks with event subscribers, so that we have more control over order of execution. Or it might be simpler: reorder the code so that dependencies are updated after the hooks are invoked. |
| quietone | Yes, it is about order of operations. All the hooks are run and then Migrate Upgrade creates the configured migration. It is that migration you want to run. However, when you you run `drush ms` or drush mim the migrations_plugin_alter_hook runs again and the configured migrations are altered for a second time before execution.I suggest in #3157708: Missing dependencies when using Migrate Upgrade#comment-13881140 that the commerce_migrate hook_alter just does not alter any configured migration. They have identifiable keys in the array of migrations. Is there any reason one would want to alter a configured migration? |
| benjifisher | @Nick Dickinson Wilde, did you ever find out whether this error was related to #3123950: 'Field discovery failed for Drupal core version 7' errors if migration source database key is not 'migrate'? |
| Michel Gryp | From my own experiences the second issue bothered me, from once I enabled the migrate_drupal module some automatically recognised files in /migrations directories (for example in config_translation) demanded a 'migrate' database to exist. (edited) |
| john.ouellet | I concur, I inherited a migration project that used upgrade as the key and it caused all sort of funky issues like this one. |
| benjifisher | I think I have run into the same problem. I never investigated it, nor tried to fix it. I just added a 'migrate' key so that I could get past the errors. |
| Michel Gryp | I skipped the 'Drupal 6' and 'Drupal 7' tags in an alter hook since I didn't require the default migrations, but I guess some people do want them |
| benjifisher | BTW, thanks for joining the discussion. Would you like to introduce yourself under 0️⃣? |
| Michel Gryp | sure :slightly_smiling_face: |
| benjifisher | Right, that is an effective solution. I think that @heddn mentions that on an issue somewhere, but I am not sure it is documented anywhere. Those tags are added by migrate_upgrade, right? |
| Michel Gryp | They seems to be in the individual migrate files in all core modules |
| Michel Gryp | So an option to exclude core migrations would be cool |
| Michel Gryp | based on those tags |
| Michel Gryp | Or some sort of filter on the MigrationPluginManager where we can skip tags, just brainstorming in here |
| john.ouellet | That would be nice |
| quietone | Using MigrationPluginManagerInterface::createInstancesByTag should help with that. |
damienmckenna, benjifisher, megachriz, john.ouellet, Michel Gryp, dinarcon, quietone, kevinquillen, mikelutz (he/him)