Core migration issues
Next video meeting 2025-01-02 (tentative)
Hello all, it’s time for the biweekly migration subsystem meeting. The meeting will take place in slack in various threads. This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every second Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. #3488926: [meeting] Migrate Meeting 2024-11-21 1400Z See the parent issue for an idea of the typical agenda.
➤Transcript will be exported and posted to the agenda issue. For anonymous comments, start with a 👤 emoji. To take a comment or thread off the record, start with a 🚫 emoji.
0️⃣ Who is here today?
| anmolgoyal74 | Hi :wave: |
| mikelutz (he/him) | Hi |
| dinarcon | :wave: |
| Dan Davis | Hello! |
| godotislate | :wave: |
| benjifisher | I posted in the channel, but maybe I should have said it here:
I have to multi-task for the next 1/2 hour. :disappointed:
|
1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today's meeting.
| dinarcon | I'd like to revisit some discussions related to performance from early in the year. One particular issue is[#3463800] but a general thread on performance would be nice (edited) |
| dinarcon | And somewhat related to performance... what are some tips and tricks for profiling slow/long running migrations? Maybe this can be included in the performance thread? (edited) |
| dinarcon | https://www.drupal.org/project/drupal/issues/2630732 is marked as needs review. @larowlan already reviewed it last time it was set to RTBC and asked to close some threads in the MR. Some of the open threads are about suggestions in code style. Others seem to have an impact on how the new feature would work. Shall we talk about how to move the issue forward? Maybe continue working on the MR suffices. |
| benjifisher | I think that AUTO_INCREMENT Alter is appropriate for the 6️⃣ Announcements thread. (edited) |
2️⃣ Action items. To be added later.
3️⃣ Statistics
4️⃣ Comment in this thread if you are looking for ways to contribute. Give us some idea of what you would like to do: documentation, code review, testing, project management, ...
5️⃣ Previous minutes.
6️⃣ Announcements
7️⃣ Performance
| Migrate Initiative Meeting | what are some tips and tricks for profiling slow/long running migrations? |
| Migrate Initiative Meeting | (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1732199709426659?thread_ts=... (edited) |
| dinarcon | I have been trying to get batch size and highwater to work without much success. Tested the patch in this issue, but ended up with an infinite loop. In another thread, @mikelutz recommends having custom source plugins. Wondering if he, or someone else, could expand on this. |
| dinarcon | In a recent project, I found that \Drush\Drupal\Migrate\MigrateExecutable::handleMissingSourceRows takes a significant amount of time to finish processing when there is a large number of records (in the 100 thousands range). In particular, this check was taking a lot of time to process...
if (!in_array($mapSourceId, $this->allSourceIdValues))
First time I notice this. Still wrapping my head around this. Wondering if someone else has noticed this to be a bottleneck in slow/long running migrations. |
| dinarcon | The above is in fact part of \Drush\Drupal\Migrate\MigrateExecutable. Not sure if it is wise to overwrite that at all. Although I think that migrate_tools does that to some extent. |
| dinarcon | In case someone is experiencing something similar... I had not realized that \Drush\Drupal\Migrate\MigrateExecutable::handleMissingSourceRows is in fact part of Drush runner, not Drupal core code. We tested the migration runner from migrate_tools module and this issue with migrations halting at the end of the import was gone. So, switching runners helped in our case. Still trying to better understand what \Drush\Drupal\Migrate\MigrateExecutable::handleMissingSourceRows does and how it can be improved. |
| benjifisher | This is the thread from two weeks ago. |
| benjifisher | Do you want to start a new thread for today's meeting? |
| dinarcon | Yes, I should have posted earlier :sweat_smile: not sure if a new thread is necessary, but open to it. |
8️⃣ Implement Entity::fields() for migration destinations
| Migrate Initiative Meeting | This issue is back to NR after at least two "rerolls". |
| Migrate Initiative Meeting | (Original Request) https://drupal.slack.com/archives/C226VLXBP/p1732200323887989?thread_ts=... (edited) |
| benjifisher | In my last review (Comment #166) I noticed (while closing threads) that one of the requested changes was either never made (despite comments that it had been made) or got lost in one of the rebases. |
| benjifisher | So I need to find time to do a complete review. It is on my to-do list. |
9️⃣ Wrap-Up
| Migrate Initiative Meeting | Thanks for coming all! See you in 2 weeks |
| benjifisher | Please consider the meeting open for at least the next 23 hours. |
| dinarcon | Thanks for facilitating Benji. |