Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
Core migration issues
Next video meeting 2022-07-07 2100Z (tentative)
The hope is that most or all of the maintainers will attend. We will try to focus on longer-term goals than in the weekly meeting.
0️⃣ Who is here today?
| benjifisher | Hi! I have been putting out :fire: at work this week. I would like to have more time to review migration issues and enjoy the weather. :disappointed: |
| alison | Here at least as a lurker :meow-wave-peek: fortunate to get to work in my back yard, though the screen is a bit hard to see :zoom_eyes: (edited) |
| quietone | Hi (edited) |
| Nick Dickinson-Wilde | Actuallllly online and not swamped during a meeting for a change! WOOO. Hey y'all |
| dinarcon | Hello :wave: Working on interesting migrations lately. |
| damienmckenna | Heyo. |
| dww | Derek @ TEN7, near Honokaa, HI (US colony) :wave: |
1️⃣ What should we talk about today? Suggest topics here and I will add threads. I will also check for comments on the issue for today’s meeting.
| quietone | Migrate plan for D10, including rest of D9. #3004480: Migrate in D9 (edited) |
| benjifisher | No issue, but https://drupal.slack.com/archives/C226VLXBP/p1652239184771149 |
| benjifisher | Short version: is it a good idea to migrate node revisions in reverse chronological order if you are preserving entity/revision IDs? |
| mikelutz (he/him) | No, not really, but Derek seemed convinced he had solved the problems. |
| mikelutz (he/him) | Thankfully dealing with postgres has forced us to NOT count on sql queries coming out in revision order when not specifying a sort, so I think most of those bugs are gone. |
| benjifisher | Can we have a thread to discuss pros and cons?Derek was asking for advice. |
| Nick Dickinson-Wilde | Does #3279517: Migration source database errors are not very useful need tests? (A ticket/9.5.x MR I created this week - will add D10 MR soon) |
2️⃣ Action items. To be added later.
3️⃣ Statistics
| mikelutz (he/him) | Fixed since last meeting, 0 |
| mikelutz (he/him) | RTBC: 13 |
| mikelutz (he/him) | Needs Review: 20, 4 older than a month |
| benjifisher | It has been a while since any migration issues have been fixed. We are not in a commit freeze, are we? |
| mikelutz (he/him) | We were with the 9.4 alpha |
| mikelutz (he/him) | Focus for the next while will be on 9.4 fixes, so we probably won’t see anything other than bugfixes prioritized unless we ask. |
4️⃣ Comment in this thread if you are looking for ways to help. Give us some idea of what you would like to do: documentation, code review, testing, project management,
5️⃣ Previous minutes.
7️⃣ Migrate plan for D10
| quietone | Can we update #3004480: Migrate in D9 with the upcoming priorities. |
| quietone | And make a list of issues |
| quietone | Or should we make a new issue? |
| benjifisher | I think a new issue is the way to go. True, we did not accomplish our "sprint goals" for Drupal 9, but I think it is best to admit that and keep the history easier to follow. |
| quietone | New issue made, #3280243: [Meta] Migrate in D10 |
| quietone | I choose to start with 'high priority' but we could use the [must-have] style that is used in #3118154: [meta] Deprecate dependencies, libraries, modules, and themes that will be removed from Drupal 10 core by 9.4.0-beta1 |
| quietone | What are your priorities? |
| quietone | What is the plan for migration_lookup? |
| mikelutz (he/him) | It needs to die a fiery death. |
| quietone | OK. What's the plan to do that and what happens to the existing issues on the current version? |
| benjifisher | I do not think a subsystem can specify must-have issues. Just priorities.At least, I am not going to try to tell xjm that we cannot release D11 until one of our must-haves is complete. :wink: (edited) |
| mikelutz (he/him) | Not sure. I can tell you waiting and hoping someone will do it hasn't worked. |
| mikelutz (he/him) | Writing the new lookup is nothing, the work is in adjusting all the core migrations to not use the old one so we can deprecate it. |
| benjifisher | Do we have an issue for replacing migration_lookup with something else? |
| mikelutz (he/him) | I think there is. I recall someone gathering a list of the things we wanted to fix and putting them in an issue. |
| mikelutz (he/him) | Don't have it off the top of my head and walked away from the computer already. |
| quietone | #3246666: Deprecate migrate_lookup and replace with cleaner process plugin |
| mikelutz (he/him) | Hey, I know that guy.. |
| mikelutz (he/him) | Looks like the type of issue he would make and then not touch again. |
| benjifisher | Tag! You're It.Maybe that is the wrong game. Does anyone actually play with hot potatoes? |
8️⃣ Changed timestamps and entity syncing
| mikelutz (he/him) | Seen this pop up in the past few days |
| mikelutz (he/him) | #2329253: Allow the ChangedItem to skip updating the entity's "changed" timestamp when synchronizing (f.e. when migrating) |
| mikelutz (he/him) | #3052115: Mark an entity as 'syncing' during a migration update |
| mikelutz (he/him) | #3280140: Running migrate:import for node entity with --update modifies CHANGED timestamp |
| mikelutz (he/him) | Longstanding known limitation of the entity api, but looks like there is potential to fix it with the new flag. |
| quietone | Are the first two duplicates? |
| mikelutz (he/him) | Not how I read them. |
| mikelutz (he/him) | One allows the syncing flag to not update timestamps, the other has the entity destinations use the sync flag. They would both be necessary, but I would call 3052115 PP on 2329253 |
| rclemings | I just changed (truncated) the title on 3052115 to "Mark an entity as 'syncing' during a migration update." I don't know if that clarifies it or not but the issue summary, which I updated yesterday, may help. |
| damienmckenna | One person said their migration worked find with the two patches while using "d7_node". OTOH it doesn't seem to work with "d7_node_complete" which I'm using on one site. (edited) |
| mikelutz (he/him) | Probably need to finish 2329253 and then write tests for both in 3052115 |
| rclemings | @damienmckenna I noticed that d7_node_complete uses "timestamp" instead of "changed" as the source for the "changed" field in the process section. I don't understand the implications of that but was wondering if it would work if changed. |
| damienmckenna | Interesting. |
| mikelutz (he/him) | No. changed and created are the fields in the node table, but d7_node_complete doesn’t use the node table. it uses the revision table. |
| mikelutz (he/him) | revisions don’t have created or changed, they just have one timestamp. The aggregation of the timestamp of the first and last revision are the created/changed timestamps for the final node. |
| rclemings | Ah, OK then. I guess that's not the problem. |
| dww | Thanks for this thread. Interesting set of issues gathered here! Glad to have these on the radar... |
| benjifisher | +1: It would be nice to be able to set the changed timestamp directly. (edited) |
| rclemings | I'm lost in the weeds here but it looks as if #3052115: Mark an entity as 'syncing' during a migration update#comment-14296910 may be relevant to @damienmckenna's issue. Or at least it mentions d7_node_complete.yml. |
9️⃣ Migrating revisions in reverse order
| mikelutz (he/him) | Thread for @benjifisher and @dww |
| benjifisher | The problem: a large D7 -> D9 migration needs to preserve revisions, but the revision migration takes about a week to complete. Site owners are impatient: they want to start editing on the D9 site ASAP. |
| dww | And they want to have the D7 history to work with, but they mostly only care about the last month's history. |
| dww | Instead of waiting for the whole week+ migration to finish before it gets to the records they care about, I proposed we try to reorder the migration so it starts at the highest vid and works down, instead of starting at vid 1 and working up. |
| benjifisher | Proposal:
- Migrate nodes (current revisions) first. Preserve node IDs. (Maybe not revision IDs?)
- Let editors get to work.
- Use a custom source plugin that provides rows in reverse order by vid.
- Run the revision migration while the site is being edited.
|
| dww | I suspect "highwater" support would be confused, but otherwise, anyone foresee any trouble with this approach? |
| benjifisher | Caveats:
- No forward revisions (possible with content moderation, other)
- Preserve
nid. - Either preserve
vid or else use the diff module with a patch: it overrides the revision listing with a sort by timestamp, not vid.
|
| dww | #3276716: Sort revisions first by timestamp, then revision ID for interested parties. :grin: |
| dww | :nerd_face: the patch adds a primary sort on timestamp, but leaves vid as the 2nd sort to break ties (e.g. in tests where you have multiple revisions in the same second). |
🔟 Migration source database errors are not very useful
| mikelutz (he/him) | #3279517: Migration source database errors are not very useful |
| Nick Dickinson-Wilde | tests... do we need tests that show those new error messages are shown? |
| benjifisher | That is a pretty small patch. |
| Nick Dickinson-Wilde | yep |
| benjifisher | If I get one of those errors, do I care whether it was a PDOException or a DatabaseAccessDeniedException? I guess the error message will tell me which line threw the exception, but I prefer to have distinguishable error messages. |
| Nick Dickinson-Wilde | It does give the full error message ($e->getMessage()), do you think that is enough? |
| benjifisher | My default answer is yes, it needs a test. And the test should be able to distinguish between the two types of failure: one reason to use distiguishable messages. |
| Nick Dickinson-Wilde | okay, marking as needs work |
| Nick Dickinson-Wilde | thanks |
| benjifisher | The issue summary shows
SQLSTATE[HY000] [1045] Access denied for user 'USER'@'IPADDRESS' (using password: YES)
I assume that is a DatabaseAccessDeniedException. Can we rely on that format? Might it be translated? |
| Nick Dickinson-Wilde | I would strongly expect the message could be translated but the error code (1045) should be the same accross languages |
| Nick Dickinson-Wilde | (also USER and IPADDRESS in that sample is replaced with real values in real cases) |
| benjifisher | So the error code is good enough for the test to distinguish between the two error codes and we trust that the (possibly translated) error message is enough for a human? |
| Nick Dickinson-Wilde | That sounds reasonable to me |
| benjifisher | What version of PHP allows
catch (A|B $e) { ... } |
| Nick Dickinson-Wilde | ummm can't remember if that is 8.1 or before... |
| benjifisher | As of PHP 7.1.0, a catch block may specify multiple exceptions using the pipe (|) character. This is useful for when different exceptions from different class hierarchies are handled the same. |
| benjifisher | From https://www.php.net/manual/en/language.exceptions.php |
| Nick Dickinson-Wilde | so easily relied upon even for 9.5 |
| benjifisher | There was another change in PHP 8:
As of PHP 8.0.0, the variable name for a caught exception is optional. If not specified, the catch block will still execute but will not have access to the thrown object.(edited)
|
1️⃣1️⃣ Wrap up! You don’t have to go home, but you can’t stay here!
| mikelutz (he/him) | Or you can totally stay here. Threads will remain open and be posted to the issue queue in about 24 hours. |
| quietone | @mikelutz (he/him) Thanks for facilitating |
| benjifisher | +1 |