Hello all, it’s time for the weekly migration subsystem meeting. The meeting will take place in slack in various threads
This meeting:
➤ Is for core migrate maintainers and developers and anybody else in the community with an interest in migrations
➤ Usually happens every Thursday and alternates between 1400 and 2100 UTC.
➤ Is done on the #migration channel in Drupal Slack (see www.drupal.org/slack for information).
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to. See the parent issue for an idea of the typical agenda.
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
2021-07-01 - The hope is that most or all of the maintainers will attend. We will try to focus on longer-term goals than in the weekly meeting.
| benjifisher | What is needed most, a summary page in the docs for Migrate API or improvements to the project pages? |
| benjifisher | Do we already have links to contrib module in the Migrate API docs? |
| dinarcon | Some official documentation would be nice. I guess that would fall under https://www.drupal.org/docs/contributed-modules |
| dinarcon | And there is this page listing some migration related modules from the upgrade guide https://www.drupal.org/docs/upgrading-drupal/drupal-8-migrate-modules |
| dinarcon | Documentation in a README or a blog post linked from the project page would also be useful. |
| dinarcon | I am bringing this up because recently I look at Media Migration module. It looks like it does a lot of what I need, but there is not much documentation. But others modules would benefit from improved documentations as well. |
| benjifisher | If there is already a page in the official docs, then anyone can edit it. If you want to add a new page, then you need one of the guide maintainers (like me) to add it to the navigation menu. |
| dinarcon | Something that would be nice is if modules provided example repos where one could see how they are intended to be configured and used. |
| benjifisher | Contrib module READMEs and project pages are different. |
| benjifisher | Example repos is a lot to ask for. Unfortunately, the expectations are pretty low for documentation of contrib modules. |
| benjifisher | We could use a community initiative to work on that. Maybe even give a few people permission to update project pages and README files for other people's projects. Probably that is too much to ask for. |
| dinarcon | One can only dream :slightly_smiling_face: Examples might not be feasible, but I would still like to find a way to improve documentation. |
| benjifisher | @chx (he/him), I see that you already posted about this in the #contribute channel, but I have not read that thread yet. Can you explain again here (and save meeting members the trouble of following a link)? |
| chx (he/him) | sure thing |
| chx (he/him) | so I have been migrating a real lot of fields and writing out the plugin: sub_process - source - process - plugin - source again and again gets real old real fast |
| chx (he/him) | so I was thinking, you know, we could just use "field_something/*/tid" to abbreviate where * notes we want to iterate over it (note the sub_process at one point was called iterator) and then tid means the field property we are using |
| chx (he/him) | similar notation exists for entity query with field_something.*.foo |
| quietone | There is an issue in migrate tools or migrate plus about reusing pipeline snippets. Haven't found it yet. Still looking. |
| chx (he/him) | it's also quite BC friendly because I sincerely doubt anyone ever had a valid source property containing an asterisk |
| chx (he/him) | more, when migrating fields like these, the reason you are not copying them is very likely because they are reference fields which were migrated themselves |
| chx (he/him) | so you have a migration_lookup in the sub process |
| chx (he/him) | so that's how I arrived to field_topics/*/tid using d7_taxonomy_term:field_topic and once again that string is looking even more BC friendly |
| chx (he/him) | the code required to parse this and translate back to normal notation is minimal |
| chx (he/him) | I have little doubt it is useful I just do not know whether it's useful enough for core |
| chx (he/him) | currently I just use my own migration class which extends the core migration class, very easy |
| chx (he/him) | I could use the alter hook as well but this is easier |
| benjifisher | I found the "pipeline" issue: #3123534: Create a "pipeline" process plugin to re-use YAML config |
| benjifisher | I do not think any of the proposed "pipeline" implementations will handle this. |
| chx (he/him) | that's a different issue, I think |
| chx (he/him) | mine is about reusing a template |
| chx (he/him) | I can write a templating system too if you want -- I always was an ardent follower of "no kill like overkill" :slightly_smiling_face: |
| benjifisher | Could it be implemented as a process plugin with configuration instead of introducing new syntax? |
| chx (he/him) | yes but the savings would be much less |
| chx (he/him) | also, while the code to normalize this is easy, the code to do this from a process plugin is slightly less easy |
| chx (he/him) | you need the process plugin manager, instantiate the sub process plugin with the right configuration |
| chx (he/him) | slightly less easy but surely doable |
| chx (he/him) | but again then you are writing three lines per field instead of one :slightly_smiling_face: still much less than the full blown field reference migration |
| benjifisher | If it is not introcuding new functionality, just simpler syntax, then I do not think there is a good case for it in core. Every new feature means more maintenance, more work for the poor testbot. |
| benjifisher | The sub_process plugin is not used all that much in core:
$ ag -ric sub_process core/modules/*/migrations
core/modules/field/migrations/d7_field_instance.yml:1
core/modules/file/migrations/d6_upload.yml:1
core/modules/filter/migrations/d6_filter_format.yml:1
core/modules/filter/migrations/d7_filter_format.yml:1
core/modules/image/migrations/d7_image_styles.yml:1
core/modules/rdf/migrations/d7_rdf_mapping.yml:1
|
| benjifisher | @dinarcon, how do you like that string of options for ag? :wink: |
| chx (he/him) | that's truly surprising |
| chx (he/him) | how are we migrating term reference fields without them? |
| dinarcon | TIL there is an ag command :sweat_smile: Nice options. |
| quietone | A point in favor of the new syntax is that may slow down the constant flow of questions about what to put in the pipeline for a field. |
| chx (he/him) | or does core still suppose primary ids do not change? |
| benjifisher | Yes, most of the core migrations preserve entity IDs. |
| quietone | The term reference, and most reference fields, get their pipeline dynamically in a MigrateField plugin, https://api.drupal.org/api/drupal/core%21modules%21taxonomy%21src%21Plug...…]21TaxonomyTermReference.php/class/TaxonomyTermReference/9.1.x |
| Matroskeen | If this syntax was available, I would use it for sure. I always find the sub_process plugin syntax a bit hard to write. Really looking forward to seeing it somewhere (core or contrib) (edited) |
| quietone | @mikelutz (he/him), what are your thoughts on this? |
| benjifisher | If I understand correctly, then a better title for this thread would be "new notation for sub_process". For example,
field_topics/*/tid using d7_taxonomy_term:field_topic
would be equivalent to something like
field_topics:
plugin: sub_process
source: <not sure>
process:
tid:
plugin: migration_lookup
migration: d7_taxonomy_term:field_topic
source: tid
That is certainly cumbersome.
|
| benjifisher | ... but if we make it too easy, then no one will need to hire "experts" like us! :wink: |
| chx (he/him) | lol |
| chx (he/him) | ` source: ` is source: field_topic |
| chx (he/him) | sorry it's a bit confusing example with the field and vocab being called the same |
| chx (he/him) | Untitled
field_topics:
plugin: sub_process
source: field_topic
process:
tid:
plugin: migration_lookup
migration: d7_taxonomy_term:topic
source: tid
vs
field_topic/*/tid using d7_taxonomy_term:topic
|
| mikelutz (he/him) | I'm not a huge fan of it in core. It doesn't add any new functionality to what migrations can do, and every layer of shorthand we code in is another layer of complexity to maintain. I think it would better live in a contrib migration_plugins_alter hook. |
| chx (he/him) | I can most certainly do that |
| chx (he/him) | I will release a small module doing that in a few weeks |
| alison | TIL there is an ag command :sweat_smile: Nice options.Me too, cool! (@benjifisher Is it correct that the main advantage over grep is that it's faster, and it excludes files/paths in gitignore and a custom .ignore file, or more than that? (If more than that, I can ask outside this thread, I don't mean to hijack.)Speaking of the thread itself: interesting convo, glad I came in to catch up - I look forward to seeing more later. (edited) |
| marvil07 | @chx (he/him) Nice to see you around migrations! :confetti_ball: |
| marvil07
| I was confused at first with the reference to the re-usable pipelines.But now that I think about it, maybe a way to do pre-processing of the whole migrate plugin may be the connection.i.e. to translate asterisk into something else using sub_process plugin, would be similar to preprocess the migration to do replace some tokens with actual process pipelines. |
| marvil07 | :+1: on trying to do it on contrib first. |
| chx (he/him) | Do or do not. There is no try :stuck_out_tongue: |
| benjifisher | Why not hijack the thread?Yes, ag (the "silver searcher" for comic book fans) is an alternative to grep. I think @alison listed the main advantages. Also, it takes 1/2 as long to type as "grep".I think it also skips composer-managed files. So it will search Drupal core if you are in a clone of Drupal core, but not if you are working on a site.A big part of its speed is all the stuff it knows to skip. |
| alison | Thank you @benjifisher! |
dinarcon, Matroskeen, benjifisher, chx (he/him), quietone, mikelutz (he/him), gaurav mahlawat, alison, Neslee, marvil07
