Problem/Motivation

The core migrate system needs a way to import things from CSV files.

Immediate use case is for the Umami Demo profile now in core in 8.6.x. The imported content is in CSV files and we have code to read it and create the relevant entries. Having migrate_source_csv in core would allow us to remove that code and use migrate instead.

See #2809635-37: Create experimental installation profile for where @larowlan suggested bring migrate_source_csv into core.

Other good reasons to have CSV as a migration source in core:

• "I use this constantly. I've seen it forked into the Contenta distro, and a few of my own projects (I just want the source plugin, not another module.) It'll also be vital to Commerce integrations as CSV is a common way to import data from an ERP." @mglaman (#4)
• "CSV format is the first format used by non technical professionnals who need to import data and to generate/edit those datas thanks to software they knew (MS Excel without mention it). It's also the easiest way to export data from many ERP and big solutions with datas." @GoZ (#5)
• "CSV is simply a really common used format and it would make migrate more usable by default for many usecases. One could say: Oh it is easy to parse CSV, but it turns out it is not. On top of that the module also doesn't load the entire CSV file upfront, but rather reads line by line. I think this makes it a good candidate also for bigger migrations out there." @dawehner (#14)
• "This would help the Out of the Box Initiative. Let's do it! Consider this my maintainer +1 :)" @phenaproxima (#18)

Proposed resolution

The migrate_source_csv module has a stable release (2.0), has been downloaded almost 100k times, and comes complete with tests. The contenta CMS has been using this for their migrations, too.

Rename the files/namespaces, and move it in, as-is, to the core migrate module.

Since this is a straight port from contrib, discussions about fetchers, etc. are postponed to follow-ups. See #16, #36 and #37 for some of the highlights of the discussion.

Follow-up: #2962091: Adopt fetchers/parsers logic for use by source csv plugin

Remaining tasks

1. Patch that renames files/namespaces and moves the contrib code into core Done.
2. Tests Done.
3. Word-smithing PHPDocs Done.
4. Decide if we're okay with a straight port (done), or if we need to hash out all the fetcher/parser stuff first, figure out the design, class names, avoid conflicts, possible BC hell, etc. Done. See comments #80 through #83. Punted to #2962091: Adopt fetchers/parsers logic for use by source csv plugin.

None.

User interface changes

None.

API changes

Adds a new migrate source plugin called "csv" in migration yml definition files. The class providing the plugin (\Drupal\migrate\Plugin\migrate\source\CSV) is heavily commented with PHPDoc.

Adds a \Drupal\migrate\CSVFileInterface interface that classes used for the optional "file_class" configuration key must implement.

Adds a \Drupal\migrate\CSVFileObject class (which extends \SplFileObject) that is the default class used if "file_class" is not overridden in your migration.

Existing users of this plugin from contrib are unharmed. The name is the same, but we haven't changed anything about how it works. So, contrib users simply disable migrate_source_csv module before upgrading to core 8.X.Y and none of their migration files have to be changed to continue working.

Data model changes

None.

Original report by @smaz

In #2809635: Create experimental installation profile, we are using CSV files to provide default content for a demo installation profile of Drupal.

In a review, @larowlan suggested bringing the CSV migration sources into core:
https://www.drupal.org/project/drupal/issues/2809635#comment-12381619

I agree that a CSV migrate source would be a very useful feature for core.

The migrate_source_csv module has a stable release (2.0), has been downloaded almost 100k times (not sure if that includes composer stats?), and comes complete with tests. The contenta CMS has been using this for their migrations, too.

So would it be ok to look at moving this into Core's migrate module? If so, I'm happy to try and turn the module into a patch.

Problem/Motivation

Add a class loader like this:

spl_autoload_register( function ($class_name) {
  file_put_contents(__DIR__ . '/classes.txt', $class_name . "\n", FILE_APPEND);
  },
  FALSE, TRUE);

It generates a classes.txt file.

Look at that file like this:

sort classes.txt | uniq -c

You'll see stuff like this:

   2 Drupal\Core\Annotation\PluginID
   3 Drupal\Core\Annotation\Translation
   5 Drupal\Core\Annotation\code
   3 Drupal\Core\Annotation\end
   3 Drupal\Core\Annotation\end_code
   5 Drupal\Core\Annotation\endcode
  25 Drupal\Core\Annotation\ingroup
   1 Drupal\Core\Annotation\property
  11 Drupal\Core\Annotation\see
   2 Drupal\Core\Annotation\todo

This means that the classloader is being asked to autoload, for instance, Drupal\Core\Annotation\ingroup 25 times in a front page load.

That's because the annotation system isn't filtering out known-bad annotations.

At absolute best this means that the autoloader has to waste time dealing with this stuff, and at worst has to do a file_exists() for each of these requests.

Proposed resolution

Have the annotation system filter out known annotations.

Remaining tasks

User interface changes

API changes

Data model changes

The Migration Lookup plugin relies on deprecated method MigrateIdMapInterface::lookupDestinationId(), which passes control over to MigrateIdMapInterface::lookupDestinationIds(), then filters out all but the first value returned.

This causes multiple-valued migration lookups to always match only one item.

The attached patch uses lookupDestinationsIds() directly and corrects this behavior.

Problem/Motivation

off-canvas.theme.css has a CSS rule that intends to override jQuery UI's default styling:

.ui-dialog.ui-dialog-off-canvas {
  ...
  border: 0 solid transparent;
  ...
}

However, this specific override is not actually applied because jQuery UI's style rule wins, as can be seen in the "before" screenshot.

Proposed resolution

A naive implementation would be to make the off-canvas rule more specific.

Are there any other options?

Remaining tasks

Discuss, review.

User interface changes

The off canvas dialog won't have the 1px white border anymore.

API changes

Nope.

Data model changes

Nope.

Hello,

I want to do a GROUP_CONCAT aggregation but I don't have the option.

Do you know how can I do ?

Thank you

Problem/Motivation

I just attended a UX meeting and learned, among other things, that the form to collect database credentials needs some improvement. Specific items pointed out were better explanations, as in where to find the database password, and more helpful error messages.

Current

And this is the form after entering incorrect data, taken from #87 in #2562845: Validate file path on Credential form

After applying patch

Proposed resolution

Add help text as suggested in #3. This has been added by another issue.

Display database related errors in the same way as when installing Drupal.

Remaining tasks

UX Review. Apply patch goto /upgrade/credentials or /upgrade and click through to the credential form.
Commit

User interface changes

Yes, the database credential form used in Drupal upgrade, at /upgrade/credentials

API changes

N/A

Data model changes

N/A

Postponed on #2225587: Migrate D6 i18n menu links

Problem/Motivation

A migration for block translated strings is needed. The block translated strings are in the block table and the language code in the i18n_blocks table.

Proposed resolution

Write a custom source plugin, refer to the existing block source plugin.
Write block translation migrations based on the existing block migrations.
And, of course, tests.

The existing method for styling the search fields in the header and the same/similar search field styling for the search results page rely on fixed widths, heights and some absolute positioning. I propose we refactor the code in search.css and search-results.css to rely more on our existing use of Flexbox, and to improve the way we apply the hover/focus effect so that it is more tightly styled and for better responsive support.

Problem/Motivation

BTB has lots of helpful methods that could be re-used in https://github.com/weitzman/drupal-test-traits for testing against existing websites.

Debug ones relevant to this issue:

• formatHtmlOutputHeaders
• getHtmlOutputHeaders
• htmlOutput
• initBrowserOutputFile

Others

We could split these up a few ways I guess but they're all pretty much related to UI testing.

RefreshVariablesTrait

• refreshVariables() - Moves this out of FunctionalTestSetupTrait becuase it is used in drupalGet() and submitForm()

BrowserHtmlDebugTrait

• formatHtmlOutputHeaders()
• getHtmlOutputHeaders()
• htmlOutput()
• initBrowserOutputFile()
• getResponseLogHandler()
• getTestMethodCaller()

UiHelperTrait

• submitForm - Handles submitting forms.
• drupalPostForm() - Same as drupalGet() + handles MarkupInterface objects for the text on buttons.
• drupalLogin()
• drupalLogout()
• assertSession() - Handy helper to get a web assertion object.
• drupalGet() - Handles converting URL objects to path strings.
• buildUrl()
• getAbsoluteUrl()
• prepareRequest()
• drupalUserIsLoggedIn()
• click()
• clickLink()
• getTextContent()
• getUrl()
• assertEquals() - Handles markup interface
• checkForMetaRefresh()
• cssSelect()

Proposed resolution

Move them to a trait(s) so the code can be shared.

• RefreshVariablesTrait
• BrowserHtmlDebugTrait
• UiHelperTrait

Remaining tasks

Reviews

Problem/Motivation

You cannot create a view and try to list rendered entities using relationship.

Steps to reproduce (based on Standard profile):

1. Add an entity reference field named "field_content" to the "page" content type. Allow content > article references.
2. Create and edit a content view.
3. Add a relationship for "Content using field_content." or "Content referenced from field_content."
4. The row style plugin is "Content" by default.
5. Click on "Teaser" in order to change the view mode, nothing happens.
6. You can't choose a view mode even if you don't want to actually use a relationship, just having one on your view triggers this.

The problem is a fatal AJAX error triggered when RowPluginBase::buildOptionsForm() calls RelationshipPluginBase::init() while providing a wrong argument type (Array instead of DisplayPluginBase).

Argument 2 passed to Drupal\views\Plugin\views\relationship\RelationshipPluginBase::init() must be an instance of Drupal\views\Plugin\views\display\DisplayPluginBase, array given, called in /app/web/core/modules/views/src/Plugin/views/row/RowPluginBase.php on line 93

Proposed resolution

Fix it, by passing along the relationship as needed and using it.

Remaining tasks

None.

User interface changes

None

API changes

None

Follow-up to #2695297: Refactor EntityFile and use process plugins instead

Problem/Motivation

File migration support includes support for migrating temporary files. Problems with this include:

1. The D6 implementation references the wrong variable name for the temporary file path (temp_directory_path rather than file_directory_temp).
2. The path for temporary files is always going to be local to the source system, and inaccessible to the destination system (unless they are both hosted on the same server).
3. There's really no reason to migrate temporary files. They are documented to be cleaned up periodically anyway, and migration is an excellent time to clean them up. I believe the only reason a file would persist in a temporary state would be if an upload were abandoned (either due to a code error, or user cancelling content creation after upload).

So, let's simply ignore any temporary files in the source DB.

Proposed resolution

Change the file source plugin queries to exclude temporary files.
Remove any code or tests that accommodate migration of temporary files.
Keep any temporary file references that exist in the database fixtures - add a test to verify that they are excluded.

Remaining tasks

The D6 migration test needs work, it modifies the dump and add temp files. Need to decide what to do here.

User interface changes

N/A

API changes

None.

Data model changes

None.

API page: https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Database%...

The example code is not correctly formatted -- it uses < code > tags instead of @code

Follow-up to #2225681: Migrate D6 i18n blocks translated strings

Problem/Motivation

A migration for custom block (D6 boxes) translated strings is needed. The translations for the title and body of custom blocks is in the i18n strings module.

Proposed resolution

Write a custom source plugin. Note the translation for the title and body of custom blocks are on different rows in the locales_target source. The source plugin needs to keep track of what item is being translated on a row.
And, of course, tests.

\Drupal\user\UserInterface extends \Drupal\Core\Entity\ContentEntityInterface which extends \Drupal\Core\Entity\RevisionableInterface.

Same goes for a whole bunch of other interfaces that extend ContentEntityInterface: those for File, , MenuLinkContent and so on.

But \Drupal\user\Entity\User does not support revisions at all. It's caught in ContentEntityBase:

  public function setNewRevision($value = TRUE) {
    if (!$this->getEntityType()->hasKey('revision')) {
      throw new \LogicException("Entity type {$this->getEntityTypeId()} does not support revisions.");
    }

This means you can't do this:

      if ($entity instanceof RevisionableInterface) {
        $entity->setNewRevision(FALSE);
      }

We shouldn't be violating the interfaces we're implementing.

Problem/Motivation

A batch builder object was added in #2401797: Introduce a batch builder class to make the batch API easier to use. The Drupal core should be updated to use the batch builder.

The code to update can most easily be found by looking for the uses of batch_set(). Example usage is in the class comment for \Drupal\Core\Batch\BatchBuilder.

Proposed resolution

Use BatchBuilder in the following core modules:

• config - #2875283: Update config module to use the new batch builder
• locale - #2875288: Update locale module to use the new batch builder
• migrate_drupal_ui - #2875291: Update migrate_drupal_ui module to use the new batch builder
• node - #2875292: Update node module to use the new batch builder
• simpletest - #2875295: Update simpletest module to use the new batch builder
• system - #2875296: Update system module to use the new batch builder
• update - #2875300: Update update module to use the new batch builder
• user - #2875307: Update user module to use the new batch builder

Remaining tasks

• Fix remaining test failures/code review items in the following core modules:
  • simpletest
  • system
  • update
• Review patch

List of contributors

overall: john@johncook.me.uk, mradcliffe
- config: rajeshwari10 at Valuebound, RajeevK at Sirius, Yogesh Pawar at QED42, James.Shee
- locale: lcngeo
- migrate_drupal_ui: rajeshwari10 at Valuebound, RajeevK at Sirius, Yogesh Pawar at QED42
- node: rajeshwari10 at Valuebound, RajeevK at Sirius, borisson_
- simpletest: time2buzzthetower, RajeevK at Sirius
- system: ccasals at Portland Webworks
- update: kavo
- user: ccasals at Portland Webworks

Problem/Motivation

There are several issues related to primary keys across the 3 database drivers:

1. Dropping a (schema) field that is part of a composite primary key throws an SQL exception on MariaDB >10.2.8. When dropping a field named id, for example:
   

   Key column 'id' does not exist in table

   Even though the field clearly does exist. https://mariadb.com/kb/en/library/alter-table explains this in more detail.

2. When dropping a (schema) field that is part of a composite primary key, MySQL and our SQLite driver implementation remove the field from the primary key but leave the reamining primary key in place. This can leave the table in an invalid state. PostgreSQL on the other hand automatically drops the primary key entirely, which is the safer behavior.
3. When renaming a field that is part of the primary key via Schema::changeField() our SQLite driver does not update the primary key definition accordingly so that the rename fails. This is because Schema::mapKeyDefinition() makes an incorrect assumption about the $mapping variable; it looks up a value by key, but it must look up a key by value.

Proposed resolution

1. See 2.
2. When dropping a field on any of the by Drupal supported databases (MySQL, PostgreSQL and SQLite), check whether it is part of a composite primary key, and - if so - drop the primary key - and do not recreate it afterwards.
3. Fix SQLite's Schema::mapKeyDefinition() so that renaming a primary key field works on SQLite.

Problem/Motivation

A bug in libraries-override #2451411: Add libraries-override to themes' *.info.yml makes it difficult to override a library or library asset that has already been overridden. You have to know the theme which last overrode a library and then use the full path to the overriding theme asset. This presents a WTF to themers and makes it difficult to reuse theme code. It also makes the sub-theme dependent on the internal file structure of the base theme.

In normal circumstances overriding an already overridden library would be and edge case, but with the introduction of the Stable base theme, a lot of system libraries have already been overridden. This also implies that possible future changes in the asset file structure of Stable could break sub-themes that override libraries already overridden by Stable.

Workaround

The current workaround is explained in #2642122-4: Overriding already overridden libraries-override requires knowledge of previous libraries-overrides and #2642122-18: Overriding already overridden libraries-override requires knowledge of previous libraries-overrides.

Proposed resolution

1. Fix the bug so that the libraries-override declaration should use the same keys as the libraries declaration in the original libraries.yml file.
2. Maintain a BC layer so that themes that have already used the workaround will not break immediately. Remove the BC layer in D9 (#2852314: Remove BC layer for libraries-override that is already overridden)

Remaining tasks

Reviews
Commit

User interface changes

None

API changes

Method signatures of 3 protected methods in LibraryDiscoveryParser are changed

Data model changes

None.

Link to Taxonomy term get invalid language path if my language is not origin. Same bug if get Image (Link to content) get same error

How to reproduce

1. enabled language and content translation
2. add another language (ex: RO)
3. create translatable taxonomy vocabulary
4. enable content translation for taxonomies, and for the fields of the new vocabulary terms
5. add to this vocabulary a term (ex: Test) and translate it (Test(Ro))
6. create a taxonomy term view that displays fields, add "Name" and "Link to taxonomy term"

Notice how the two different fields have different links for the term. The term "Name" field ads correct link for both languages (en: "/taxonomy/term/1", ro: "/ro/taxonomy/term/1"), but the "Link to taxonomy term" adds for both the default language link (en: "/taxonomy/term/1", ro: "/taxonomy/term/1"

This bug for all my project. That i made start at 8.2.2

Problem/Motivation

Custom blocks are now publishable, but the Custom block library page (/admin/structure/block/block-content) doesn't not reflect this change. Since the custom blocks as publish-able, make the UI close to node admin page (/admin/content):

This work was started In #2834546: UI for publishing/unpublishing block_content blocks, but it was decided to split the changes to block_content view and related into a follow up issue, because this requires an upgrade path.#2834546: UI for publishing/unpublishing block_content blocks should be committed before this issue gets in.

Proposed resolution

Introduce below changes:
1. Bulk operation form
2. Add published column
3. Add publish status filter

and write the upgrade path in this issue.

Remaining tasks

- Implementation - Done
- Manual testing - Done
- More reviews
- Commit!

User interface changes

Adds a bulk publish / unpublish actions and a new "published" column to the Custom block library page /admin/structure/block/block-content

API changes

None

Data model changes

None

Problem/Motivation

We should remove usages of AssertLegacyTrait::getRawContent() with $this->getSession()->getPage()->getContent().

Proposed resolution

Replace AssertLegacyTrait::getRawContent() with $this->getSession()->getPage()->getContent()

Remaining tasks

User interface changes

API changes

Data model changes