Step to reproduce:
And now we can see the following item in the log:
Missing text format: basic_html.
The patch does the following:
- Changes the empty text for the dblog view to not use a text format.
- Provides an update hook that updates this part of the view if it hasn't been customized, for existing sites, and a test for the update path.
- Removes test code in DbLogViewsTest that was working around this bug by creating the Basic HTML format before running the parent method. The parent method (which tests the no-views version of the log) is now testing that you can clear the log and see the empty text.
- Adds a new test that ensures the empty text in this view doesn't get updated later to use a text format.
True, there is a MigrateEvents::PRE_ROW_SAVE event that allows subscribers to react before the row is passed to the destination plugin, but this is not enough. For the case when the destination is an entity (and this is probably >95% of the cases), there's now way to intercept the already formed entity object and manipulate it as an object.
In the D7 contrib module we had ::prepare(stdClass $entity, stdClass $row), where the entity was already populated but not saved. We need something similar because, sometimes, we have the case where we need to use also the methods on the entity, thus we need the entity object. For example I have a project, where I'll have to add the translations in a non-canonical way, so I need to intercept the $entity object, the just $entity->addTranslation(...).. I can imagine a lot of cases where we want to intercept the entity object just before being saved by the destination plugin.
Add a new MigrateEvents::PRE_ENTITY_SAVE event and let the entity destination plugins dispatch it. Create a new event wrapper MigratePreEntitySaveEvent that allows also the event to be retrieved along with the $row.
None.
None.
New:
MigrateEvents::PRE_ENTITY_SAVE\Drupal\migrate\Event\MigratePreEntitySaveEventNone.
Issue #2554321: Clean up Migrate's test suite introduces helper methods in MigrateDrupal6TestBase, that allow to perform bundled migrations in tests, such as migrateUsers(), migrateContentTypes(), migrateFields(), ...
Add D7 variants of these helper methods to MigrateDrupal7TestBase
Postponed on #2687843: Add back incremental migrations through the UI
A review of the patch in #2687843: Add back incremental migrations through the UI comment #98 included some points that were outside the scope of the issue and need to be addressed.
The relevant is in comment#99
This is a copy of the points to be addressed in this issue
Now in IncrementalForm.
2. +++ b/core/modules/migrate_drupal_ui/src/Form/MigrateUpgradeForm.php
@@ -236,6 +244,51 @@ public function submitOverviewForm(array &$form, FormStateInterface $form_state)
+ $database_state_key = $this->state->get('migrate.fallback_state_key', '');
+ if ($database_state_key) {
Why would we use the fallback state key here? The user has entered explicit database credentials here, so I don't understand why we wouldn't just use those.
Now in OverviewForm.
3. +++ b/core/modules/migrate_drupal_ui/src/Form/MigrateUpgradeForm.php
@@ -236,6 +244,51 @@ public function submitOverviewForm(array &$form, FormStateInterface $form_state)
+ $form_state->setValue('step', 'credentials');
+ $form_state->setRebuild();
Nit: setValue() is chainable, so we could do ->setValue()->setRebuild().
Now in MigrateUPgradeFormBase.
4. +++ b/core/modules/migrate_drupal_ui/src/Form/MigrateUpgradeForm.php
@@ -848,6 +877,44 @@ protected function getDatabaseTypes() {
+ * Setup migrations.
+ *
+ * @param array $database
+ * Database array representing the source Drupal database.
+ * @param \Drupal\Core\Form\FormStateInterface $form_state
+ * The current state of the form.
+ */
+ protected function setupMigrations(array $database, FormStateInterface $form_state) {
Can this doc comment be expanded a bit? "Setup migrations" tells me nothing more than the method name already does.
Now in CredentialForm.
5. +++ b/core/modules/migrate_drupal_ui/src/Form/MigrateUpgradeForm.php
@@ -848,6 +877,44 @@ protected function getDatabaseTypes() {
+ $form_state->set('version', $version);
+ $form_state->set('migrations', $migration_array);
Nit: ->set() is chainable.
10. +++ b/core/modules/migrate_drupal_ui/tests/src/Functional/MigrateUpgradeTestBase.php
@@ -315,4 +281,110 @@ protected function translatePostValues(array $values) {
+ * @param $session
+ * The currenct session.
$session needs a type hint (also in the method signature), and there's a typo in 'current'.
11. +++ b/core/modules/migrate_drupal_ui/tests/src/Functional/MigrateUpgradeTestBase.php
@@ -315,4 +281,110 @@ protected function translatePostValues(array $values) {
+ * @param $session
+ * The currenct session.
+ */
+ protected function assertIdConflict($session) {
$session needs to be type hinted, and 'current' is misspelled. :)
12. +++ b/core/modules/migrate_drupal_ui/tests/src/Functional/MigrateUpgradeTestBase.php
@@ -315,4 +281,110 @@ protected function translatePostValues(array $values) {
+ // Have to reset all the statics after migration to ensure entities are
'statics' should be 'static caches'.
Respond to the feedback
This will needs a patch, a review etc.
Follow-up from #2791041: Migrate source plugins should use dependency injection
They currently do not, and it causes a lot of weird acrobatics and unsavory hacking in Migrate's tests. The way around this problem, in my opinion, is the path through the jungle: add proper dependency injection to all Migrate source plugins in core and, ultimately, completely remove all mentions of \Drupal from SourcePluginBase and its many descendants.
This follow-up is to help smooth the way to the cleanup for D9 in D8 considering Migrate is in beta but Migrate Drupal is still in experimental.
This will only touch the parts in migrate_drupal that are BC breaking considering it's in experimental
Sources and destinations are sharing some common methods like getIds() and fields(). They are referring exactly the same things but each on the opposite endpoint of the pipe.
Create a new interface \Drupal\migrate\Plugin\MigrateEndpointInterface and move ::getIds() and fields() from MigrateSourceInterface and MigrateDestinationInterface there. Make MigrateSourceInterface and MigrateDestinationInterface extend the new MigrateEndpointInterface. I'm not sure that the "endpoint" term from the interface name is the best - I'm not a native English speaker. By that, I want to describe something that is either source or destination. Opinions?
Right now, this hard blocked by #2543568: Remove the md_entity destination plugin hack but depends also on #2783715: Entity destination ID schema should match entity ID definition in terms of documentation.
Un-postpone when #2543568: Remove the md_entity destination plugin hack is in.
None.
None.
None.
I have tested that code:
process:
uri:
plugin: file_copy
reuse: true
source:
- source_file_path_remote
- destination_file_path
In the transform method of file_copy process plugin the download process plugin is called when a source file is remote. So the file is downloaded ignoring the reuse option.
In older versions of Firefox, details elements in the Seven theme in Drupal 8.x, such as on admin/config/system/site-information , looked like this screenshot -- the summary line (which is a link to open/close the details area) showed up in blue, with a little triangle open/close icon next to it:
With current versions of Firefox, however, it isn't so nice. The summary line is black, and the open/close icon is missing:
This is a big usability problem. It's not obvious without the little arrow or the blue color that this is a link to open/close the box, at least to me.
After some debugging, the problem was traced to the normalize.css file [see original issue summary in first comment for details]. We are currently using a 3.x version of the Normalize library, and it has "summary" elements with display: block;. The current version of the Normalize library uses display: list-item; instead.
So, we need to either:
a) Update to the current version of normalize. This option was rejected by the core maintainers in comment #13, and has been moved to its own issue: #2821525: Update normalize.css to 7.0.0
b) Add a small bit of CSS to our normalize library to override display: block for summary elements. That is what is in the current patch.
a) [done] Figure out what to do [Make a small patch to add CSS to override that one piece of the normalize library].
b) [done] Make a patch.
c) Test the patch on multiple browsers:
- Firefox - works - tested in comment # 20
- Chrome - works - tested in comment # 22
- Apply the patch on a local Drupal site. It doesn't really matter which version of Drupal you are using.
- Clear the cache.
- Visit a page like admin/config/system/site-information , using the current version of a browser that has not yet been tested (see Proposed Resolution section, above this in the issue summary).
- See if the summary lines on details elements are formatted correctly (with the drop-arrow) or not.
- Make a screenshot similar to the ones in comments #20 and #22.
- Add a comment telling which browser you tested, and upload your screenshot in the Files section.
Details elements on admin pages will again have the drop-down arrow applied to them in current versions of Firefox. Right now they don't and it's rather confusing UI.
None.
None.
While migrating content to Drupal 8, it would be great if the HTML could be cleaned up a bit. All part of Authoring Tools Accessibility Guidelines Principle B.1: Fully automatic processes produce accessible content
B.1.1.2 Content Auto-Generation During Authoring Sessions
"If the authoring tool provides the functionality for automatically generating web content during an authoring session, then at least one of the following is true: (Level A to meet WCAG 2.0 Level A success criteria; Level AA to meet WCAG 2.0 Level A and AA success criteria; Level AAA to meet all WCAG 2.0 success criteria)
(a) Accessible: The content is accessible web content (WCAG) without author input; or
(b) Prompting: During the automatic generation process, authors are prompted for any required accessibility information (WCAG); or
(c) Automatic Checking: After the automatic generation process, accessibility checking is automatically performed; or
(d) Checking Suggested: After the automatic generation process, the authoring tool prompts authors to perform accessibility checking."
https://www.w3.org/TR/2015/NOTE-IMPLEMENTING-ATAG20-20150924/#sc_b112
B.1.2.1 Restructuring and Recoding Transformations
"If the authoring tool provides restructuring transformations or re-coding transformations, and if equivalent mechanisms exist in the web content technology of the output, then at least one of the following is true: (Level A to meet WCAG 2.0 Level A success criteria; Level AA to meet WCAG 2.0 Level A and AA success criteria; Level AAA to meet all WCAG 2.0 success criteria)
(a) Preserve: Accessibility information (WCAG) is preserved in the output; or
(b) Warning: Authors have the default option to be warned that accessibility information (WCAG) may be lost (e.g. when saving a vector graphic into a raster image format); or
(c) Automatic Checking: After the transformation, accessibility checking is automatically performed; or
(d) Checking Suggested: After the transformation, the authoring tool prompts authors to perform accessibility checking."
https://www.w3.org/TR/2015/NOTE-IMPLEMENTING-ATAG20-20150924/#sc_b121
B.1.2.2 Copy-Paste Inside Authoring Tool
"If the authoring tool supports copy and paste of structured content, then any accessibility information (WCAG) in the copied content is preserved when the authoring tool is both the source and destination of the copy-paste and the source and destination use the same web content technology."
https://www.w3.org/TR/2015/NOTE-IMPLEMENTING-ATAG20-20150924/#sc_b122
B.1.2.4 Text Alternatives for Non-Text Content are Preserved
"If the authoring tool provides web content transformations that preserve non-text content in the output, then any text alternatives for that non-text content are also preserved, if equivalent mechanisms exist in the web content technology of the output."
https://www.w3.org/TR/2015/NOTE-IMPLEMENTING-ATAG20-20150924/#sc_b124
Would be nice to evaluate the accessibility of pages you are importing using something like aXe or Pa11y.
There's no reason we couldn't eliminate (or at least identify) a number of common accessibility problems through the migration process.
The UI standard page for Dropdownn needs to be updated https://www.drupal.org/docs/develop/user-interface-standards/dropbutton
Rewrite the page and add missing screenshots.
Check whether core modules follow this standard.
1. How-Why-Examples. Change the order of the text to first say how dropdowns should be used, then why, and then follow it up by a good example.
2. If screenshots are missing: add them.
3. Add a one sentences summary that's useful on https://www.drupal.org/docs/develop/user-interface-standards
4. Add the correct Drupal 8 API links
5. Check several core admin pages whether they use the dropdowns this way. Add in a comment which pages you checked.
6. If you find a admin page that does not follow this standard: make an issue to fix this.
Migration plugins throughout core and contrib keep having to do this sort of thing:
$delimiter = isset($this->configuration['delimiter']) ? $this->configuration['delimiter'] : '';
Implementing https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Component%21Plug... means that plugins can define default values for any configuration in defaultConfiguration().
The result is:
- better DX as you don't have to keep checking array values in $this->configuration are defined
- better DX as the default value for something in $this->configuration is in one place only
- better DX as there's a place where all the configuration options for the class are listed (well the class docs should have that too... but it nudges modules that are lax on docs towards listing them)
This can be done in SourcePluginBase, ProcessPluginBase, & DestinationBase.
From https://www.drupal.org/node/2281691#comment-10286177...
webchick:
OBSERVATION There doesn't seem to be any validation around the file path field? Or at least I wasn't able to trigger it despite typing in garbage values for both local filepath and http:// address (that could've been due to "migrate already happened" errors tho, see below... but in any case we should probably validate the form first before we proceed).
mikeryan:
So, file_exists() on a local filepath and... get_headers() on an HTTP URL? Or, anything with ://, no reason you couldn't put the files on ftp://...
Current screenshot from #30 showing error when tested with Drupal 7 source and both of the file fields have errors
We recently noticed a situation where a cached page on a reverse-proxy (in our case, cloudflare) was referring to a deleted aggregated js file, and so was getting a 404 response on that file, so the page was broken.
Drupal uses a configuration setting (system.performance.stale_file_threshold) to perform garbage collection of optimized js and css assets.
This is set by default to 30 days.
Our problem is that the js file was deleted from the file system, even though a reference for the js file still exists in the cloudflare cache.
This is a large-scale site, so we avoid purging the cloudflare cache if we can help it, so the situation may arise when the cloudflare page is referencing a deleted js file, simply because the js file was generated more than 30 days ago.
Looking at the following code (src/docroot/core/lib/Drupal/Core/Asset/JsCollectionOptimizer.php:177)
public function deleteAll() {
$this->state->delete('system.js_cache_files');
$delete_stale = function ($uri) {
// Default stale file threshold is 30 days.
if (REQUEST_TIME - filemtime($uri) > \Drupal::config('system.performance')->get('stale_file_threshold')) {
file_unmanaged_delete($uri);
}
};
file_scan_directory('public://js', '/.*/', ['callback' => $delete_stale]);
}
... I am wondering if it might be better (depending on a config setting, and if the underlying file system supports it) to use fileatime() as the check instead of filemtime().
In other words, we would only mark the asset file as stale, if it hasn't been *accessed* in over 30 days, rather than depending on whether it was *modified* over 30 days ago.
Field names are limited to 32 characters, but profile field names may be longer on older sites. A Drupal 6 site with profile field names longer than 32 characters cannot be fully migrated.
Output from drush for a d6 site:
Attempt to create a field storage with an name longer than 32 characters: profile_expertise_electoralsystem [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Attempt to create a field storage with an name longer than 32 characters: profile_expertise_executivesystem [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Attempt to create a field storage with an name longer than 32 characters: profile_expertise_legislativesystem [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Attempt to create a field storage with an name longer than 32 characters: profile_expertise_naturalresources [error]
(/www/cn8/core/modules/field/src/Entity/FieldStorageConfig.php:315)
Processed 39 items (35 created, 0 updated, 4 failed, 0 ignored) - done with 'upgrade_user_profile_field' [status]
The resolution should take into account the field storage limitation to migrate Drupal 6/7 profile fields as field storage and fields, and map these new field names so that a migration of profile values will succeed.
All these requirements are met with make_unique_entity_field and setting a limit to 30 chars. That gives us 99 possible fields on a d6 user and surely is enough for the vast majority of sites.
- Update issue summary to make sure everything is clear going forward- Fix test/patch- Review
N/A
n/a
n/a
The Taxonomy and User Representative Node (GroupwiseMax) using a RepresentativeView does not work because the existing code generates SQLSTATE[42000]: Syntax error or access violation and SQLSTATE[HY093]: Invalid parameter number: no parameters were bound.
To reproduce:
Result:
SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '***CURRENT_USER*** AND ***CURRENT_USER*** <> 0 AND ***VIEW_OWN_UNPUBLISHED_NODES' at line 3: SELECT COUNT(*) AS expression FROM (SELECT 1 AS expression FROM {users_field_data} users_field_data LEFT JOIN {node_field_data} node_field_data_users_field_data ON (SELECT node_field_dataINNER.nid AS nidINNER FROM {node_field_data} node_field_dataINNER INNER JOIN {users_field_data} users_field_data_node_field_dataINNER ON "node_field_dataINNER".uid = "users_field_data_node_field_dataINNER".uid LEFT JOIN {users_field_data} users_field_data_node_field_data_1INNER ON "node_field_dataINNER".uid = "users_field_data_node_field_data_1INNER".uid WHERE (("users_field_data_node_field_data_1INNER".uid = users_field_data.uid )) AND (("node_field_dataINNER".status = "1 OR (node_field_dataINNER".uid = ***CURRENT_USER*** AND ***CURRENT_USER*** <> 0 AND ***VIEW_OWN_UNPUBLISHED_NODES*** = 1) OR ***BYPASS_NODE_ACCESS*** = 1)) ORDER BY node_field_dataINNER.changed DESC LIMIT 1 OFFSET 0) = node_field_data_users_field_data.nid WHERE (users_field_data.default_langcode = :db_condition_placeholder_0) AND (users_field_data.uid != :db_condition_placeholder_1)) subquery; Array ( [:db_condition_placeholder_0] => 1 [:db_condition_placeholder_1] => 0 )
Apply this patch, clear the cache, and refresh the View edit page to see it work. Add a node if you want to see it get data.
Tests are needed for this and it seems GroupwiseMax in general (without the Representative View.) Maybe additional discussion is needed? Maybe there is a better way to replace the **CORRELATED** placeholder? Maybe tests should be a different issue?
This issue also exists in Views for Drupal 7. I created the patch for D7 Views over a year ago here: https://www.drupal.org/node/1417090 #35 with a recent update in #53. The patch in #35 has ~10 RTBC's.
It is hard to notice that static_map was the reason a migration row was skipped.
Log a message to MigrateSkipRowException to clearly explain why the row is skipped.
Patch
N.A.
N.A.
N.A.
At the moment when $settings['skip_permissions_hardening'] = TRUE, the system module's system_requirements unconditionally issues a warning about the site's config files' permissions without actually checking them:
if (file_exists($full_path) && (Settings::get('skip_permissions_hardening') || !drupal_verify_install_file($full_path, FILE_EXIST | FILE_READABLE | FILE_NOT_WRITABLE))) {
$conf_errors[] = t("The file %file is not protected from modifications and poses a security risk. You must change the file's permissions to be non-writable.", ['%file' => $full_path]);
}
This is quite confusing and misleading (see related issues).
Drupal should either skip the checks and not issue any warnings about the state of specific conf files, or perform the checks but only issue any valid warnings (as opposed to errors).
A simple warning to remind developers not to carry this configuration over into production would seem sufficient.
* Decide whether to skip checks, or perform them and issue only valid warnings.
* Patch the existing code accordingly.
* Update tests as appropriate.
* Removal of invalid warnings when config file permissions are not actually incorrect.
* None.
* None.
Wondering why there is no corresponding module_get_weight() for module_set_weight(). The typical use case is to set my module's weight to one higher than another module, but I don't necessarily know what the weight of the other module is:
$weight = (int) db_query("SELECT weight FROM {system} WHERE name = 'dependency'")->fetchField(); db_update('system') ->fields(array('weight' => $weight + 1)) ->condition('name', 'mymodule') ->execute();I don't see a way to possibly do this with the newly-converted code.