Problem/Motivation

On Drupal sites where users are uploading files clean filenames are hard to achieve because Drupal accepts most filenames as valid input. However lots of Drupal installs override this functionality, for example Thunder has been shipping a similar solution for a very long time. As @dww writes in #132

Content creators name their files all kinds of weird and unfortunate things. Even if everything is in English, it's still great to convert spaces to dashes, remove weird punctuation, special characters, etc. Plus, many users find themselves in a mix of case-sensitive and case-insensitive filesystems, so it's great to have Drupal automatically convert everything to lowercase. I constantly build custom functionality into my sites to cleanup filenames on upload so as to prevent the editors from making a mess of things. It'd be much better if core did that for me automatically.

Follow-up to #1842718: Use new Transliteration functionality in core for machine names and #567832: Transliteration in core.

@catch mentioned in #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols) we may want to transliterate filenames in core, so that we can have a database-level unique constraint on the URI field in the database. (However, this would presumably mean that transliteration of the field could not be optional).

Proposed resolution

Add a fieldset to the File system form (/admin/config/media/file-system) to opt-in for various kinds of filename sanitization:

• Transliterate the filename. Disabled by default, because it's not useful for some languages like Japanese.
• Replace whitespace with dash (-) or underscore (_).
• Replace any characters other than 0-9A-Za-z or - or _ or . with dash (-) or underscore (_). Requires that the 'Transliterate' option is enabled.
• Remove duplicate separator characters ie -- or .. or __ with be replaced with - or . or _.
• Convert to lowercase (to prevent issues on case-insensitive file systems).

On all files uploaded through the API and UI (i.e. not aggregated JS/CSS files, etc), the filename will be sanitized in whatever ways the site is configured to enforce.

We listen to the \Drupal\file\Event\FileUploadSanitizeNameEvent and change the filename according the the configuration.

Note: These sanitization settings only impact new file uploads, all files already uploaded to a site are not affected.

Remaining tasks

1. Decide if all whitespace or only spaces are converted. Update code and UI text to match. All whitespace as of #159.
2. Design and dispatch the appropriate event(s) where core's sanitization is done. Complete via #187.
3. Decide if the FileUploadEvent should also let listeners munge the destination. If so, update the Event accordingly. No (via @alexpott in #192.2).
4. Decide if we give the user enough feedback about renaming their file upon upload. No (via @alexpott in #203). If not, implement the appropriate feedback (Complete via #214).
5. Finish cleaning up the UI text, settings names, etc: Done via #219.
   • What's the right label for the 'strip' or 'remove' option for replace_non_alphanumeric? For now: "- Strip -". Previous: "- Remove (do not replace) -". Other? Option D: remove it entirely, via #219.
   • Should the options for the two selects (replace_*) be capitalized? Sure. ;) Via #194
   • Any other concerns/edits/fixes?
6. Finish writing Kernel test cases for (all?) the combinations of settings. Done via #213 and #219.
7. Upload screenshot(s) of final settings UI and update the summary. Done via #219.
8. Final code cleanup and fixes. No more @todo via #219.
9. Update/fix the change record. Done via #221
10. Add tests for REST integration
11. Decide if we're allowing transliteration as an option(!). See #272.
12. Decide if we should let people pick a transliteration language. See #270 and #272.
13. Usability review:
    • Decide if we want a UI at all, or if sites should simply configure this via settings.php (like they do for the public + private files dir paths). See #272.
    • If yes to a UI:
      • Decide whether we want to be able to have different replacement characters for whitespace, non-alphanumeric characters, and transliteration unknown characters. See #180, #246, #258, #261, and others...
      • If we have it as a separate / shared setting, decide if the "replacement character" setting should be "near" the checkboxes that provide replacement or at the end of the fieldset (see #263 for comparisons)
      • Decide if "replace non-alphanumeric" should be "near" transliterate option. See #241 and #242.
      • ...
14. Final string/UI review + signoff
15. Final reviews + RTBC.
16. Commit.

User interface changes

New "Filename sanitization" fieldset of options on FileSystemForm (/admin/config/media/file-system).

Pre-#245

Screenshot of the new settings UI :

The "Replace non-alphanumeric characters..." setting depends on "Transliterate" being enabled:

Available options under either "Replace ..." setting:

#245

Alternate UI from #245:

Options under "Replacement character" setting:

API changes

None

Data model changes

New configuration mapping file.settings:filename_sanitization, with options for:

• transliterate: Transliterate
• replace_whitespace: Replace whitespace
• replace_non_alphanumeric: Replace non-alphanumeric characters
• dedupe_separators: Remove duplicate dots (..), underscores (__) or dashes (--)
• lowercase: Convert to lowercase

Release notes snippet

@todo

Problem/Motivation

Somehow, somewhere, an incorrect indentation slipped into /core/modules/tour/src/TourViewBuilder.php which now results in any change in that file being rejected by TestBot.

FILE: /var/www/html/core/modules/tour/src/TourViewBuilder.php
----------------------------------------------------------------------
FOUND 2 ERRORS AFFECTING 2 LINES
----------------------------------------------------------------------
 74 | ERROR | Comment indentation error, expected only 1 spaces
    |       | (Drupal.Commenting.InlineComment.SpacingBefore)
 78 | ERROR | Comment indentation error, expected only 1 spaces
    |       | (Drupal.Commenting.InlineComment.SpacingBefore)
----------------------------------------------------------------------

This was first spotted by @quietone here and confirmed by @daffie here.

Steps to reproduce

- Change something in /core/modules/tour/src/TourViewBuilder.php
- Create a patch/MR with that change
- See TestBot reject the patch in the Custom Commands phase

Proposed resolution

Attached MR should fix the indentation problems.

Remaining tasks

- Review
- Commit
- Rejoice

https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Render%21...

The attribute description "(string, optional) A string containing the textual contents of the tag" does not tell the reader whether HTML escaping is performed. I'd prefer instead:

#value: (HTML string, optional) The markup content of the tag. No escaping is performed on output.

Agenda items

Standing items

• Who is here
• Any other suggested topics
• Wins / Thanks / Blockers
• Statistics and functionality update. X new bugs created in last two weeks. Y closed by the initiative.
• The bug I would most like to see fixed
• Update on target items from last meeting
• What should be the initiative focus for the next 2 weeks?

Statistics

Statistics graph

New Items

Add here

ToDos

TBA

Agenda items

Standing items

• Who is here
• Any other suggested topics
• Wins / Thanks / Blockers
• Statistics and functionality update. X new bugs created in last two weeks. Y closed by the initiative.
• The bug I would most like to see fixed
• Update on target items from last meeting
• What should be the initiative focus for the next 2 weeks?

Statistics

Statistics graph

0️⃣ Who is here today? Comment in the thread to introduce yourself. We’ll keep the meeting open for 24 hours to allow for all timezones.

0️⃣ .5️⃣ What are you reading at the moment?

1️⃣ Do you have suggested topics you are looking to discuss? Post in this thread and we’ll open threads for them as appropriate.

2️⃣.1️⃣ Wins - what wins have you had since the last meeting? Share your successes!

2️⃣.2️⃣ Thanks - a space for callouts to other contributors who’ve helped you

2️⃣.3️⃣ Blockers - what’s getting in the way of your issues / goals

3️⃣ Core bug statistics - current state of open bugs

Today: 2021-07-20

From: 2021-07-06 to 2021-07-20

  All bugs
  Open: 56, including 17 fixed
  Closed: 166
    Age   Count
     00   37
     01   18
     02   09
     03   04
     04   01
     05   07
     06   03
     07   15
     08   18
     09   22
     10   18
     11   08
     12   01
     13   03
     14   01
     17   01

           Status            Count
  closed (duplicate)           26
  closed (won't fix)            4
  closed (works as designed)   16
  closed (cannot reproduce)    35
  closed (outdated)            29
  closed (fixed)               56

  Approximately 1034 yr reduction in total number of years of all open bugs.

  Only bugs tagged Bug Smash Initiative
  Open: 12, including 9 fixed
  Closed: 95
    Age   Count
     00   11
     01   07
     02   05
     03   01
     04   01
     05   04
     06   03
     07   11
     08   13
     09   13
     10   14
     11   06
     12   01
     13   03
     14   01
     17   01

           Status            Count
  closed (duplicate)           16
  closed (won't fix)            2
  closed (works as designed)    9
  closed (cannot reproduce)    23
  closed (outdated)            19
  closed (fixed)               26

  Approximately 746 yr reduction in total number of years of all open bugs.

4️⃣ Nominate your pet issue for a bug-smash target for the next fortnight

5️⃣ Help me - Is there anything you could use a :hand:

6️⃣ @quietone - Approving minutes of last meeting (edited) 

:bell: Thanks bugsmashers, the meeting is over. Please keep chatting in the threads.

Participants:

quietone, pameeela, larowlan, Spokje, catch, kim.pepper, mohit_aghera, Paulo Henrique Cota Starling, lendude, Griffyn Heels

Introduction

The Bug Smash Initiative is a community effort to fix known bugs in Drupal core. See the Bug Smash Initiative handbook page for more information.

We meet fortnightly on Tuesday at 0400 UTC in the #bugsmash channel in Drupal Slack. Beware, this might be Monday evening for some people:

• 2:00 pm Tuesday - Australian Eastern Standard Time (AEST)
• 9:30 am Tuesday - India Standard Time (IST)
• 6:00 am Tuesday - Central European Standard Time (CET)
• 9:00 pm Monday - Pacific Daylight Time (PDT)
• When in doubt, ask the oracle.

All meetings are posted as child issues.

Please do not comment on this issueexcept to add a new child issue for an upcoming meeting and change the title here. Thanks!

Next meeting

The next meeting is on Tuesday 2021-08-03.

#3224867: Meeting of the Bug Smash Initiative 2021-08-03

Past meetings

Please see the "Child issues" block in the right sidebar or at the bottom of this page for links to past meetings.

Problem/Motivation

In #3211780: Deprecate Connection::queryTemporary() we deprecated Connection::queryTemporary but forgot to deprecate Connection::$temporaryNameIndex too.

Proposed resolution

Add a deprecation notice so we can remove in D10.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

This issue is part of the composer initiative: #2958021: Proposal: Composer Support in Core initiative

Problem/Motivation

The vendor cleanup scripts were moved into a Composer plugin in #3057094: Add Composer vendor/ hardening plugin to core. This is a follow-on issue to that task.

The original sanitization scripts still exist in the repository at Drupal\Core\Composer\Composer::vendorTestCodeCleanup(), and are called as an event script in drupal/drupal's composer.json like this:

"scripts": {
        "post-package-install": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
        "post-package-update": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",

These scripts are now deprecated, and should be removed.

Proposed resolution

Once the deprecated scripts are no longer being called, they should be removed. There are three potential places where these scripts may be called:

1. From sites installed from a tarball download of Drupal
2. From git clones of the Drupal repository (core development)
3. External sources

The first set of calls will be removed by #2982680: Add composer-ready project templates to Drupal core. When the Drupal template projects are created, they will use the new Vendor Hardening plugin rather than the cleanup scripts. Once tarballs are generated from the template projects, the scripts will be unused in tarball-based sites (managed via `drush pm-update` etc.).

The second use-case is Drupal core development. To stop using the scripts during core development, the Vendor Hardening plugin will need to be added as a "path" repository of the root-level composer.json file of drupal/drupal, and the references to the scripts in the "scripts" section will need to be removed. This work can be done as part of this issue.

The final place where these scripts may be called is from external sources, e.g. community-managed Drupal template projects. These cleanup scripts are not needed for Drupal sites that utilize a relocated document root, so drupal-composer/drupal-project, for example, does not call these scripts. At the moment, there are no known examples of external projects that use these scripts. If there are any that exist, then this work cannot be done until Drupal 9.

Remaining tasks

Decide on the timing of when to do this work.

User interface changes

None.

API changes

None.

Data model changes

None.

Release notes snippet

tbd

Tl;dr

oEmbed automatically lowercases scheme urls making it impossible to use schemes that should have capital letters in them. It also makes it impossible to accept any urls with capital letters in them (when these capital letters are part of the scheme).

Description

Drupal relies on https://oembed.com/providers.json to indicate the possible schemes for streaming video urls from various services. Vimeo has the scheme "https:\/\/player.vimeo.com\/video\/*" etc. You can also implement your own oEmbed providers.json.

The bug is that the oEmbed Endpoint class lowercases the schemes automatically, but compares them against a non-lowercased user input with a case sensitive regex. This means that any capital letters in the scheme will cause the regex to fail, making it impossible to create your own scheme that has capital letters.

I discovered the bug while creating a custom oEmbed provider for Panopto (academia video streaming service). The scheme looks like this:

"https:\/\/demo.hosted.panopto.com\/Panopto\/Pages\/Viewer.aspx*"

As you can see, there are uppercase words in the url scheme and lowercasing the scheme automatically and then comparing it case sensitively to a Panopto supplied video url will always fail.

The offending code is in Drupal\media\OEmbed\Endpoint.php on line #72 (array map of schemes to mb_strtolower) and in the method matchUrl (line #155) where the case sensitive regex is made.

Steps to reproduce

Unfortunately this is a chore to reproduce, since there are currently no official oembed schemes with capital letters.

- You need to implement your own @MediaSource class that extends OEmbed.
- You also need to implement your own oEmbed providers.json and configure a provider (such as Panopto) with a variable case scheme.
- You also need to override media.settings for your custom oEmbed providers.json so Drupal reads it instead of the canonical one.
- Don't forget to clear your cache to load the new oEmbed providers.
- Now add your own media type (structure / media types / add) using the MediaSource class you just implemented.
- In the case of Panopto, which doesn't have its own oEmbed endpoint, we have to simulate / implement one ourselves in Drupal.
- Now try to add a Panopto video using the example above. It will never be accepted, because Drupal lowercases your scheme and compares it against your Panopto url with capital letters in it.

Expected behavior

I expect the media entity to be accepted when its url matches my oEmbed provider scheme even if the scheme has capital letters in it.

What happened instead

The media is rejected with the error message "The provided URL does not represent a valid oEmbed resource.".

Proposed solution

I can think of two easy ways:

- Change the regex to be case insensitive in Endpoint.php #155 OR
- Remove the lowercasing of schemes in Endpoint.php #72

At first glance, the latter seems the more appropriate one. We should not lowercase schemes if there is a chance a valid url (such as in the case of Panopto) has capital letters in it.

Split off from #3219958: [META] Make Olivero the default theme for Drupal 9.3.

Note that we hope to have this completed by Drupal 9.3. There will be work on this as discover which tests are dependent on Bartik.

EntityReferenceFieldItemList::referencedEntities is returning the entities from default translation, but should translate the entity directly in the translation of the parent

This is split off from #3219959: Update standard profile so Olivero is the default theme and is closely related to #3151553: Make core views grid styles responsive.

Problem

Olivero does not respect the following Views setting when the user selects the "grid" format option.

Why?

Because Olivero's implementation of Views Grid is responsive. The number of columns will change depending on the width of the browser's viewport. This IMO should be a requirement.

You can test this out at

• Horizontal grid view: https://tugboat-aqrmztryfqsezpvnghut1cszck2wwasr.tugboat.qa/grid-view
• Vertical grid view: https://tugboat-aqrmztryfqsezpvnghut1cszck2wwasr.tugboat.qa/grid-view/vert

Because the default implementation of Views Grid was written so long ago (before responsive web design was a thing), there were wrappers put around each row (or column). The settings within the Views UI allow the site-builder to add CSS classes onto those wrappers. Because Olivero does away with those wrappers, we have no place to put these classes.

What to do?

It's obviously not optimal to have UI settings that do nothing.

Options:

• We could add the wrappers back in place. This would make the grid no longer responsive. I'm personally hoping this doesn't happen.
• We could continue as is, and deprecate those options in Drupal 10 and fix #3151553: Make core views grid styles responsive.
• ??

Problem/Motivation

When taxonomy terms were converted to be revisionable in #2880149: Convert taxonomy terms to be revisionable we disabled Content Moderation integration because the UI was not ready yet.

Proposed resolution

After #2899923: UI for publishing/unpublishing taxonomy terms lands, we should re-enable CM integration.

Remaining tasks

Do it :)

User interface changes

Nope.

API changes

Nope.

Data model changes

Nope.

Release notes snippet

Nope.

Problem/Motivation

The log-in pages for my Drupal sites are being indexed by google. I'm not sure if this is due to recent changes in how google handles the robots file, or spiders sites, but it seems I am not alone in this problem: https://www.drupal.org/forum/general/general-discussion/2012-06-27/is-it...

Steps to reproduce

1) Google the domian name of one of your live sites
2) see if Google recommends the Log In page in the list of pages beneath the search result for the home page

Proposed resolution

Remove trailing slashes from the robots.txt file (or, add alternatives that don't include the trailing slashes)

Remaining tasks

update the robots.txt file

User interface changes

none

API changes

none

Data model changes

none

Release notes snippet

TBD

After upgrading from 8.9.16 to 8.9.17 site is broken with 503 error.
PHP is php-7.4.21 with fpm

If I move the site to PHP 7.2.34 then it works.
Site was allready running php7.4 when it was 8.9.16

I have seg faults on messages and sometimes I get these errors in fpm:

[22-Jul-2021 15:04:56] PHP Fatal error: Uncaught InvalidArgumentException: Invalid database prefix: navigate in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php:85
Stack trace:
#0 /var/www/html/drupal/core/lib/Drupal/Core/DrupalKernel.php(1028): Drupal\Core\Test\TestDatabase->__construct()
#1 /var/www/html/drupal/core/lib/Drupal/Core/DrupalKernel.php(695): Drupal\Core\DrupalKernel::bootEnvironment()
#2 /var/www/html/drupal/index.php(19): Drupal\Core\DrupalKernel->handle()
#3 {main}
thrown in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php on line 85
[22-Jul-2021 15:04:57] PHP Fatal error: Uncaught InvalidArgumentException: Invalid database prefix: ?1 in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php:85
Stack trace:
#0 /var/www/html/drupal/core/lib/Drupal/Core/DrupalKernel.php(1028): Drupal\Core\Test\TestDatabase->__construct()
#1 /var/www/html/drupal/core/lib/Drupal/Core/DrupalKernel.php(695): Drupal\Core\DrupalKernel::bootEnvironment()
#2 /var/www/html/drupal/index.php(19): Drupal\Core\DrupalKernel->handle()
#3 {main}
thrown in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php on line 85
[22-Jul-2021 15:05:08] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 137438953444 bytes) in /var/www/html/drupal/core/includes/bootstrap.inc on line 1106

The database prefix error message sometimes list strange things like:

[21-Jul-2021 21:20:01] PHP Fatal error: Uncaught InvalidArgumentException:Invalid database prefix: Wget/1.14 (linux-gnu) in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php:85
[21-Jul-2021 21:22:20] PHP Fatal error: Uncaught InvalidArgumentException:Invalid database prefix: www.example.com in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php:85
[22-Jul-2021 15:04:56] PHP Fatal error: Uncaught InvalidArgumentException:Invalid database prefix: navigate in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php:85
[22-Jul-2021 15:04:57] PHP Fatal error: Uncaught InvalidArgumentException:Invalid database prefix: ?1 in /var/www/html/drupal/core/lib/Drupal/Core/Test/TestDatabase.php:85

the first load of the site works with no errors (full refresh on browser)
Subsequent load on whatever page get the errors, usually only:

[22-Jul-2021 15:07:43 Europe/Athens] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 137438953444 bytes) in /var/www/html/drupal/core/includes/bootstrap.inc on line 1106

No errors on drupal error log page.

I have an exact server (OS) with same php and drupal but different modules with no errors.

The major difference between those two, is that the site that is broken
has it's modules under sites/all/modules and not under modules/

It's super frustrating when you want to have a text field (255chars) but the only way to see the entire 255 chars in that field is a really really reeeeally long textfield. Text areas are a thing. We should be able to use them (and not just for longtext text fields).

in Drupal 7, to use a textarea you need to create a whole new field and migrate all your data into it, but the underlying data structure is exactly the same!! This should not be necessary. Let's add the textarea widget as an option for text fields (but leave the existing widget as default)

Hello Everyone,

I installed drupal 8.9.16. Installation process is completed successfully but when i tried to login with admin details I got error "
TypeError: Argument 1 passed to Drupal\Core\Render\Renderer::doTrustedCallback() must be callable, array given, called in /var/www/html/docroot/core/lib/Drupal/Core/Render/Renderer.php on line 781 in Drupal\Core\Render\Renderer->doTrustedCallback() (line 51 of core/lib/Drupal/Core/Security/DoTrustedCallbackTrait.php)."

Please anyone can help to resolve this issue.

Thanks

Problem/Motivation

There is hardly any point maintaining a dragon of a regular expression when Symfony already does the work for us.

Proposed resolution

Just use the regexp in Symfony\Component\Validator\Constraints\UrlValidator::PATTERN

Remaining tasks

Investigate the six remaining failures.

User interface changes

API changes

Data model changes

Release notes snippet

Schema fails to validate.

For example, install the config_inspector module and run "drush inspect_config"

system.rss:channel missing schema

system.rss:items.limit missing schema

views.settings:ui.show.master_display missing schema

The attached patch fixes it.

Problem/Motivation

While playing with lazy loading attribute that is coming from the core and our own implementation in #3060605: Support lazy loading of images @sasanikolic noticed that responsive image tag is missing width and height attribute. This bothered me because core lazy loading implementation will not work for these cases at all. Why the core is not setting these attributes for lazy image loading which basically disable lazy loading for all responsive images?

I've read quite a bit of info related to lazy loading and images and responsive images in next resources:

• https://www.smashingmagazine.com/2020/03/setting-height-width-images-imp...
• https://github.com/whatwg/html/pull/4952
• https://bugs.chromium.org/p/chromium/issues/detail?id=979891
• https://bugzilla.mozilla.org/show_bug.cgi?id=1547231
• https://bugs.webkit.org/show_bug.cgi?id=201641

In short, at least as I understand, the problem with lazy loading and responsive images was that in one moment browsers were not properly supporting width/height attributes when images were lazy-loaded. So the browsers were not able to calculate the aspect ratio of images and show image area while image was still loading - and this resulted in recalculating page layout on image loaded. However, this should be fixed in most or all browsers now and this combination should work.

Steps to reproduce

Create an image field with the responsive display, check HTML and you will see that width and height attributes are missing.

Proposed resolution

Add width and height attributes to the responsive image tag.

Remaining tasks

Test picture tag implementation.

User interface changes

None.

API changes

None.

Data model changes

None.