Problem/Motivation
Fix robots.txt to allow search engines access to CSS, JavaScript and image files
Steps to reproduce
View the robots.txt which has allowed css, js files , images from core and profiles folder but the same is not the case for for modules and themes folder
Proposed resolution
To allow css, js and images under modules and themes folder.
#modules
Allow: /modules/*.css$
Allow: /modules/*.css?
Allow: /modules/*.js$
Allow: /modules/*.js?
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /modules/*.svg
#themes
Allow: /themes/*.css$
Allow: /themes/*.css?
Allow: /themes/*.js$
Allow: /themes/*.js?
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
Allow: /themes/*.svg
Remaining tasks
User interface changes
Introduced terminology
API changes
Data model changes
Release notes snippet
Similar issue was fixed in Drupal 7 : https://www.drupal.org/project/drupal/issues/2364343
But it also did not include sites/all/modules or sites/all/themes (as is the path for core and contrib modules/themes in drupal 7)
Was that done for security reasons ?