Redirection through a route subscriber redirects to home page, not specified URL (only on browser, not command-line curl)

Problem/Motivation

Redirection through a route subscriber redirects to home page, not specified URL (only on browser, not command-line curl)

Steps to reproduce

To reproduce, create a module called my_custom_module which contains, in addition to the info.yml file:

./my_custom_module.routing.yml

my_custom_module.redirect-without-route-subscriber:
  path: '/redirect-without-route-subscriber'
  defaults: { _controller: '\Drupal\my_custom_module\Controller\RedirectController::load' }
  requirements:
    _access: 'TRUE'

./my_custom_module.services.yml

services:
  mymodule.route_subscriber:
    class: Drupal\my_custom_module\Routing\RouteSubscriber
    tags:
      - { name: event_subscriber }

./src/Routing/RouteSubscriber.php

namespace Drupal\my_custom_module\Routing;

use Drupal\Core\Routing\RouteSubscriberBase;
use Symfony\Component\Routing\RouteCollection;

class RouteSubscriber extends RouteSubscriberBase {

  /**
   * {@inheritdoc}
   */
  protected function alterRoutes(RouteCollection $collection) {
    // Check whether the route is a 404 route, if it is, we will get a
    // Route object, otherwise we will get NULL and can move on.
    $route = $collection->get('system.404');
    if ($route) {
      $route->setDefault('_controller', '\Drupal\my_custom_module\Controller\RedirectController::load');
    }
  }

}

./src/Controller/RedirectController.php

namespace Drupal\my_custom_module\Controller;

use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Cache\CacheableMetadata;
use Drupal\Core\Routing\TrustedRedirectResponse;

/**
 * Dummy redirect controller.
 */
class RedirectController extends ControllerBase {

  public function load() {
    $metadata = (new CacheableMetadata())
      ->setCacheMaxAge(0);

    return (new TrustedRedirectResponse('https://duckduckgo.com', 301))
      ->addCacheableDependency($metadata);
  }

}

The goad of this module is to redirect all "404 Not Found" pages to an extenal site (in this example https://duckduckgo.com).

Now enable the module and go to /redirect-without-route-subscriber

This will redirect you to "https://duckduckgo.com", as expected.

However, if on a browser (I used a browser I haven't used in months to avoid caching issues) you go to /this/path/is/not/found, one would expect to be redirected to 'https://duckduckgo.com', however, we are redirected to the home page.

But using CURL, this works fine:

$ curl -I http://0.0.0.0:49238/this/is/not/found
HTTP/1.1 301 Moved Permanently
Date: Sat, 30 Oct 2021 15:11:27 GMT
Server: Apache/2.4.38 (Debian)
X-Powered-By: PHP/8.0.12
Location: https://duckduckgo.com
X-Drupal-Dynamic-Cache: UNCACHEABLE
X-UA-Compatible: IE=edge
Content-language: en
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Permissions-Policy: interest-cohort=()
Cache-Control: must-revalidate, no-cache, private
Expires: Sun, 19 Nov 1978 05:00:00 GMT
X-Generator: Drupal 9 (https://www.drupal.org)
X-Drupal-Cache: MISS
Content-Type: text/html; charset=UTF-8

Here is my system information:

root@fa814eb36b0d:/var/www/html# drush status
 Drupal version   : 9.2.7                                      
 Site URI         : http://default                             
 DB driver        : mysql                                      
 DB hostname      : mysql                                      
 DB port          : 3306                                       
 DB username      : root                                       
 DB name          : drupal                                     
 Database         : Connected                                  
 Drupal bootstrap : Successful                                 
 Default theme    : my_custom_theme                            
 Admin theme      : seven                                      
 PHP binary       : /usr/local/bin/php                         
 PHP config       : /usr/local/etc/php/php.ini                 
 PHP OS           : Linux                                      
 Drush script     : /var/www/html/vendor/drush/drush/drush     
 Drush version    : 10.6.1                                     
 Drush temp       : /tmp                                       
 Drush configs    : /var/www/html/vendor/drush/drush/drush.yml 
 Install profile  : standard                                   
 Drupal root      : /var/www/html/.                            
 Site path        : sites/default                              
 Files, Public    : sites/default/files                        
 Files, Private   : /drupal-private-files                      
 Files, Temp      : /tmp                                       
root@fa814eb36b0d:/var/www/html# php -v
PHP 8.0.12 (cli) (built: Oct 21 2021 15:52:30) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.12, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.12, Copyright (c), by Zend Technologies

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet