Routes are being cached with no parameters (ex: 404 routes)

Drupal's Routing/RouteProvider caches all incoming request's routes and parameters for future use of course. This happens after all other processes are done and after all processInbounds (most importantly from the content POV).

The case: there's a link on the current page to a place that isn't accessible from that page's context - could be language, could be permissions, domain - just a link to node that's not supposed to be there and restricted by some logic - anything will do as long as it's entity reference.

When the page renders, Drupal looks for the routes and controllers responsible for this link (most importantly it should be entity reference) - and can't find them in the current context (let's take different language context and using the link field).

What happens - it can't find the routes and controllers for this reference (in the current language since the linked node is in different language) but still caches the empty route params for this "unaccessible" path. If you hit that path it should work as expected but it don't since it's already cached as 404, so it starts throwing 404.

The inconsistent part is that if you first hit the real thing - it's going to be cached correctly and later won't break. Unfortunately that's a pretty big "IF" and not very consistent and logical.

My solution was to add a patch in the core/lib/Drupal/Core/Routing/RouteProvider.php to prevent empty routes being cached.

Here is an example of broken (empty) cached route:

a:3:{s:4:"path";s:66:"/patients/prevention/plant-products/cancer-prevention-sulforaphane";s:5:"query";a:0:{}s:6:"routes";O:41:"Symfony\Component\Routing\RouteCollection":2:{s:49:" Symfony\Component\Routing\RouteCollection routes";a:0:{}s:52:" Symfony\Component\Routing\RouteCollection resources";a:0:{}}}

Readable:

Array
(
    [path] => /patients/prevention/plant-products/cancer-prevention-sulforaphane
    [query] => Array()
    [routes] => __PHP_Incomplete_Class Object
        (
            [__PHP_Incomplete_Class_Name] => Symfony\Component\Routing\RouteCollection
            [ Symfony\Component\Routing\RouteCollection routes] => Array()
            [ Symfony\Component\Routing\RouteCollection resources] => Array()
        )
)

The questions here are:
1. how big of an impact is it to not cache empty routes?
2. why does Drupal cache them at all?
3. is there a better way/place than what I found/did to fix this issue?