Nested contextual links are always hidden at first page load

Problem/Motivation

I enabled core modules:
- Toolbar
- Contextual Links

I generated a custom content entity with Drupal Console, added a `my_module.links.contextual.yml` file and added `{{ title_suffix }}` to the entity template.
So far so good. Then, at first page load, if toolbar icon is already active, all blocks and nodes contextual links are visible, but not the ones for my custom entity.
The button is there but not visible, because class `.visually-hidden` is still present on the element.

If I disable all links and enable again (by clicking on the toolbar icon), even the links for the custom entity are visible.
This is due to `adjustIfNestedAndOverlapping()` function called on `contextual.js`, which removes the class and hides nested contextual links anyhow, after adjusting the position.

$trigger.removeClass('visually-hidden');
height = $nestedContextual.height();
$trigger.addClass('visually-hidden');

Proposed resolution

It's necessary to remove the class only if the state of contextual links is not visible.