The following issue is against Drupal 8. It is a known performance issue and there is a working patch that is still not in Drupal core:
http://drupal.org/node/1710656
Just before _menu_link_translate($item); in the _menu_tree_check_access() function of /includes/menu.inc:1027 we need:
// Do not load hidden menu items if not in active breadcrumb trail and
// user can't administer the menu.
if (!empty($item['hidden']) && empty($item['in_active_trail']) && !user_access('administer menu')) {
continue;
}