When managing default layouts for view mode of an entity type, users must navigate to Admin -> Structure -> Content Types -> [Node Type] -> Manage Display -> Manage Layout.
When managing a layout override for an individual entity, users must simply click the "Layout" tab alongside the existing View/Edit tabs.
In Panelizer world (at least in D7), the action for managing the layout of a page is nearly identical if you're modifying the default layout or an override. The only difference in the UI is that the button is either "Save as Default" or "Save as Override".
Something like this could work I think but is still not great:
1) When viewing an entity that does not have overrides enabled, add a "Layout" tab and link it to the page for managing the layout for the entire view mode.
2) When viewing an entity that does have overrides enabled, have a tabs "Layout - Override" and "Layout - Default"
My use case: I have a "Landing Page" content type that allows overrides (because every node of this type will always have its own unique layout). I also have a "News" content type, that does not allow overrides. I still want site builders to be able to manage the layout for news articles, but I don't want to confuse them by offering two very different ways of accessing the layout page.