It would make sense if we could just move all the contextual links related functions/hooks into views_ui module, as in theory we shouldn't really need views contextual links if views UI is turned off.
Let's at least explore this, might turn out we definitely can't do this. Would be nice though.