In #1043198: Convert view modes to ConfigEntity we attempted to add a UI for the view modes as well, but there was no consensus on how the UI should work.
#1043198-36: Convert view modes to ConfigEntity shows how I attempted to build this listing (modeled after Display Suite).
#1043198-40: Convert view modes to ConfigEntity shows how Entity View Mode displays the listing.
Entity View Mode's approach actually matches the structure of the config entities better, and is similar to how our other listings have turned out since then.
So I will implement the form and list controllers in the same vein as core, and hopefully finish this once and for all.