Problem/Motivation
As we added ViewMode template suggestions for Nodes here #1503464: Automatically add theme hook suggestion for node view mode and most probably for all entities: #2270883: Automatically add theme hook suggestions for all entity types.
We should also have suggestions for Fields.
Also because you can configure your Field Formatter per Viewmode, I think this is necessary.
Proposed resolution
Add Theme Hook Suggestions for Fields with View Modes
Remaining tasks
Write Patch