Found when debugging some fails in #1969728: Implement Field API "field types" as TypedData Plugins.
When an uploaded file is configured not to be displayed, file_field_prepare_view() simply deletes it from the list of values.
This affects the entity for the rest of the request.
That's not how it should work, items that are supposed to remain hidden should stay out of the way, but they cannot be simply removed.