Responsive images (img with srcset and sizes attributes) are scaled up beyond the pixel size. Only CSS should ever upscale images like this, and the HTML should restrict itself to choosing the image file and setting the pixel density, but the specification we've got is the one the browsers are implementing, so it seems we have to live with it.
As the Mozilla documentation puts it, "selected source size affects the intrinsic size of the image". The sizes attribute is required when srcset is defined (or else it defaults to 100vw).
This post covers the phenomenon well, and this self-answered question provides probably the best solution for Drupal.
If a responsive image style set is built with an option to prevent making images larger than they really are (separate from the option in the image style itself, regarding resizing powered by a graphics processing library) then Drupal should do the introspection necessary to drop off from the responsive image output the larger-than-the-original-image srcset options.