Aggregation settings are not removed from fields when turned off

When creating a view, if 'use aggregation' has been turned on and aggregation settings set for a field, the field remains aggregated even after 'use aggregation' has been turned off.

Steps to reproduce

1. Create a new view displaying fields
2. Turn on aggregation for the view display under advanced settings
3. Include a field, e.g. NID or Title and set your aggregation settings for the field, e.g. count.
4. Verify that your view preview is showing an aggregate result
5. Turn off aggregation under advanced settings
6. Verify that the preview is still showing an aggregate result.

1. And for bonus points...
2. Remove the field, clear caches etc.
3. Add in the same field again.
4. Verify that the preview is still showing an aggregate result.

Expected Behaviour

Expected behaviour is that when 'use aggregation' is turned off, the field goes back to behaving default, non-aggregate behaviour.

What Happened Instead

Instead, the field retained its aggregate settings.