Steps to reproduce:
1. Enable page caching.
2. Visit the front page as an anonymous user to get it into the page cache.
3. Use the Ban module to ban that user's IP address.
4. Visit the front page as that user again. Instead of being banned, you will still be able to see the cached version of the front page.
This is a change in behavior from Drupal 7, where the IP address would be banned.
For several reasons, one could probably make the argument that this behavior change is OK. However, at a minimum it should be documented, since it's certainly unexpected. And regardless of which behavior is the correct one, there should be a test for it - I have the impression this changed as an unintended side effect of #1794754: Convert ban_boot() to an event listener...