The ConfigFactory class has a simple static cache that keeps track of the Config objects that it handles. This static cache does not have awareness of the cacheability metadata of the items it holds. When for example some of its items have their cache tags invalidated these are not removed from the cache, and the ConfigFactory will return stale items.
We can fix this by replacing the simple static cache with a @cache.static memory backend.