DisplayPluginBase ignores view.settings:skip_cache unless the view is being edited.

    $skip_cache = \Drupal::config('views.settings')->get('skip_cache');

    if (empty($view->editing) || !$skip_cache) {
        // [caching code]
    }
    else {
        // ...
    }

The first branch (the caching) is only skipped if skip_cache is non-empty AND $view->editing is true. If I'm reading the code right, it looks as though either of these conditions should skip the cache, and therefore that || should be a &&.