[Intel-gfx] [PATCH 18/30] drm/i915: Add an interface to dynamically change the cache level

[Daniel Vetter]

> @@ -3002,6 +3002,44 @@ i915_gem_object_set_to_gtt_domain(struct drm_i915_gem_object *obj, bool write)
>  	return 0;
>  }
>  
> +int i915_gem_object_set_cache_level(struct drm_i915_gem_object *obj,
> +				    enum i915_cache_level cache_level)
> +{
> +	int ret;
> +
> +	if (obj->cache_level == cache_level)
> +		return 0;
> +
> +	if (obj->gtt_space) {
> +		ret = i915_gem_object_flush_gpu(obj);
> +		if (ret)
> +			return ret;
> +
> +		ret = i915_gem_gtt_bind_object(obj, cache_level);
> +		if (ret)
> +			return ret;

This momentarily confused me till I've noticed that the fake agp driver
does the right thing and does not re-create a dmar mapping if it already
exists. So much for remembering my own code. Still, maybe extract
i915_gem_gtt_rebind_object from restore_gtt_mappings and use that one
here? Should make the intent clearer.

> +		/* Ensure that we invalidate the GPU's caches and TLBs. */
> +		obj->base.read_domains &= I915_GEM_GPU_DOMAINS;

I can't make sense of this. Either we really want to ensure that the gpu
buffers get invalidated on next use. But then it's probably

		read_domains &= ~GPU_DOMAINS

and would fit better grouped together with the call to object_flush_gpu
(the rebind can't actually fail if the dmar mappings already exist). Or
this is something else and I'm blind.

> +	}
> +
> +	if (cache_level == I915_CACHE_NONE) {
> +		/* If we're coming from LLC cached, then we haven't
> +		 * actually been tracking whether the data is in the
> +		 * CPU cache or not, since we only allow one bit set
> +		 * in obj->write_domain and have been skipping the clflushes.
> +		 * Just set it to the CPU cache for now.
> +		 */
> +		WARN_ON(obj->base.write_domain & I915_GEM_GPU_DOMAINS);
> +
> +		obj->base.read_domains |= I915_GEM_DOMAIN_CPU;

This breaks the invariant that write_domain != 0 implies write_domain ==
read_domains. Yes, if nothing prefetches and we clflush in due time the
caches should still be valid, but paranoid me deems that a bit fragile.
Also future patches shoot down fences, so we might as well shoot down the
gtt mapping completely. That seems required for the redirect gtt mappings
patch, too.

> +		obj->base.write_domain = I915_GEM_DOMAIN_CPU;

We might end up here with a write_domain == DOMAIN_GTT. Feels a tad bit
unsafe. I'd prefer either a WARN_ON and push the problem out to callers or
to call flush_gtt_write_domain somewhere in set_cache_level.

This looks like the critical part of the whole patch series so perhaps
fold the follow-up patches in here, too (like fence teardown). This way
there's just one spot that requires _really_ careful thinking.

Also, I haven't thought too hard about the uncached->cached transition on
live objects, which is not (yet) required. Maybe some more careful
handling of the gtt domain (mappings teardown) is needed for that.
-Daniel
-- 
Daniel Vetter
Mail: daniel at ffwll.ch
Mobile: +41 (0)79 365 57 48