[Mesa-dev] [PATCH] st/vdpau: avoid sending buffers with 10bit channels to X

[Michel Dänzer]

On 19/01/17 12:27 AM, Nayan Deshmukh wrote:
> PresentPixmap only works if the pixmap depth matches the window
> depth, otherwise it returns a BadMatch protocol error.
> 
> Signed-off-by: Nayan Deshmukh <nayan26deshmukh at gmail.com>
> ---
>  src/gallium/state_trackers/vdpau/output.c        | 6 ++++++
>  src/gallium/state_trackers/vdpau/presentation.c  | 6 +++---
>  src/gallium/state_trackers/vdpau/vdpau_private.h | 1 +
>  3 files changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/src/gallium/state_trackers/vdpau/output.c b/src/gallium/state_trackers/vdpau/output.c
> index 98a8011..4466483 100644
> --- a/src/gallium/state_trackers/vdpau/output.c
> +++ b/src/gallium/state_trackers/vdpau/output.c
> @@ -75,6 +75,12 @@ vlVdpOutputSurfaceCreate(VdpDevice device,
>  
>     memset(&res_tmpl, 0, sizeof(res_tmpl));
>  
> +   if (rgba_format == VDP_RGBA_FORMAT_B10G10R10A2 ||
> +       rgba_format ==  VDP_RGBA_FORMAT_R10G10B10A2)
> +      vlsurface->send_to_X = false;
> +   else
> +      vlsurface->send_to_X = true;

This isn't sufficient: The window depth could be != 24 (in particular 16
or 15, or 30). Even if the depths match, the result won't look correctly
if the VDPAU RGB component order doesn't match the X11 one. So
technically this actually needs to check the window's visual instead of
just the depth (though in practice I think all X11 visuals use the same
RGB component order).


-- 
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer