[Mesa-dev] [PATCH] st/vdpau: avoid sending buffers with 10bit channels to X

[Nayan Deshmukh]

On Wed, Jan 18, 2017 at 9:21 PM, Michel Dänzer <michel at daenzer.net> wrote:
> On 19/01/17 12:27 AM, Nayan Deshmukh wrote:
>> PresentPixmap only works if the pixmap depth matches the window
>> depth, otherwise it returns a BadMatch protocol error.
>>
>> Signed-off-by: Nayan Deshmukh <nayan26deshmukh at gmail.com>
>> ---
>>  src/gallium/state_trackers/vdpau/output.c        | 6 ++++++
>>  src/gallium/state_trackers/vdpau/presentation.c  | 6 +++---
>>  src/gallium/state_trackers/vdpau/vdpau_private.h | 1 +
>>  3 files changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/src/gallium/state_trackers/vdpau/output.c b/src/gallium/state_trackers/vdpau/output.c
>> index 98a8011..4466483 100644
>> --- a/src/gallium/state_trackers/vdpau/output.c
>> +++ b/src/gallium/state_trackers/vdpau/output.c
>> @@ -75,6 +75,12 @@ vlVdpOutputSurfaceCreate(VdpDevice device,
>>
>>     memset(&res_tmpl, 0, sizeof(res_tmpl));
>>
>> +   if (rgba_format == VDP_RGBA_FORMAT_B10G10R10A2 ||
>> +       rgba_format ==  VDP_RGBA_FORMAT_R10G10B10A2)
>> +      vlsurface->send_to_X = false;
>> +   else
>> +      vlsurface->send_to_X = true;
>
> This isn't sufficient: The window depth could be != 24 (in particular 16
> or 15, or 30). Even if the depths match, the result won't look correctly
In case the depth is not 24 then vl_winsys_dri3.c will return null and
it will follow the older code path.

> if the VDPAU RGB component order doesn't match the X11 one. So
> technically this actually needs to check the window's visual instead of
> just the depth (though in practice I think all X11 visuals use the same
> RGB component order).
>
What is the order that they use?

If they practically use the same order it will be easier to encode it instead
of checking it every time.

Regards,
Nayan

>
> --
> Earthling Michel Dänzer               |               http://www.amd.com
> Libre software enthusiast             |             Mesa and X developer