[PATCH] render: Don't generate invalid pixman format when using a 24bpp framebuffer with a 32bit depth visual.

[Robert Ancell]

Some background on this patch:

I've been trying to investigate why gnome-terminal doesn't render
correctly when running Ubuntu 14.04 LTS inside qemu (on Ubuntu 14.10).
The window is compressed in the X axis and has transparency (seems to
be a buffer somewhere that is being written in RGB and read as RGBA).

The issue seems to be GTK+ applications using the RGBA visual when
qemu/X/fb is providing a depth 24 root window. Playing around with
test cases showed the assertion mentioned in the patch. I'm not sure
if fb should be allowed to have a depth greater than the bpp and if
the bug is actually there but this patch allows correct rendering in
my test case now. I guess the Picture is being converted from RGBA to
RGB when it hits the root window?

Note this patch doesn't fix the gnome-terminal rendering issue, so
I'll continue down the rabbit hole. Any suggestions as to the causes
of this are greatly appreciated.

Attached is a program that doesn't render on the current master and
does with the applied patch.

--Robert

On Fri, Jun 6, 2014 at 4:36 PM, Robert Ancell
<robert.ancell at canonical.com> wrote:
> When using the fb backend at 24bpp it allows a visual with 32 bit depth.
> This would cause RENDER to try and create an invalid pixman buffer and hit the
> following assertion when trying to render to it:
>
> *** BUG ***
> In create_bits_image_internal: The expression PIXMAN_FORMAT_BPP (format) >= PIXMAN_FORMAT_DEPTH (format) was false
> Set a breakpoint on '_pixman_log_error' to debug
>
> Fix is to ensure that the bpp is always at least as big as the drawable depth.
> ---
>  render/picture.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/render/picture.c b/render/picture.c
> index 7da9310..f411521 100644
> --- a/render/picture.c
> +++ b/render/picture.c
> @@ -762,6 +762,7 @@ CreatePicture(Picture pid,
>  {
>      PicturePtr pPicture;
>      PictureScreenPtr ps = GetPictureScreen(pDrawable->pScreen);
> +    int bpp;
>
>      pPicture = dixAllocateScreenObjectWithPrivates(pDrawable->pScreen,
>                                                     PictureRec, PRIVATE_PICTURE);
> @@ -773,7 +774,10 @@ CreatePicture(Picture pid,
>      pPicture->id = pid;
>      pPicture->pDrawable = pDrawable;
>      pPicture->pFormat = pFormat;
> -    pPicture->format = pFormat->format | (pDrawable->bitsPerPixel << 24);
> +    bpp = pDrawable->bitsPerPixel;
> +    if (bpp < pFormat->depth)
> +        bpp = BitsPerPixel (pFormat->depth);
> +    pPicture->format = pFormat->format | (bpp << 24);
>
>      /* security creation/labeling check */
>      *error = XaceHook(XACE_RESOURCE_ACCESS, client, pid, PictureType, pPicture,
> --
> 2.0.0
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test24.c
Type: text/x-csrc
Size: 2537 bytes
Desc: not available
URL: <http://lists.x.org/archives/xorg-devel/attachments/20140606/8fbbe3e8/attachment-0001.c>