[PATCH] render: Avoid infinite loops in alpha map handling (#23581)

[Soeren Sandmann]

Keith Packard <keithp at keithp.com> writes:

> > in which I suggest to simply BadMatch out if you try to set a
> > picture as its own alpha map or set an alpha map that already have
> > an alpha map.
> 
> Setting a picture as its own alpha map seems well defined (the same as
> None).

It's not really well defined if the alpha origin is different from 
(0, 0), though.

> The problem with the other case is that you can *later on* change
> the picture used as an alpha map to add an alpha map there.

Can we simply say that if an image has an alphamap with an alphamap,
then you get BadMatch from Composite and the other Render ops?

> > The protocol should possibly also say that you get unpredictable
> > rendering if a picture and its alpha map point to the same underlying
> > drawable. (I don't think pixman will crash in that case, and
> > acceleration code ignores alpha maps generally).
> 
> Yeah, just making the self-referential case the same as None should fix
> that. 

The problem is if the underlying *drawable* is the same. When storing,
the RGB channels will first be written to the drawable, and then the
alpha channel will be written, which will overwrite the RGB
information. (Since pixman doesn't guarantee that x channels will be
left alone for performance reasons).

If the alpha map has a non-zero origin relative to the picture, the
behavior gets even more unpredictable.



Soren