[PATCH] fullscreen-shell: Fix modeset on transformed outputs

[Bryce Harrington]

On Thu, Aug 06, 2015 at 01:17:38PM -0500, Derek Foreman wrote:
> As this is the oldest patch in patchwork, I figured I'd test it out and
> see if we can land it.  :)

Great idea!

> On 08/01/15 10:57 AM, Jason Ekstrand wrote:
> > Previously, we blindly created a mode for the output based on surface size
> > and completely ignoring the output transform.  This caused modesets to fail
> > on outputs that were transformed by 90 or 270 degrees.  We should be
> > swapping the width and the height in this case.
> > ---
> >  fullscreen-shell/fullscreen-shell.c | 23 +++++++++++++++++++++--
> >  1 file changed, 21 insertions(+), 2 deletions(-)
> > 
> > diff --git a/fullscreen-shell/fullscreen-shell.c b/fullscreen-shell/fullscreen-shell.c
> > index 35e6d8f..5fd7cd6 100644
> > --- a/fullscreen-shell/fullscreen-shell.c
> > +++ b/fullscreen-shell/fullscreen-shell.c
> > @@ -463,9 +463,28 @@ fs_output_configure_for_mode(struct fs_output *fsout,
> >  					&surf_x, &surf_y,
> >  					&surf_width, &surf_height);
> >  
> > +	/* The actual output mode is in physical units.  We need to
> > +	 * transform the surface size to physical unit size by flipping ans
> > +	 * possibly scaling it.
> > +	 */
> 
> ans -> and
> 
> > +	switch (fsout->output->transform) {
> > +	case WL_OUTPUT_TRANSFORM_90:
> > +	case WL_OUTPUT_TRANSFORM_FLIPPED_90:
> > +	case WL_OUTPUT_TRANSFORM_270:
> > +	case WL_OUTPUT_TRANSFORM_FLIPPED_270:
> > +		mode.width = surf_height * fsout->output->native_scale;
> > +		mode.height = surf_width * fsout->output->native_scale;
> > +		break;
> > +
> > +	case WL_OUTPUT_TRANSFORM_NORMAL:
> > +	case WL_OUTPUT_TRANSFORM_FLIPPED:
> > +	case WL_OUTPUT_TRANSFORM_180:
> > +	case WL_OUTPUT_TRANSFORM_FLIPPED_180:
> > +	default:
> > +		mode.width = surf_width * fsout->output->native_scale;
> > +		mode.height = surf_height * fsout->output->native_scale;
> > +	}
> >  	mode.flags = 0;
> > -	mode.width = surf_width * fsout->output->native_scale;
> > -	mode.height = surf_height * fsout->output->native_scale;
> >  	mode.refresh = fsout->pending.framerate;
> >  
> >  	ret = weston_output_mode_switch_to_temporary(fsout->output, &mode,
> > 
> 
> This looks correct to me - I hope whoever lands it doesn't mind fixing
> the comment typo.  It still applies, just offset a few lines. :)

I've fixed the typo.
 
> I've tested this by changing the fullscreen client to have a 480x640
> surface size and confirmed that it will let me switch an output to
> 640x480 when the test surface is that size and the output is rotated 90
> degrees.

Smells like a unit test case?  :-)

> It also no longer allows mode switching when the surface size matches
> the unrotated output size.
> 
> Reviewed-by: Derek Foreman <derekf at osg.samsung.com>
> Tested-by: Derek Foreman <derekf at osg.samsung.com>

Thanks, pushed with the typo fix:
To ssh://git.freedesktop.org/git/wayland/weston
   87c862a..58106d7  master -> master

Bryce