[PATCH] fullscreen-shell: Fix modeset on transformed outputs

[Derek Foreman]

As this is the oldest patch in patchwork, I figured I'd test it out and
see if we can land it.  :)

On 08/01/15 10:57 AM, Jason Ekstrand wrote:
> Previously, we blindly created a mode for the output based on surface size
> and completely ignoring the output transform.  This caused modesets to fail
> on outputs that were transformed by 90 or 270 degrees.  We should be
> swapping the width and the height in this case.
> ---
>  fullscreen-shell/fullscreen-shell.c | 23 +++++++++++++++++++++--
>  1 file changed, 21 insertions(+), 2 deletions(-)
> 
> diff --git a/fullscreen-shell/fullscreen-shell.c b/fullscreen-shell/fullscreen-shell.c
> index 35e6d8f..5fd7cd6 100644
> --- a/fullscreen-shell/fullscreen-shell.c
> +++ b/fullscreen-shell/fullscreen-shell.c
> @@ -463,9 +463,28 @@ fs_output_configure_for_mode(struct fs_output *fsout,
>  					&surf_x, &surf_y,
>  					&surf_width, &surf_height);
>  
> +	/* The actual output mode is in physical units.  We need to
> +	 * transform the surface size to physical unit size by flipping ans
> +	 * possibly scaling it.
> +	 */

ans -> and

> +	switch (fsout->output->transform) {
> +	case WL_OUTPUT_TRANSFORM_90:
> +	case WL_OUTPUT_TRANSFORM_FLIPPED_90:
> +	case WL_OUTPUT_TRANSFORM_270:
> +	case WL_OUTPUT_TRANSFORM_FLIPPED_270:
> +		mode.width = surf_height * fsout->output->native_scale;
> +		mode.height = surf_width * fsout->output->native_scale;
> +		break;
> +
> +	case WL_OUTPUT_TRANSFORM_NORMAL:
> +	case WL_OUTPUT_TRANSFORM_FLIPPED:
> +	case WL_OUTPUT_TRANSFORM_180:
> +	case WL_OUTPUT_TRANSFORM_FLIPPED_180:
> +	default:
> +		mode.width = surf_width * fsout->output->native_scale;
> +		mode.height = surf_height * fsout->output->native_scale;
> +	}
>  	mode.flags = 0;
> -	mode.width = surf_width * fsout->output->native_scale;
> -	mode.height = surf_height * fsout->output->native_scale;
>  	mode.refresh = fsout->pending.framerate;
>  
>  	ret = weston_output_mode_switch_to_temporary(fsout->output, &mode,
> 

This looks correct to me - I hope whoever lands it doesn't mind fixing
the comment typo.  It still applies, just offset a few lines. :)

I've tested this by changing the fullscreen client to have a 480x640
surface size and confirmed that it will let me switch an output to
640x480 when the test surface is that size and the output is rotated 90
degrees.

It also no longer allows mode switching when the surface size matches
the unrotated output size.

Reviewed-by: Derek Foreman <derekf at osg.samsung.com>
Tested-by: Derek Foreman <derekf at osg.samsung.com>