[Spice-devel] [PATCH xf86-video-qxl master 1/2] limit calculated virtual size to fit within the framebuffer

[Hans de Goede]

When running inside a vm with a qxl device with a 8MB framebuffer the
code, for automatically setting the virtualsize to the largest
width and height seen in the resolution list, would lead to a too
large virtual size. The 8MB list has both 1920x1080 and 1600x1200
resolution which lead to a virtualsize of 1920x1200 which does not fit.
---
 src/qxl_driver.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/src/qxl_driver.c b/src/qxl_driver.c
index cdf4843..f9a6647 100644
--- a/src/qxl_driver.c
+++ b/src/qxl_driver.c
@@ -1360,6 +1360,12 @@ qxl_pre_init(ScrnInfoPtr pScrn, int flags)
                 max_y = qxl->modes[i].y_res;
         }
 
+    /* It is possible for the largest x + largest y size combined leading
+       to a virtual size which will not fit into the framebuffer when this
+       happens we prefer max width and make height as large as possible */
+    if (max_x * max_y * (pScrn->bitsPerPixel / 8) > qxl->draw_area_size)
+        max_y = qxl->draw_area_size / (max_x * (pScrn->bitsPerPixel / 8));
+
     if (pScrn->display->virtualX == 0 && pScrn->display->virtualY == 0) {
     	pScrn->display->virtualX = max_x;
     	pScrn->display->virtualY = max_y;
-- 
1.7.1