Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

Thu Feb 29 13:39:28 UTC 2024

On Wed, 28 Feb 2024 18:04:28 +0000
Terry Barnaby <terry1 at beam.ltd.uk> wrote:

> Hi Pekka,
> 
> Some questions below:
> 
> Thanks
> 
> Terry
> On 26/02/2024 15:56, Pekka Paalanen wrote:
> > Ok. What Wayland API requests cause a surface to actually be mapped  
> >> (Sorry don't really know Wayland protocol) ?  
> > Hi Terry,
> >
> > the basic protocol object is wl_surface. The wl_surface needs to be
> > given a "role". This is a specific protocol term. xdg_surface and
> > xdg_toplevel can give a wl_surface the role of a top-level window,
> > which means it can get mapped when you play by the rules set in the
> > xdg_toplevel specification.
> >
> > Sub-surface is another role.
> >
> > So the rules are always role specific, but at some point they all
> > require content on the wl_surface, which is given with the attach
> > request followed by a commit request. Role rules specify when and how
> > that can be done.  
> 
> Yes, I have heard that. But what I don't knoe is from the client:
> 
>  1. How do I find out the surfaces role ?

It is what you (or Qt, or Gst) set it to. There is no way to query it
(or any other thing) back by Wayland.

If you look at a protocol dump (e.g. WAYLAND_DEBUG=client in
environment), you can could follow the protocol messages and trace back
what the role was set to.

>  2. How would I set the surface to have a role such that it would be
>     mapped and thus visible ? Just wondering if I can work around what I
>     think is a QtWayland bug/issue/feature to make sure by second
>     Widgets surface is mapped/visible so that the waylandsink subsurface
>     can work. With X11 there were API calls to change the Windows state
>     and I was looking for something similar with Wayland.

There is no simple answer to this. You pick a role you need, and then
play by the protocol spec.

You do not have any surfaces without roles, though, so this would not
help you anyway. Roles cannot be changed, only set once per wl_surface
life time. Sub-surface is a role.

> 
> I need to find some way to actually display video, simply and 
> efficiently on an embedded platform, in a Qt application in the year 2024 :)
> 
> I have tried lots of work arounds but none have worked due to either Qt 
> issues, Wayland restrictions, Gstreamer restrictions, Weston 
> issues/restrictions, NXP hardware engine issues/restrictions etc. Any 
> ideas gratefully received!
> 

Did you try making the "middle" QWidget *not* have a wl_surface of its
own?

> >  
> >> The higher level surfaces are created/managed by QtWayland, but maybe I
> >> can override somehow.
> >>  
> > That does not feel like a good idea to me. But I also cannot really
> > help you, because this all seems to be pointing at Qt which I know
> > nothing about.  
> 
> Yes, thats probably true. But I need to get this to work somehow, even 
> if a kludge for now.

Hack on Qt, then? Sorry, but I don't understand this insistence that
what sounds like a Qt bug must be workaround-able via Wayland.

> >  
> >>>     
> >>>> As mentioned before, If I use QPainter to draw into the video QWidget it
> >>>> actually draws into the top QWidgets 16 surface using Wayland protocol.
> >>>> I would have thought it would draw into its own QWidget surface 18 as it
> >>>> has Qt::WA_NativeWindow set ?  
> > This question seems to be the essence. If Qt worked like you expected,
> > then I think the whole program would work.
> >
> > However, there is no need (from Wayland perspective) to have a
> > wl_surface as "surface 18" in the middle. What would be best is if you
> > could somehow have that "middle" QWidget but without it's own
> > wl_surface. Have the QWidget control the GStreamer wl_surface position
> > through wl_subsurface interface, while GStreamer plays the video
> > through wl_surface interface.
> >
> > Wayland does not relay sub-surface resizing or positioning between two
> > client-side components at all. There is not even a way query a
> > sub-surface position. So the positioning and sizing is all done in
> > Qt<->GStreamer somehow without Wayland in between. Only the end result
> > gets sent over Wayland to display: Qt sets up the position and
> > GStreamer sets up the size and content.  
> 
> I think this middle surface is needed so that Qt can manage the 
> "Windows" at this level, like raise, lower, resize et. and Wayland 

Hmm, that does not sound right to me, but then again, I don't know Qt.

Wayland certainly does not impose such demand.

> sink's subsurface that is below this is separate and can be de-synced 
> for the video display etc. I normally (on X11 and with Qt5/Wayland), 
> respond to QWidget resizes and use the Gstreamer API calls to 
> position/resize the waylandsink's sub-surface. This all works quite 
> nicely under X11 and worked (although not nicely) under Qt5/Wayland.
> 
> >  
> >>>> I assume that Qtwayland is not "activating" the video QWidget's surface
> >>>> or using it for some reason (send subsurface expose events ?) ?
> >>>>     
> >>> If that's true, then it is very relevant. A sub-surface becomes mapped
> >>> and visible when:
> >>>
> >>> - its parent surface is mapped and visible, and
> >>>
> >>> - the parent surface is committed after the sub-surface has been
> >>>     created and associated, and
> >>>
> >>> - if the sub-surface is in synchronized mode, there also needs to be a
> >>>     parent surface commit after every sub-surface commit you want to
> >>>     become visible. So if you do the first sub-surface sync-mode commit
> >>>     with a buffer after the parent has already committed the
> >>>     sub-surface's creation, the parent surface needs too commit again.
> >>>
> >>> This applies recursively, too, and you have a sub-sub-surface there.
> >>>
> >>> Do you actually need to sub-surface in the middle? Have you tried
> >>> without it?  
> >> I am not doing anything with Wayland directly. Qt is managing the higher
> >> level surfaces/subsurfaces and then GStreamers waylandsink is passed one
> >> of these Qt managed surfaces and it creates the subsurface 44. Looking
> >> at waylandsink it should set this subsurface to be desynced so it can
> >> draw into this surface without synchronising to the parents surface
> >> managed by Qt.  
> > Right, and desync is not enough if its parent is not mapped.
> >  
> >> All I am trying to do is use the technique as mentioned in various
> >> forums/lists to get GStreamer to display a video such that it "appears"
> >> where a QWidget is on the screen. Mind you all of this info is very
> >> rough and ready. For X11 it appears stable, but for Qt/Wayland the info,
> >> and it appears functionality, is not quite all there.
> >>
> >> When you say a sub-surface in the middle I presume you mean the surface
> >> 18 of the lower QWidget ? Well I have tried using the top most QWidget's
> >> surface 16 and the video is displayed, although all over the
> >> application. I really need to manage this surface so it can be raised,
> >> lowered and resizes amongst the other QWidgets somehow. I have tried
> >> using direct Wayland API calls to create a subsurface manually from the
> >> top surface but so far I have just got protocol errors while trying
> >> this. It may be my bad Wayland client code or how it is interfering with
> >> Qt's Wayland interface.
> >>
> >> I have even tried using a separate top level surface. Unfortunately as
> >> the standard Wayland protocols do not allow an application to move or
> >> manage top level Windows this is not useful. I guess I could create an
> >> extra Wayland/Weston server protocol to allow my app (embedded system)
> >> to manage these windows, but I was trying to do things in as standard a
> >> way as possible.  
> > Rather than "more Wayland", I suspect you should look into "more Qt". I
> > think it would only get even more fragile if you attempted to override
> > what Qt does. They must have a way to make this work somehow.  
> 
> Well I am coming to the conclusion that it doesn't currently work under 
> QtWayland. I will start delving into that, but again there is not much 
> documentation on how it works that I have found to give me a pointer on 
> where to look. The fact that QWidgets are not drawing into their own 
> surfaces is a bit worrying, maybe some issue were actually found when 
> trying to do that.

Right.

Thanks,
pq

> >>>> I note in the qtwayland change logs, for the earlier QT5 for subsurface
> >>>> changes:
> >>>> dist/changes-5.6.3: - [QTBUG-52118] Fixed subsurface positions sometimes
> >>>> not being committed.
> >>>> dist/changes-5.11.2: - [QTBUG-69643] Fixed a bug where subsurfaces would
> >>>> not be rendered if clients added them before a WaylandQuickItem was
> >>>> created for the parent surface
> >>>> dist/changes-5.12.0: - [QTBUG-49809] Added support for
> >>>> wl_subsurface.place_above and place_below in WaylandQuickItem.
> >>>> dist/changes-5.15.2: - [QTBUG-86176] We now send subsurface expose
> >>>> events when a different toplevel (such as a dialog) is configured.
> >>>>
> >>>> Could any of these be related ?
> >>>>
> >>>> Terry  
> >>  

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20240229/facf84f5/attachment.sig>