Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Pekka Paalanen]

On Mon, 26 Feb 2024 15:18:27 +0000
Terry Barnaby <terry1 at beam.ltd.uk> wrote:

> Hi Pekka,
> 
> Thanks for the response. Notes below:
> 
> Terry
> 
> On 26/02/2024 13:28, Pekka Paalanen wrote:
> > On Sun, 25 Feb 2024 08:04:30 +0000
> > Terry Barnaby <terry1 at beam.ltd.uk> wrote:
> >  
> >> Hi,
> >>
> >> I have investigated a bit further. I have built my own Weston server to
> >> run under X11 on Fedora37 so I can add printf's and debug more easily
> >> than using a cross compiled iMX8mp target system etc. I added a new
> >> dsmc-shell to Weston which is identical to kiosk-shell (apart from
> >> names) so I could play with that.
> >>
> >> When It run my simple QWidget test program (test016-qt6-video-example)
> >> which creates one top level QWidget with a child QWidget to display the
> >> Gstreamer video in it, I see the following surfaces/subsurfaces when
> >> desktop_surface_committed() is called in the dsmc-shell.
> >>
> >> Surface: 16 1044,620 mapped: 1 opaque: 0
> >>    View: 0x29182b0
> >>    Surface: 18 0,0 mapped: 0 opaque: 0
> >>     Surface: 44 640,480 mapped: 1 opaque: 0
> >>     Surface: 18 0,0 mapped: 0 opaque: 0
> >>    Surface: 17 0,0 mapped: 0 opaque: 0
> >>    Surface: 16 1044,620 mapped: 1 opaque: 0  
> > Btw. the sub-surface list also contains the parent weston_surface in
> > it, that's why surface 18 and 16 show up twice, I guess. It's used for
> > z-ordering.  
> 
> Yes, that was my view.
> 
> 
> >  
> >> Surface 16 is used by the top level QWidget, surface 18 is used by the
> >> Video QWidget and surface 44 is the GStreamer video display surface (I
> >> think!). This list is generated traversing the weston_surface's views
> >> and subsurface_list lists. The mapped is the "is_mapped" field of the
> >> surface.
> >> Only the top level weston_surface has a weston_view in the views list it
> >> that is any relevance. "weston-debug scene-graph" only shows the tope
> >> most surface, no subsurfaces.  
> > Right.
> >
> > A sub-surface requires its parent surface to be mapped in order to show
> > up on screen. This applies recursively, so surface 18 not being mapped
> > prevents surface 44 from showing up.
> >
> > IIRC, more or less only "fully mapped" weston_surfaces (as in, if it's
> > a sub-surface, the whole sub-surface ancestry path up to a top-level is
> > mapped) have a weston_view. weston_view defines where on screen a
> > weston_surface will be presented, so without a view it cannot show up.  
> 
> Ok. What Wayland API requests cause a surface to actually be mapped 
> (Sorry don't really know Wayland protocol) ?

Hi Terry,

the basic protocol object is wl_surface. The wl_surface needs to be
given a "role". This is a specific protocol term. xdg_surface and
xdg_toplevel can give a wl_surface the role of a top-level window,
which means it can get mapped when you play by the rules set in the
xdg_toplevel specification.

Sub-surface is another role.

So the rules are always role specific, but at some point they all
require content on the wl_surface, which is given with the attach
request followed by a commit request. Role rules specify when and how
that can be done.

> The higher level surfaces are created/managed by QtWayland, but maybe I 
> can override somehow.
> 

That does not feel like a good idea to me. But I also cannot really
help you, because this all seems to be pointing at Qt which I know
nothing about.

> >  
> >> As mentioned before, If I use QPainter to draw into the video QWidget it
> >> actually draws into the top QWidgets 16 surface using Wayland protocol.
> >> I would have thought it would draw into its own QWidget surface 18 as it
> >> has Qt::WA_NativeWindow set ?

This question seems to be the essence. If Qt worked like you expected,
then I think the whole program would work.

However, there is no need (from Wayland perspective) to have a
wl_surface as "surface 18" in the middle. What would be best is if you
could somehow have that "middle" QWidget but without it's own
wl_surface. Have the QWidget control the GStreamer wl_surface position
through wl_subsurface interface, while GStreamer plays the video
through wl_surface interface.

Wayland does not relay sub-surface resizing or positioning between two
client-side components at all. There is not even a way query a
sub-surface position. So the positioning and sizing is all done in
Qt<->GStreamer somehow without Wayland in between. Only the end result
gets sent over Wayland to display: Qt sets up the position and
GStreamer sets up the size and content.

> >>
> >> I assume that Qtwayland is not "activating" the video QWidget's surface
> >> or using it for some reason (send subsurface expose events ?) ?
> >>  
> > If that's true, then it is very relevant. A sub-surface becomes mapped
> > and visible when:
> >
> > - its parent surface is mapped and visible, and
> >
> > - the parent surface is committed after the sub-surface has been
> >    created and associated, and
> >
> > - if the sub-surface is in synchronized mode, there also needs to be a
> >    parent surface commit after every sub-surface commit you want to
> >    become visible. So if you do the first sub-surface sync-mode commit
> >    with a buffer after the parent has already committed the
> >    sub-surface's creation, the parent surface needs too commit again.
> >
> > This applies recursively, too, and you have a sub-sub-surface there.
> >
> > Do you actually need to sub-surface in the middle? Have you tried
> > without it?  
> 
> I am not doing anything with Wayland directly. Qt is managing the higher 
> level surfaces/subsurfaces and then GStreamers waylandsink is passed one 
> of these Qt managed surfaces and it creates the subsurface 44. Looking 
> at waylandsink it should set this subsurface to be desynced so it can 
> draw into this surface without synchronising to the parents surface 
> managed by Qt.

Right, and desync is not enough if its parent is not mapped.

> All I am trying to do is use the technique as mentioned in various 
> forums/lists to get GStreamer to display a video such that it "appears" 
> where a QWidget is on the screen. Mind you all of this info is very 
> rough and ready. For X11 it appears stable, but for Qt/Wayland the info, 
> and it appears functionality, is not quite all there.
> 
> When you say a sub-surface in the middle I presume you mean the surface 
> 18 of the lower QWidget ? Well I have tried using the top most QWidget's 
> surface 16 and the video is displayed, although all over the 
> application. I really need to manage this surface so it can be raised, 
> lowered and resizes amongst the other QWidgets somehow. I have tried 
> using direct Wayland API calls to create a subsurface manually from the 
> top surface but so far I have just got protocol errors while trying 
> this. It may be my bad Wayland client code or how it is interfering with 
> Qt's Wayland interface.
> 
> I have even tried using a separate top level surface. Unfortunately as 
> the standard Wayland protocols do not allow an application to move or 
> manage top level Windows this is not useful. I guess I could create an 
> extra Wayland/Weston server protocol to allow my app (embedded system) 
> to manage these windows, but I was trying to do things in as standard a 
> way as possible.

Rather than "more Wayland", I suspect you should look into "more Qt". I
think it would only get even more fragile if you attempted to override
what Qt does. They must have a way to make this work somehow.


Thanks,
pq

> >> I note in the qtwayland change logs, for the earlier QT5 for subsurface
> >> changes:
> >> dist/changes-5.6.3: - [QTBUG-52118] Fixed subsurface positions sometimes
> >> not being committed.
> >> dist/changes-5.11.2: - [QTBUG-69643] Fixed a bug where subsurfaces would
> >> not be rendered if clients added them before a WaylandQuickItem was
> >> created for the parent surface
> >> dist/changes-5.12.0: - [QTBUG-49809] Added support for
> >> wl_subsurface.place_above and place_below in WaylandQuickItem.
> >> dist/changes-5.15.2: - [QTBUG-86176] We now send subsurface expose
> >> events when a different toplevel (such as a dialog) is configured.
> >>
> >> Could any of these be related ?
> >>
> >> Terry  
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20240226/108e9f04/attachment.sig>