Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

Fri Feb 23 08:41:43 UTC 2024

Hi,
On Fri, Feb 23, 2024 at 06:14:11AM +0000, Terry Barnaby wrote:
> I have tried using "weston-debug scene-graph" and I am coming to the
> conclusion that qtwayland 6.5.0 is not really using native Wayland surfaces
> when Qt::WA_NativeWindow is used. From what I can see (and I could easily be
> wrong) the Wayland protocol shows wl_surfaces being created and two
> QWidget's QPlatformNativeInterface nativeResourceForWindow("surface",
> windowHandle()) function does return different wl_surface pointers but even
> at the QWidget level (ignoring gstreamer), a QPainter paint into each of
> these QWidgets actually uses Wayland to draw into just the one top level
> surface and "weston-debug scene-graph" shows only one application
> xdg_toplevel surface and no subsurfaces. I don't know how to determine the
> Wayland surface ID from a wl_surface pointer unfortunately to really check
> this.
I suppose this is to expected given that you don't actually see the video. 
> 
> If my Video QWidget(0) is a top level QWidget, then video is shown and
> "weston-debug scene-graph" shows the application xdg_toplevel and two
> wl_subsurfaces as children.
> 
> Unfortunately I think "weston-debug scene-graph" only shows surfaces that
> are actually "active" so I can't see all of the surfaces that Weston
> actually knows about (is there a method of doing this ?).
Mapped or not, Weston will print out views associated with a surface, if
those views are part of a layer. I don't know what active means in this
case, but you won't be activating wl_surfaces but rather the top-level
xdg-shell window.  Depending on the Weston version it would explicit say
that or not (surface/view being not mapped).
> 
> My feeling is that although Qtwayland is creating native surfaces, it
> actually only uses the one top level one and presumably doesn't "activate"
> (set a role, do something ?) with the other surfaces.
WAYLAND_DEBUG=1 could tell if it creates or not subsurfaces underneath.
> 
> Does anyone know a good list/place where I can ask such detailed qtwayland
> questions ?
https://bugreports.qt.io/projects/QTBUG/issues/QTBUG-122683?filter=allopenissues
> 
> I guess I can work around this by manually creating a Wayland subsurface
> from the Qt top level surface and handing that to waylandsink and then
> manage this subsurface, like hiding, showing and resizing, when the QWidget
> is hidden/shown/resized.
> 
> Or could there be a way of "activating" the child QWidget's Wayland surface
> ?
> 
> 
> 
> On 22/02/2024 18:44, Terry Barnaby wrote:
> > Hi Marius,
> > 
> > Many thanks for the info.
> > 
> > Some notes/questions below:
> > 
> > Terry
> > On 22/02/2024 17:49, Marius Vlad wrote:
> > > Hi,
> > > On Thu, Feb 22, 2024 at 03:21:01PM +0000, Terry Barnaby wrote:
> > > > Hi,
> > > > 
> > > > We are developing a video processing system that runs on an NXP imx8
> > > > processor using a Yocto embedded Linux system that has Qt6, GStreamer,
> > > > Wayland and Weston.
> > > > 
> > > > We are having a problem displaying the video stream from GStreamer on a
> > > > QWidget. In the past we had this working with Qt5 and older GStreamer,
> > > > Wayland and Weston.
> > > > 
> > > > A simple test program also shows the issue on Fedora37 with QT6 and
> > > > KDE/Plasma/Wayland.
> > > I'm tempted to say if this happens on a desktop with the same Qt
> > > version and
> > > other compositors to be an issue with Qt rather than waylandsink or
> > > the compositor. Note that on NXP they have their own modified Weston
> > > version.
> > 
> > That is my current feeling and is one reason why I tried it on Fedora
> > with whatever Wayland compositor KDE/Plasma is using.
> > 
> > 
> > > > The technique we are using is to get the Wayland surface from
> > > > the QWidget is
> > > > using (It has been configured to use a Qt::WA_NativeWindow) and
> > > > pass this to
> > > > the GStreamer's waylandsink which should then update this
> > > > surface with video
> > > > frames (via hardware). This works when the QWidget is a top
> > > > level Window
> > > > widget (QWidget(0)), but if this QWidget is below others in the
> > > > hierarchy no
> > > > video is seen and the gstreamer pipeline line is stalled.
> > > So the assumption is that aren't there other widgets which obscures this
> > > one, when you move it below others?
> > 
> > My simple test example has two QWidgets with the one for video being
> > created as a child of the first so it should be above all others. I have
> > even tried drawing in it to make sure and it displays its Qt drawn
> > contents fine, just not the video stream.
> > 
> > 
> > > > It appears that waylandsink does:
> > > > 
> > > > Creates a surface callback:
> > > > 
> > > >    callback = wl_surface_frame (surface);
> > > > 
> > > >    wl_callback_add_listener (callback, &frame_callback_listener, self);
> > > > 
> > > > Then adds a buffer to a surface:
> > > > 
> > > >      gst_wl_buffer_attach (buffer, priv->video_surface_wrapper);
> > > >      wl_surface_set_buffer_scale (priv->video_surface_wrapper,
> > > > priv->scale);
> > > >      wl_surface_damage_buffer (priv->video_surface_wrapper, 0,
> > > > 0, G_MAXINT32,
> > > > G_MAXINT32);
> > > >      wl_surface_commit (priv->video_surface_wrapper);
> > > > 
> > > > But never gets a callback and just sits in a loop awaiting that
> > > > callback.
> > > > 
> > > > I assume that the surface waylandsink is using, which is created
> > > > using the
> > > > original QWidget surface (sub-surface ? with window ?) is not
> > > > "active" for
> > > > some reason.
> > > Possibly when QWidget is below in hierarcy to be a child of of a parent,
> > > as described in
> > > https://wayland.app/protocols/xdg-shell#xdg_toplevel:request:set_parent,
> > > so I assume to have a different surface than the parent one. This would
> > > be easy to determine with WAYLAND_DEBUG. Seems unlikely to a itself a
> > > sub-surface of a surface.
> > 
> > I haven't really got the gist of whats going on, but waylandsink
> > certainly creates a subsurface from the QWidget surface, in fact it
> > seems to create a few things.
> > 
> > I assume a subsurface is used so the video can be displayed in that
> > subsurface separately from the parent (de synced from it).
> > 
> > 
> > > > 
> > > > I am trying to debug this, but this graphics stack is quite
> > > > complicated with
> > > > waylandsink, qtwayland, wayland-lib and Weston not to mention the NXP
> > > > hardware levels. My thoughts are that it is something qtwayland
> > > > is doing
> > > > with the surface stack or thread locking issues (gstreamer uses
> > > > separate
> > > > threads). I also don't understand Wayland or Weston in detail. So some
> > > > questions:
> > > > 
> > > > 1. Anyone seen something like this ?
> > > Someone else reported something similar but that by causing damage,
> > > or moving pointer to make the video sub-surface to show up:
> > > https://gitlab.freedesktop.org/wayland/weston/-/issues/843.
> > 
> > Thanks, I will have a look. Moving the mouse cursor in my case (at least
> > with Weston) does not affect things.
> > 
> > 
> > > > 2. Anyone any idea one where to look ?
> > > > 
> > > > 3. Given the wl_surface in the Qt app or in waylandsink is there
> > > > a way I can
> > > > print out its state and the surface hierarchy easily ?
> > > In Weston there's something called scene-graph. You can grab it by
> > > starting Weston with with the --debug argument, then you can print
> > > with `weston-debug scene-graph` command. A more recent Weston version
> > > would indent sub-surfaces by their (main) surface parent.
> > 
> > Thanks, that could be useful.
> > 
> > 
> > > > 4. Any idea on any debug methods to use ?
> > > WAYLAND_DEBUG=1 as env variable.
> > 
> > Any idea on how to get a surfaces ID from a C pointer so I can match up
> > the QtWidget/waylandsink surface with the Wayland debug output ?
> > 
> > 
> > > > Cheers
> > > > 
> > > > Terry
> > > > 
> > > > 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20240223/77918ee5/attachment.sig>