Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

I have tried using "weston-debug scene-graph" and I am coming to the 
conclusion that qtwayland 6.5.0 is not really using native Wayland 
surfaces when Qt::WA_NativeWindow is used. From what I can see (and I 
could easily be wrong) the Wayland protocol shows wl_surfaces being 
created and two QWidget's QPlatformNativeInterface 
nativeResourceForWindow("surface", windowHandle()) function does return 
different wl_surface pointers but even at the QWidget level (ignoring 
gstreamer), a QPainter paint into each of these QWidgets actually uses 
Wayland to draw into just the one top level surface and "weston-debug 
scene-graph" shows only one application xdg_toplevel surface and no 
subsurfaces. I don't know how to determine the Wayland surface ID from a 
wl_surface pointer unfortunately to really check this.

If my Video QWidget(0) is a top level QWidget, then video is shown and 
"weston-debug scene-graph" shows the application xdg_toplevel and two 
wl_subsurfaces as children.

Unfortunately I think "weston-debug scene-graph" only shows surfaces 
that are actually "active" so I can't see all of the surfaces that 
Weston actually knows about (is there a method of doing this ?).

My feeling is that although Qtwayland is creating native surfaces, it 
actually only uses the one top level one and presumably doesn't 
"activate" (set a role, do something ?) with the other surfaces.

Does anyone know a good list/place where I can ask such detailed 
qtwayland questions ?

I guess I can work around this by manually creating a Wayland subsurface 
from the Qt top level surface and handing that to waylandsink and then 
manage this subsurface, like hiding, showing and resizing, when the 
QWidget is hidden/shown/resized.

Or could there be a way of "activating" the child QWidget's Wayland 
surface ?



On 22/02/2024 18:44, Terry Barnaby wrote:
> Hi Marius,
>
> Many thanks for the info.
>
> Some notes/questions below:
>
> Terry
> On 22/02/2024 17:49, Marius Vlad wrote:
>> Hi,
>> On Thu, Feb 22, 2024 at 03:21:01PM +0000, Terry Barnaby wrote:
>>> Hi,
>>>
>>> We are developing a video processing system that runs on an NXP imx8
>>> processor using a Yocto embedded Linux system that has Qt6, GStreamer,
>>> Wayland and Weston.
>>>
>>> We are having a problem displaying the video stream from GStreamer on a
>>> QWidget. In the past we had this working with Qt5 and older GStreamer,
>>> Wayland and Weston.
>>>
>>> A simple test program also shows the issue on Fedora37 with QT6 and
>>> KDE/Plasma/Wayland.
>> I'm tempted to say if this happens on a desktop with the same Qt 
>> version and
>> other compositors to be an issue with Qt rather than waylandsink or
>> the compositor. Note that on NXP they have their own modified Weston 
>> version.
>
> That is my current feeling and is one reason why I tried it on Fedora 
> with whatever Wayland compositor KDE/Plasma is using.
>
>
>>> The technique we are using is to get the Wayland surface from the 
>>> QWidget is
>>> using (It has been configured to use a Qt::WA_NativeWindow) and pass 
>>> this to
>>> the GStreamer's waylandsink which should then update this surface 
>>> with video
>>> frames (via hardware). This works when the QWidget is a top level 
>>> Window
>>> widget (QWidget(0)), but if this QWidget is below others in the 
>>> hierarchy no
>>> video is seen and the gstreamer pipeline line is stalled.
>> So the assumption is that aren't there other widgets which obscures this
>> one, when you move it below others?
>
> My simple test example has two QWidgets with the one for video being 
> created as a child of the first so it should be above all others. I 
> have even tried drawing in it to make sure and it displays its Qt 
> drawn contents fine, just not the video stream.
>
>
>>> It appears that waylandsink does:
>>>
>>> Creates a surface callback:
>>>
>>>    callback = wl_surface_frame (surface);
>>>
>>>    wl_callback_add_listener (callback, &frame_callback_listener, self);
>>>
>>> Then adds a buffer to a surface:
>>>
>>>      gst_wl_buffer_attach (buffer, priv->video_surface_wrapper);
>>>      wl_surface_set_buffer_scale (priv->video_surface_wrapper, 
>>> priv->scale);
>>>      wl_surface_damage_buffer (priv->video_surface_wrapper, 0, 0, 
>>> G_MAXINT32,
>>> G_MAXINT32);
>>>      wl_surface_commit (priv->video_surface_wrapper);
>>>
>>> But never gets a callback and just sits in a loop awaiting that 
>>> callback.
>>>
>>> I assume that the surface waylandsink is using, which is created 
>>> using the
>>> original QWidget surface (sub-surface ? with window ?) is not 
>>> "active" for
>>> some reason.
>> Possibly when QWidget is below in hierarcy to be a child of of a parent,
>> as described in 
>> https://wayland.app/protocols/xdg-shell#xdg_toplevel:request:set_parent,
>> so I assume to have a different surface than the parent one. This would
>> be easy to determine with WAYLAND_DEBUG. Seems unlikely to a itself a
>> sub-surface of a surface.
>
> I haven't really got the gist of whats going on, but waylandsink 
> certainly creates a subsurface from the QWidget surface, in fact it 
> seems to create a few things.
>
> I assume a subsurface is used so the video can be displayed in that 
> subsurface separately from the parent (de synced from it).
>
>
>>>
>>> I am trying to debug this, but this graphics stack is quite 
>>> complicated with
>>> waylandsink, qtwayland, wayland-lib and Weston not to mention the NXP
>>> hardware levels. My thoughts are that it is something qtwayland is 
>>> doing
>>> with the surface stack or thread locking issues (gstreamer uses 
>>> separate
>>> threads). I also don't understand Wayland or Weston in detail. So some
>>> questions:
>>>
>>> 1. Anyone seen something like this ?
>> Someone else reported something similar but that by causing damage,
>> or moving pointer to make the video sub-surface to show up:
>> https://gitlab.freedesktop.org/wayland/weston/-/issues/843.
>
> Thanks, I will have a look. Moving the mouse cursor in my case (at 
> least with Weston) does not affect things.
>
>
>>> 2. Anyone any idea one where to look ?
>>>
>>> 3. Given the wl_surface in the Qt app or in waylandsink is there a 
>>> way I can
>>> print out its state and the surface hierarchy easily ?
>> In Weston there's something called scene-graph. You can grab it by
>> starting Weston with with the --debug argument, then you can print
>> with `weston-debug scene-graph` command. A more recent Weston version
>> would indent sub-surfaces by their (main) surface parent.
>
> Thanks, that could be useful.
>
>
>>> 4. Any idea on any debug methods to use ?
>> WAYLAND_DEBUG=1 as env variable.
>
> Any idea on how to get a surfaces ID from a C pointer so I can match 
> up the QtWidget/waylandsink surface with the Wayland debug output ?
>
>
>>> Cheers
>>>
>>> Terry
>>>
>>>