Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

Hi Marius,

Many thanks for the info.

Some notes/questions below:

Terry
On 22/02/2024 17:49, Marius Vlad wrote:
> Hi,
> On Thu, Feb 22, 2024 at 03:21:01PM +0000, Terry Barnaby wrote:
>> Hi,
>>
>> We are developing a video processing system that runs on an NXP imx8
>> processor using a Yocto embedded Linux system that has Qt6, GStreamer,
>> Wayland and Weston.
>>
>> We are having a problem displaying the video stream from GStreamer on a
>> QWidget. In the past we had this working with Qt5 and older GStreamer,
>> Wayland and Weston.
>>
>> A simple test program also shows the issue on Fedora37 with QT6 and
>> KDE/Plasma/Wayland.
> I'm tempted to say if this happens on a desktop with the same Qt version and
> other compositors to be an issue with Qt rather than waylandsink or
> the compositor. Note that on NXP they have their own modified Weston version.

That is my current feeling and is one reason why I tried it on Fedora 
with whatever Wayland compositor KDE/Plasma is using.


>> The technique we are using is to get the Wayland surface from the QWidget is
>> using (It has been configured to use a Qt::WA_NativeWindow) and pass this to
>> the GStreamer's waylandsink which should then update this surface with video
>> frames (via hardware). This works when the QWidget is a top level Window
>> widget (QWidget(0)), but if this QWidget is below others in the hierarchy no
>> video is seen and the gstreamer pipeline line is stalled.
> So the assumption is that aren't there other widgets which obscures this
> one, when you move it below others?

My simple test example has two QWidgets with the one for video being 
created as a child of the first so it should be above all others. I have 
even tried drawing in it to make sure and it displays its Qt drawn 
contents fine, just not the video stream.


>> It appears that waylandsink does:
>>
>> Creates a surface callback:
>>
>>    callback = wl_surface_frame (surface);
>>
>>    wl_callback_add_listener (callback, &frame_callback_listener, self);
>>
>> Then adds a buffer to a surface:
>>
>>      gst_wl_buffer_attach (buffer, priv->video_surface_wrapper);
>>      wl_surface_set_buffer_scale (priv->video_surface_wrapper, priv->scale);
>>      wl_surface_damage_buffer (priv->video_surface_wrapper, 0, 0, G_MAXINT32,
>> G_MAXINT32);
>>      wl_surface_commit (priv->video_surface_wrapper);
>>
>> But never gets a callback and just sits in a loop awaiting that callback.
>>
>> I assume that the surface waylandsink is using, which is created using the
>> original QWidget surface (sub-surface ? with window ?) is not "active" for
>> some reason.
> Possibly when QWidget is below in hierarcy to be a child of of a parent,
> as described in https://wayland.app/protocols/xdg-shell#xdg_toplevel:request:set_parent,
> so I assume to have a different surface than the parent one. This would
> be easy to determine with WAYLAND_DEBUG. Seems unlikely to a itself a
> sub-surface of a surface.

I haven't really got the gist of whats going on, but waylandsink 
certainly creates a subsurface from the QWidget surface, in fact it 
seems to create a few things.

I assume a subsurface is used so the video can be displayed in that 
subsurface separately from the parent (de synced from it).


>>
>> I am trying to debug this, but this graphics stack is quite complicated with
>> waylandsink, qtwayland, wayland-lib and Weston not to mention the NXP
>> hardware levels. My thoughts are that it is something qtwayland is doing
>> with the surface stack or thread locking issues (gstreamer uses separate
>> threads). I also don't understand Wayland or Weston in detail. So some
>> questions:
>>
>> 1. Anyone seen something like this ?
> Someone else reported something similar but that by causing damage,
> or moving pointer to make the video sub-surface to show up:
> https://gitlab.freedesktop.org/wayland/weston/-/issues/843.

Thanks, I will have a look. Moving the mouse cursor in my case (at least 
with Weston) does not affect things.


>> 2. Anyone any idea one where to look ?
>>
>> 3. Given the wl_surface in the Qt app or in waylandsink is there a way I can
>> print out its state and the surface hierarchy easily ?
> In Weston there's something called scene-graph. You can grab it by
> starting Weston with with the --debug argument, then you can print
> with `weston-debug scene-graph` command. A more recent Weston version
> would indent sub-surfaces by their (main) surface parent.

Thanks, that could be useful.


>> 4. Any idea on any debug methods to use ?
> WAYLAND_DEBUG=1 as env variable.

Any idea on how to get a surfaces ID from a C pointer so I can match up 
the QtWidget/waylandsink surface with the Wayland debug output ?


>> Cheers
>>
>> Terry
>>
>>