Wayland debugging with Qtwayland, gstreamer waylandsink, wayland-lib and Weston

[Terry Barnaby]

On 04/03/2024 14:14, Pekka Paalanen wrote:
> On Mon, 4 Mar 2024 13:24:56 +0000
> Terry Barnaby <terry1 at beam.ltd.uk> wrote:
>
>> On 04/03/2024 09:41, Pekka Paalanen wrote:
>>> On Mon, 4 Mar 2024 08:12:10 +0000
>>> Terry Barnaby <terry1 at beam.ltd.uk> wrote:
>>>   
>>>> While I am trying to investigate my issue in the QtWayland arena via the
>>>> Qt Jira Bug system, I thought I would try taking Qt out of the equation
>>>> to simplify the application a bit more to try and gain some
>>>> understanding of what is going on and how this should all work.
>>>>
>>>> So I have created a pure GStreamer/Wayland/Weston application to test
>>>> out how this should work. This is at:
>>>> https://portal.beam.ltd.uk/public//test022-wayland-video-example.tar.gz
>>>>
>>>> This tries to implement a C++ Widget style application using native
>>>> Wayland. It is rough and could easily be doing things wrong wrt Wayland.
>>>> However it does work to a reasonable degree.
>>>>
>>>> However, I appear to see the same sort of issue I see with my Qt based
>>>> system in that when a subsurface of a subsurface is used, the Gstreamer
>>>> video is not seen.
>>>>
>>>> This example normally (UseWidgetTop=0) has a top level xdg_toplevel
>>>> desktop surface (Gui), a subsurface to that (Video) and then waylandsink
>>>> creates a subsurface to that which it sets to de-sync mode.
>>>>
>>>> When this example is run with UseWidgetTop=0 the video frames from
>>>> gstreamer are only shown shown when the top subsurface is manually
>>>> committed with gvideo->update() every second, otherwise the video
>>>> pipeline is stalled.
>>> This is intentional. From wl_subsurface specification:
>>>
>>>         Even if a sub-surface is in desynchronized mode, it will behave as
>>>         in synchronized mode, if its parent surface behaves as in
>>>         synchronized mode. This rule is applied recursively throughout the
>>>         tree of surfaces. This means, that one can set a sub-surface into
>>>         synchronized mode, and then assume that all its child and grand-child
>>>         sub-surfaces are synchronized, too, without explicitly setting them.
>>>
>>> This is derived from the design decision that a wl_surface and its
>>> immediate sub-surfaces form a seamlessly integrated unit that works
>>> like a single wl_surface without sub-surfaces would. wl_subsurface
>>> state is state in the sub-surface's parent, so that the parent controls
>>> everything as if there was just a single wl_surface. If the parent sets
>>> its sub-surface as desynchronized, it explicitly gives the sub-surface
>>> the permission to update on screen regardless of the parent's updates.
>>> When the sub-surface is in synchronized mode, the parent surface wants
>>> to be updated in sync with the sub-surface in an atomic fashion.
>>>
>>> When your surface stack looks like:
>>>
>>> - main surface A, top-level, root surface (implicitly desynchronized)
>>>     - sub-surface B, synchronized
>>>       - sub-surface C, desynchronized
>>>
>>> Updates to surface C are immediately made part of surface B, because
>>> surface C is in desynchronized mode. If B was the root surface, all C
>>> updates would simply go through.
>>>
>>> However, surface B is a part of surface A, and surface B is in
>>> synchronized mode. This means that the client wants surface A updates to
>>> be explicit and atomic. Nothing must change on screen until A is
>>> explicitly committed itself. So any update to surface B requires a
>>> commit on surface A to become visible. Surface C does not get to
>>> override the atomicity requirement of surface A updates.
>>>
>>> This has been designed so that software component A can control surface
>>> A, and delegate a part of surface A to component B which happens to the
>>> using a sub-surface: surface B. If surface B parts are further
>>> delegated to another component C, then component A can still be sure
>>> that nothing updates on surface A until it says so. Component A sets
>>> surface B to synchronized to ensure that.
>>>
>>> That's the rationale behind the Wayland design.
>>>
>>>
>>> Thanks,
>>> pq
>> Ah, thanks for the info, that may be why this is not working even in Qt
>> then.
>>
>> This seems a dropoff in Wayland to me. If a software module wants to
>> display Video into an area on the screen at its own rate, setting that
>> surface to de-synced mode is no use in the general case with this
>> policy.
> It is of use, if you don't have unnecessary sub-surfaces in synchronized
> mode in between, or you set all those extra sub-surfaces to
> desynchronized as well.

Well they may not be necessary from the Wayland perspective, but from 
the higher level software they are useful to modularise/separate/provide 
a join for the software modules especially when software modules are 
separate like Qt and GStreamer.


>
>> I would have thought that if a subsurface was explicitly set to
>> de-synced mode then that would be honoured. I can't see a usage case for
>> it to be ignored and its commits synchronised up the tree ?
> Resizing the window is the main use case.
>
> In order to resize surface A, you also need to resize and paint surface
> B, and for surface B you also need to resize and paint surface C. Then
> you need to guarantee that all the updates from surface C, B and A are
> applied atomically on screen.
>
> Either you have component APIs good enough to negotiate the
> stop-resize-paint-resume on your own, or if the sub-components are
> free-running regardless of frame callbacks, component A can just
> temporarily set surface B to synchronized, resize and reposition it,
> and resume.

I would have thought that the Wayland server could/would synchronise 
screen updates when a higher level surface is resized/moved by itself.

As the software components are separately developed systems it is 
difficult to sync between them without changing them, but may be possible.


>
>> So is there a way to actually display Video on a subsurface many levels
>> deep in a surface hierarchy, would setting all of the surfaces up to the
>> subsurface just below the desktop top level one work (although not ideal
>> as it would mean overriding other software modules surfaces at the
>> Wayland level) ?
> Setting to what?

I meant setting all the subsurfaces in the tree to be desynced. You have 
answered this one above. I can try that by setting the Video widget to 
desyned and see how that works with my simple application and a 
QtWidget, although Qt might object.


>
>> Or can desynced subsurfaces really only work to one level deep ?
> You can set your middle sub-surface to desynchronized, too, at least
> from Wayland perspective. I don't know if Qt let's you.
>
> There are implementation issues with nested sub-surfaces in Weston, but
> this does not seem to be one of them.

As above, I will try this. Thanks for the info.


>
>> If it is just one subsurface level deep that video can be displayed, I
>> guess I will have to get GStreamers waylandsink to create its subsurface
>> off the top most surface and add calls to manage its surface from my
>> app.
> That should have been the first idea to try.
>
> Is Gst waylandsink API the kind that it internally creates a new
> wl_surface for itself and makes it a sub-surface of the given surface,
> or is there an option to tell Gst to just push frames into a given
> wl_surface?
>
> If the former, then waylandsink is supposed to somehow give you an API
> to set the sub-surface position and z-order wrt. its parent and
> siblings. If the latter, you would create wl_subsurface yourself and
> keep control of it to set the sub-surface position and z-order.
>
> Either way, the optimal result is one top-level wl_surface, with one
> sub wl_surface drawn by Gst, and no surfaces in between in the
> hierarchy.

Yes, the Gst waylandsink API creates a new subsurface for itself from 
the GUI's managed surface to separate itself from the GUI (Qt/Gnomes) 
surfaces. It doesn't allow you to provide a surface to directly use. I 
don't think it allows the surface to be moved/resized although it can 
display video at an offset and size as far as I know (although it may 
actually change the surface to do this I will have a look). It doesn't 
allow the z-order to be changed I think. It expects the GUI to change 
its surface and I guess assumes its subsurface would effectively move in 
z and xy position due to the GUI moving/raising/lowering its surface 
(the parent) in a similar manner to how X11 would have done this.

I will try the middle desync and/or this method by managing the 
waylandsink surface outside of waylandsink if I can and if it doesn't 
mess up either Qt's or waylandsink's operations.

Thanks for the input.


>
>> Or maybe get waylandsinks subsurface and manipulate it behind
>> waylandsinks back. Not sure what this will do to the Qt level though.
>> Using the QWidgets subsurface as its base should have allowed isolation
>> to a degree between the Qt and wayland sink sub systems, its a much more
>> modular approach.
> You cannot punt between-components integration to Wayland. The
> sub-surfaces design tried and failed, and as a result the sub-surface
> protocol is both complex and insufficient. You still need explicit
> communication between your client-side components when you resize.
>
>> There is no documentation on this Wayland restriction in waylandsink or
>> other places, I can try and feed this back.
>>
>> Oh for X11 !
> Yeah, the blink of garbage when windows are resized, or the blue
> colorkey following far behind when a window is moved.

Never had that and it did work :)


>
>
> Thanks,
> pq