[PATCH wayland 0/4] Untangle the symbol export duplication

[Daniel Stone]

Hi,
I do think this is best discussed in-person later in the week, but one
quick note ...

On 16 September 2016 at 14:47, Emil Velikov <emil.l.velikov at gmail.com> wrote:
> On 16 September 2016 at 10:46, Pekka Paalanen <ppaalanen at gmail.com> wrote:
>> The static approach can break only when the interface symbols are
>> exported and therefore possible to have conflicting symbols from
>> different binaries all loaded into the same process. I think that is
>> solved well enough by making wayland-scanner to stop forcing the export
>> of the interface symbols.
>
> Going a level up, abstracting ourselves of the actual wayland
> specifics, there are a few simple (imho) points:
>  - exposing the same symbols from different libraries is bad

Entirely in the abstract, you're right. But dropping back down into
the specifics, these interfaces iterate reasonably rapidly; it's
nothing like, say, EGL. As the interface structs aren't versioned by
name, if you build against (say) v7, you need to ensure that v7 of
that struct is available. If it's shipped in a DSO, to ensure this
dependency, you either need a separate DSO for
libwayland-extension-vN.so, which doesn't work since they'll all want
to provide the same symbol, or libwayland-extension.so.N, which again
means you'll have separate DSOs fighting over which should provide the
symbol. Alternately, apps would need to check that the runtime-bound
symbol is of (at least) the version they're expecting.

This isn't quite so bad in a totally homogeneous Yocto-style system,
where you can statically guarantee that this all works, but when you
throw in distributions with separately-upgraded packages from separate
sources, backports, people shipping their own dependencies in a
container style (Steam) or otherwise (Chrome), I don't think it's a
stable approach to work with.

Giving clients explicit control over the generation and visibility of
those symbols, OTOH, does allow you to make that guarantee. Yes at a
cost of maybe 20kB, but when you add 8MB just from going from double-
to triple-buffered, I don't think that's such a huge deal.

(Unless you just mean that external symbol collisions / ambiguous
resolution is bad: yes, we all agree on that.)

>  - hiding the exports will break applications and people will notice
> that _after_ things are already broken.

Yes; how about a scanner option to allow you to hide it?

>  - any arguments against shared libraries is applicable to every
> project in the wild.

In the abstract, yes; it's only when you get down to details like this
that it becomes clear as to which trade-offs are worthwhile.

>> Wayland-protocols would need to install 15 libraries already and that
>> number will just grow and never go down.
> Considering new protocols are added how could one expect _anything_ to
> get less ?
> If the number (15) of libraries is an issue one can fold things into a
> single library. It will be a tiny bit of a cheat, but it's perfectly
> reasonable.

Again in the abstract or in cases like libEGL, where you have a single
source of truth based on the idea of specifications which only evolve
quite slowly in a single uniform direction, 'perfectly reasonable' is
true. Looking at the pace and diversity of Wayland protocol
development though, I don't believe that to be true here. Trying to
manage this would, I believe, lead to a rapid explosion in soversion,
needing a lot of rebuilds which would make life difficult enough for
people that they'd bypass it. I can't imagine it'd be popular with
distributions either.

>> A project cannot make use of an interface v5 if it only implements
>> support for v4. Using the library you suggest or not does not change
>> that, support for a new interface version always requires new
>> hand-written code for the new semantics. Therefore it is good enough to
>> build the interface v4 definitions into the program directly, and avoid
>> the runtime dependency hassle. The built-in v4 will work just fine,
>> even if the other side of IPC supported only up to v3 or all the way up
>> to v8.
>>
> The above does not consider/attribute for buggy applications (as
> mentioned below), as mentioned before.

If an application blindly negotiates versions it has no concept of,
this is the case. Similarly, if an application blindly negotiates an
EGL context with GL 4.4 core profile only, and then expects the full
fixed-function 1.x pipeline to continue working, then ... yeah, not
much you can do.

>> We can fix that for everything except libwayland by not exporting the
>> symbols.
>
> As before: one _cannot_ "unexport" the symbols since this will break
> applications. And you really don't want to intentionally break
> applications, when there is a way to avoid that ?

Not by default, no. But it can be optional for people who want to
DTRT, and eventually cause the scanner to emit a warning when an
explicit --export or --no-export isn't specified.

>>> Combine that with the lack
>>> of MIN(...) in (some) clients, equals a recipe for disaster.
>>> Admittedly this (complete lack of MIN) is an application bug, yet it's
>>> something that cannot (should not) be fixed/handled in wayland,
>>> correct ?
>>
>> Are you talking about ignoring available_in_symbols version, or not
>> doing version checks at all?
>>
> Not doing version checks _at all_.
>
> Yes it is an application bug, and the proposed solution lets one
> mitigate things. That obviously does _not_ mean that applications
> should not be fixed.

I don't think they can mitigate things, no. Regardless of how the
client is linked, if it requests a higher version number from the
server than the client is prepared to deal with, then the semantics of
how that protocol works on the wire can and do change. This just isn't
something we can fix without touching those apps.

>> Yes, so let's stop the forced exporting where we can.
>
> There's more to it - you're suggesting to hide them by default (the
> non libwayland ones of course). I'm saying that one should keep them
> around (in alternative form) by default and allow people to hide them.

In the spirit of full backwards compatibility, I agree.

>>> Similar to libwayland-util.so, any old and new projects will continue
>>> to work and the symbol duplication will be resolved.
>>
>> You only resolve the duplication between libwayland-server and
>> libwayland-client, which has never been a problem heard about in
>> upstream. The symbols exported by them are identical, so it does not
>> matter which one provides them.
>>
> The "don't provide duplicate symbols" rule/suggestion/etc. is there
> for a reason.
>
> Just because in this particular place (project) and time those have
> identical implementation there is no guarantee that things will stay
> the same.

Well, taking it as a given that we can't break ABI on either by
unexporting symbols, then the only conclusion we can arrive at is that
we have to route around them when doing new development.

>> Does libva not carry a copy wayland-drm.xml and generate its own code
>> from it during build?
>>
>> Or is it really depending on the... hmm, libEGL.so? - exporting
>> wl_drm_interface?
>
> IIRC it carries the XML only for the client header, yet it depends on
> libEGL.so. And yes, they _expect_ to have a cannonical provider for
> the _interface symbol(s). It's only fair to assume that they're not
> the only ones (but perhaps for other _interface symbols).

Yeah, that's pretty broken. But VA-API isn't read-only: we can send
patches and help educate them on best practice. We can spread that
around wherever we find it - as we've already done for bad
documentation, sample clients, and real implementations - and over
time, this will take root.

>>> tldr: I'm suggesting creating a libwayland-interface.so shared library

I don't think this will work, for the reasons outlined above.

>>> Using shared library:
>>> pros: code reuse, single provider for the _interface symbol.
>>> cons: extra bit in dependency tracking/tree

Cons: Requires totally linear development process for these extensions
tracked inside the lib.
Cons: Requires total backwards compatibility for all extensions in that lib.

>>> Using static linking:
>>> pros: simpler dependency tree/tracking
>>> cons: 'bloated' binaries, duplicated symbols which may or may not be
>>> conflicting (in one keeps the WL_EXPORT), broken applications (if one
>>> drops WL_EXPORT)
>>
>> IMO, let's go with static only. There really isn't enough to share in
>> a .so to warrant all the work needed for it all over.
> If it makes sense, but you won't be able to help just opt for "I can
> see it, sadly I don't have time for it atm.". Objecting/blocking based
> on your present availability is a bit strage.

I don't think it's anything to do with his time, but in fact technical
objections - same with me.

Cheers,
Daniel