Enums, bitfields and wl_arrays in the .xml file

[Victor Berger]

Hi,

First of all, thanks for your replies.

On 2015-09-24 20:35, Bill Spitzak wrote :
> 
> Um, that is entirely the point of this change! The current xml does
> not provide enough information so that a language binding can know
> that some arrangements are illegal. Therefore current language
> bindings don't produce errors for invalid code. With this extra
> information, the compiler now knows that the arrangement is invalid,
> and can produce an error. If a new error is a "breaking change" then
> this patch MUST be a breaking change, or it is not working.

Yes I agree with you from the perspective of my Rust bindings. However, 
from discussions on IRC, there is a strong position towards 
backwards-compatibility.
 From what I understood, it seems like XML files that are currently 
accepted as valid by the C scanner will continue to be accepted by it.

Thus, we risk reaching a point where different scanners accept different 
XML formats. And for now, the specification of the XML files is defined 
by the implementation of the C scanner, which is likely to be the most 
tolerant scanner.

> What do you mean by "variant"? My guess is this does not mean "a new
> enum value that did not exist before", but instead the ability to send
> arbitrary numbers. But that seems to be duplicated by your next
> statement:

Actually, I was exactly meaning "a new enum value that did not exist 
before". Rust "enums" are actually tagged unions, and if you match over 
an instance of these "enums", the compiler will force you to match all 
possible values. Hence, introducing a new possible value for a Rust enum 
is a breaking change of the API (in terms of the generated code).

But indeed, given there is a need to support arbitrary values for enums 
in some protocols, this problem can be solved at the same time as the 
next:

>> - some enums, like xdg_shell::xdg_surface::states are by definition
>> not complete.
> 
> Therefore the language binding must provide a method to send an
> arbitrary number. However this can require extra syntax so that it is
> clear that the programmer intended it. In C++ this is done by casting
> that number to the enumeration value, ie you can't put 42 in the
> argument list, but you can put EnumerationType(42). Your description
> of how you would handle this in Rust sounds the same.

In Rust it would require some more overhead (casting an arbitrary number 
as an enum is unsafe and undefined behavior in Rust, even if the enum is 
internally represented by the same type as the given integer), but it is 
essentially the same idea, yes.

Actually, my concern was more about *receiving* and arbitrary number 
than *sending* it. The bindings must be able to gracefully handle a 
value returned by the compositor that does not fit into the enums 
described by the protocol. It is however mostly the same problem, and 
can be fixed in the same way.

>> - and most likely other similar issues with "array" types or so
> 
>> These questions need to be resolved, and a proper specification of
>> what should contain the .xml files will certainly need to be written
>> to properly handle all this.
> 
> The proposed additions provide EXACTLY what is needed by many language
> bindings. Yes maybe there willl be new things added later. That is not
> a reason to stop the current proposal.

What I meant here is that the format and contents of the XML files is 
currently defined by the implementation of the C scanner, which is a 
less than optimal situation to discuss evolutions of this format.

There will most likely be a need to write a proper document describing 
the format of the XML files, as well as the semantic meanings of each 
field and attribute it contains.
The questions about how breaking evolutions will be handled need to be 
specified as well: how should an old scanner behave when it encounters a 
more recent protocol file, containing fields or attributes it does not 
recognize ? Ignore them ? Fail and declare being "not compatible with 
this protocol format, please upgrade" ?

> It does sound a bit like it would be useful for Rust to know whether
> it needs to support arbitrary values ("variant"s?), because an enum
> that does not support them might be a lot simpler (in fact this is
> probably the main reason for the bitfield indicator, in C++ and Python
> and probably many languages a type that supports bit operations is
> much more complex and ugly than an enum). So maybe a new field should
> be added to enum/bitfields to indicate that arbitrary values must be
> supported. However that is a NEW field, it does not change the current
> proposal at all, and there is no reason to hold off useful and
> important changes until this new field is decided on.

I first thought knowing when to support arbitrary values would be an 
important point, but actually, as I described a few paragraphs earlier 
in this email, it is mostly the same issue as future-proofing the enums 
to handle newer values the compositor might send.

As I said described earlier in this mail, adding a new field/attribute 
to an entry in the XML files can be a breaking change for bindings for 
some languages.

I don't think other bindings than the C-scanner already have so many 
uses that they can't afford a breaking change (correct me if I'm wrong), 
so this is most likely the ideal time to properly specify the protocol 
file format in a way that contains all information required for a proper 
type-safety as needed by more high-level languages than C.

This would indeed be a breaking-change in terms of what XML files are 
considered as valid, _however_ it would likely not change the code 
generated by the C-scanner. Thus, the migration for protocol writers 
would mostly be fearless (even more if new XML files are still accepted 
by the current C-scanner), and the downstream user of these protocols 
would just need to update the protocol files to the new format when they 
update their C-scanner. Which I think would allow for a mostly painless 
transition, for the benefit of everyone.

Future changes to the XML format would likely not be as painless, when 
various languages bindings will have many downstream users. Thus it is 
important to get a good version of the XML specification now. As there 
are bindings projects for various languages (at least C++, Java, Rust 
and Haskell that I am aware of), we can probably together reach a state 
handling (almost ?) all needs of type-safety.

I am aware this is a more consequent work than simply patching the xml 
files on the go as the need is encountered, but providing a good and 
complete proposition will most likely be better received by the wayland 
devs and the downstream users than just "breaking things as we need it".

----
Victor