Enums, bitfields and wl_arrays

[Auke Booij]

This email contains a clear suggestions that I'd like everyone to
read, not just Nils and Erik. But Nils and Erik brought up a few
points that I'd like to have settled, since they have been brought up
quite frequently now.

On 3 October 2015 at 11:14, Nils Chr. Brause <nilschrbrause at gmail.com> wrote:
> Pekka Paalanen wrote:
>> Later that thread also brought up two incompatible definitions of a
>> "bitfield":
>> - a type where bit-wise logic operations make sense
>> - a type where all listed values are powers of two (POT), and bit-wise
>>   logic operations make sense
>>
>> As a practical example, is wl_output::transform a bitfield or an enum?
>> - it is used as an int, not uint
>> - it lists all possible values
>> - the listed values are not all POT
>> - bit-wise operations make sense (check for flipped)
>>
>> Arguably it should have been an uint, but I don't think we can change
>> it now.
>
> I have a rather pragmatic view at this: If it looks and feels like a
> bitfield, it is a bitfield.
> And when looking an the definition, it for all the wold looks like a
> bitfield to me.
> Bit 0 means 90°, bit 1 means 180° and bit 3 means flipped.
> The signedness doesn't matter for a bitfield, since it is just a
> collection of bits without a numerical meaning.

So perhaps we should say:

"An enum can be specified to be a bitfield if the primary interface to
its numeric value deals with bitwise operations."

Is that agreeable?

> Pekka Paalanen wrote:
>> yeah, adding the interface name makes perfect sense. It could be
>> optional if wanted. No dot would mean the enum is in the same
>> interface, a dot would signify a specific interface. I don't think
>> there is any use for an anonymous global namespace like Bill mentioned.
>
> Sounds good to me. Although I think it would be easier to implement,
> if we'd just use the dot notation everywhere. This would result in
> less code and (statistically) less bugs.

It is the "dotless" version that is already supported by my patches.
The dotted one requires new code. So apart from a single if statement,
this is not really more work. But I guess this is getting into
bikeshedding terrain.

> Auke Booij wrote:
>> The enum and bitfield attributes are in principle for documentation
>> purposes only.
>
> No. The whole point of the enum an bitfield attributes is to help
> non-C languages to generate type-safe bindings.
>
> Auke Booij wrote:
>> The enum and bitfield attributes may also be used by
>> bindings, but only in such a way that code written prior to the
>> specification of these attributes still works after their
>> specification.
>
> I think every language binding should ultimately decide for
> themselves, how they are using the new attributes. There is no way you
> can dictate anyone how to use them.

I think that we have an agreement in principle, just not in terms (and
this also goes back to my comment to Victor Berger). What I meant to
express is that any bindings that violate this rule are on the risk of
the bindings writers, rather than the wayland and wayland protocol
developers.

Would you agree with the following? Otherwise please suggest a
phrasing you would agree with.

[start]
The purpose of the enum and bitfield attributes is to document what
arguments refer to which enums, and to document which numeric enum
values are primarily accessed using bitwise operations.
Additionally, the enum and bitfield attributes may be used by other
code, such as bindings to other languages, for example to enhance type
safety of code. However, such usage is only supported if the following
property is satisfied: code written prior to the specification of
these attributes still works after their specification. In other
words, specifying an attribute for an argument, that previously did
not have an enum or bitfield attribute, should not break API. Code
that does not satisfy this rule is not guaranteed to obey backwards
compatibility.
[end]

> Auke Booij wrote:
>> I agree with your concern. However, note that "for documentation
>> purposes" does not mean that it is just a documentation string: we
>> document a very clear and precise fact, for the purpose of
>> understanding the API (ie documentation). My previous patches indeed
>> do some checking wrt enum name cross-matching. This would have to be
>> extended for enum names that refer to a different interface.
>
> You shouldn't write "for documentation purposes only" then, as it is
> clearly not. It is mainly for the type-safety of non-C
> language-bindings.

Please do comment on the above, as I understand your point, but also
want to be pragmatic here.

On 3 October 2015 at 20:05, Erik De Rijcke <derijcke.erik at gmail.com> wrote:
> - Defining an uint (bitfield) or int (single enum) in the xml for an enum
> argument is the wrong approach. It prescribes, not describes. In some
> languages eg Java, there is no such thing as an uint. What is important, is
> to know how many enum values you are allowed to pass in an enum argument.
> How this translates low level is an implementation detail (wire format). We
> don't define opcodes either. The wire format in then end, is implicitly
> defined by the C generator based on the argument type in the xml (for an
> enum arg, some cases use an array of shorts, some cases use a int
> bitfield.). It would be better to define that relationship.

I chose to fix bitfield enums to be uint because other options suggest
a mistake. The case where an old bitfield-style enum is actually
transported as an int is rare (although apparently exists in
wl_output::transform), and I think catching mistakes is more important
here. But please suggest improvements if you think otherwise.

> General concerns as a language binding dev:
>
> - The definition of an enum, at least in Java (and I assume in a lot of
> other languages too), is a fixed set of values that are know at compile
> time. This set is unmodifiable at runtime. This means no new values of an
> enum type can be (safely) added/defined at runtime. It seems to me this
> definition is ultimately incompatible with wayland enums for several
> reasons.
>
> First reason is the case of an 'open' enum, as seen in xdg-shell state. From
> my understanding, a client or server should be able to act on any 'open'
> enum value it receives over the wire, even if this value was not known at
> compile time. As such a generated binding can not simply map all unknown
> wire values to the same "UNKNOWN" language specific enum type as the raw
> value would be lost in such case. It also makes it impossible to send out
> any undefined raw value. An 'open' enum is thus impossible to implement
> using a (language specific) enum as type. The core reason being that a
> wayland 'open' enum is not really an enum but a set of pre-defined
> constants.
>
> The second reason is the case of the 'closed' enum, as seen in all other
> enum definitions. A 'closed' wayland enum does allow to be directly mapped
> to a language specific enum type. However there is still the case of enum
> values known only by the other side (=different protocol versions). Again
> the solution(s) for this are the same as the first reason. Map to an
> 'UNKNOWN" value and have your raw value lost, or pass on the raw value but
> loose your language enum type usage.
>
> Both concerns seem to have no relevance to wayland per se, and should simply
> be solved by the language binding. I believe this not to be the case as
> there is the open question (for me) on what should be considered as
> acceptable behaviour when dealing with wayland enums in language bindings.
> If it is acceptable to have your raw value lost for a 'closed' enum, but not
> for an open value, then the language binding must be able to deal with that
> appropriately. However for that it most know in some way when an enum is
> open, and when it is closed (either by implicit ruling, eg it's an array
> thus open, or explicitly through an xml attribute).
>
> amen

Obviously this would be nice information to have on the user's side.
And we have been discussing this for a long time now. But if it is not
clearly specified on the protocol/wayland side, then we *will* get
issues with this.

I'd like to suggest that anyone who wishes to further the open/closed
enums debate suggests a firm specification that we can work with.
Otherwise we can keep bringing this up, and then someone brings up a
counterargument, ad infinitum.