<div dir="auto">Counter proposal: just continue to use the underscore locale variant and save a lot of heartache...? R</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 6 Apr 2023, 17:15 Matthias Klumpp, <<a href="mailto:matthias@tenstral.net">matthias@tenstral.net</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi! While investigating why zh_TW/zh_CN translations were not showing up in AppStream-based software centers, we found out that this was due to the locale being listed as "zh-TW" in the XML. This was first noticed at KDE, which was going to edit their tools to use POSIX locale in the XML instead[1]. I looked into the matter and found out that when using XML, the contents of the xml:lang tag are not arbitrary or UNIX locale though, but need to follow the IETF BCP47 specification. So by assuming POSIX locale, AppStream was doing the wrong thing for a really long time! We only didn't notice this so far for two reasons: One, AppStream's locale matching is quite good and will fall back to country codes if a dedicated translation wasn't found, which coincidentally is where POSIX and BCP47 locale are pretty much identical. So the issue wasn't as noticeable unless you were from a language depending on the territory specifier. Secondly though, GNOME's tooling is also wrong in many cases and seems to be using POSIX locale while BCP47 locale should be used. I thought quite a while about this, and I think the best worst thing to do here is to make AppStream use BCP47 locale, with hopefully not too much breakage.. I hate making this change (it complicates locale handling in AppStream quite a bit, especially since BCP47/POSIX locale can't easily be mapped 1:1 (see[3])), but I think doing it is the most sane change. Just ignoring the IETF specification and having POSIX locale there seemed attractive at first, but a lot of tools that translate XML do not handle this well and will continue to output BCP47 locale, such as the ones used by KDE and itstool. So even if KDE would switch, we would set a trap for many other projects using alternative translation solutions, with no easy way for projects to solve this. Making this change will cause problems for projects which also did it "the wrong way", but on the other hand it will fix a real bug for projects which were using BCP47 all along. Since there is a specification for this, I think not following established practices for XML is a pretty bad choice. So, I implemented a locale mapping algorithm in AppStream that translates POSIX to BCP47 based on the same rules that itstool[4] uses to do the same task. That will work for pretty much all cases, I hope. AppStream will also do its best to find good locale in case someone did use POSIX in xml:lang, but for performance reasons I can't implement anything that just accepts both locale - that would not only be a lot of engineering work for little gain, but it would also slow down parsing for everyone. I may adjust appstream-compose though the correct wrong locale, which should also fix the issue for people before it reaches users. All of the changes are not yet merged into master, but I intend to do that soon, unless there are objections or feedback that I haven't considered. So, what do you think? It's a pretty bad situation to be in, but it needs to be addressed somehow. Cheers, Matthias P.S: I think POSIX locale are better than BCP47, their format is just a lot more versatile and expressive and can easily be split into individual language/territory/modifier parts, where it's less clear for BCP47. I wish we would have only one format to identify locale, but that ship sailed in 1995/2001. [1]: <a href="https://invent.kde.org/sysadmin/l10n-scripty/-/merge_requests/61" rel="noreferrer noreferrer" target="_blank">https://invent.kde.org/sysadmin/l10n-scripty/-/merge_requests/61</a> [2]: <a href="https://en.wikipedia.org/wiki/IETF_language_tag" rel="noreferrer noreferrer" target="_blank">https://en.wikipedia.org/wiki/IETF_language_tag</a> [3]: <a href="https://wiki.openoffice.org/wiki/LocaleMapping" rel="noreferrer noreferrer" target="_blank">https://wiki.openoffice.org/wiki/LocaleMapping</a> [4]: <a href="https://itstool.org/" rel="noreferrer noreferrer" target="_blank">https://itstool.org/</a> -- I welcome VSRE emails. See <a href="http://vsre.info/" rel="noreferrer noreferrer" target="_blank">http://vsre.info/</a> </blockquote></div>