[HarfBuzz] Sinhala split matra

[Harshula]

Hi Behdad,

On Wed, 2012-11-14 at 12:01 -0800, Behdad Esfahbod wrote:
> I reiterate what I've said many many times on this list already.  Hopefully
> this puts an end to this particular topic.  More below.

Are you referring to your mantra, "I consider the fonts broken if they
don't work with Uniscribe." ?
 
> On 12-11-13 06:15 PM, Harshula wrote:
> > Hi Behdad & Jonathan,
> > 
> > On Tue, 2012-11-13 at 12:40 -0800, Behdad Esfahbod wrote:
> >> Jonathan and I decided to workaround the Sinhala issue with split matras and
> >> free fonts, such that they continue working with new HarfBuzz.  That's in
> >> master now.
> > 
> > 1) GNU Free Font maintainer, Steve, went to a lot of effort to make
> > FreeSerif work with Uniscribe and the Harfbuzz behaviour we agreed to in
> > September.
> > (http://lists.freedesktop.org/archives/harfbuzz/2012-September/002458.html). See SVN version of FreeSerif. He is also working hard to do a release soon.
> 
> Good that he did.  Because in general we don't care about supporting fonts
> that are broken with Uniscribe.

That contradicts the intent of Commit
0736915b8ed789a209205fec762997af3a8af89c ([Indic] Decompose Sinhala
split matras the way old HarfBuzz / Pango did). So what makes this an
exception to the rule? In practice, this Commit adds support for 1 font
that is incompatible with Uniscribe that I no longer maintain.

> > 2) You refer to "free fonts". Are you referring to GNU Free Font or to
> > the two GPL'd fonts that support Sinhala? One of them is deprecated,
> > there are problems with glyphs etc, and the other, FreeSerif, that is
> > meant to replace the former works well with Harfbuzz from a week ago.
> 
> I meant fonts that were designed to work with the Free Software shaping
> engines (old HarfBuzz, Pango, ICU LayoutEngine).

There are two such fonts. One of which is deprecated and I've stopped
maintaining it. So you've fixed a 'problem' that doesn't exist once the
next release of FreeFont ships.

> > 3) The big win, from the behaviour agreed to in September, is that any
> > Windows font will just work with Harfbuzz. With the FreeSerif changes,
> > that meant that all maintained fonts just work. From the perspective of
> > actual Linux Sinhala script users, that increases their font options
> > significantly.
> 
> What we decided to do yesterday allows for both set of fonts to work with
> harfbuzz-ng.  This is a superset of the fonts that HarfBuzz before yesterday
> could render correctly, and a superset of fonts that worked with old HarfBuzz
> / Pango, and a superset of fonts that work with Uniscribe.  So it can't be worse.

How exactly is an end-user, for example using GNOME 3.6, going to be
able to use a superset of fonts if Harfbuzz needs an environment flag
HB_OT_INDIC_OPTIONS=uniscribe-bug-compatible in-order for fonts made for
Uniscribe to work? When the flag is *set* or *not set*, under each of
these conditions only a strict subset will work. How can a superset of
fonts work at the same time?

I'd be very happy if an end-user could use a superset of fonts
encompassing those made for Pango/ICU and Uniscribe with Harfbuzz. But
that does not appear to be the case in reality.

> > 4) Commit 0736915b8ed789a209205fec762997af3a8af89c ([Indic] Decompose
> > Sinhala split matras the way old HarfBuzz / Pango did) has a bug. The
> > five fonts I test with do not render correctly now. I suspect no fonts
> > render correctly with current TOT Harfbuzz. i.e. A regression. Try කේ
> > (<U+0d9a><U+0dda>).
> 
> Ouch.  Fix pushed out.  We will make another release at the end of the week.

Tested. Harfbuzz at least now works for some Sinhala fonts. Thanks.

> > 5) Since this is a FOSS project, any chance you and Jonathon could
> > elaborate a bit on the problem you guys are trying to fix and the
> > reasoning behind the decision to overturn what was agreed to in
> > September?
> 
> I already explained the problem in response to your email back in July:
> 
>   http://lists.freedesktop.org/archives/harfbuzz/2012-July/002261.html

That was 25 July 2012, you said:
------------------------------------------
"The new shaper follows Uniscribe here.   You can disable that by removing a
few lines from hb-unicode.cc.  Just search for DDC.  That would make the new
shaper match HarfBuzz-old.  Though, I consider the fonts broken if they don't
work with Uniscribe.

So, in this particular case, it may make sense to limit the Uniscribe behavior
to the uniscribe-bug-compatibilty mode.  Jonathan, what do you think?"
------------------------------------------

You nor Jonathan responded to that email to indicate the direction you
had decided. So I followed it up on 3 September 2012
(http://lists.freedesktop.org/archives/harfbuzz/2012-September/002450.html) to which you responded (http://lists.freedesktop.org/archives/harfbuzz/2012-September/002458.html):
------------------------------------------
> I have been considering this conundrum and here are my thoughts:
> 
> 1) Ideally any Unicode Sinhala font that 'works' with Uniscribe should
> just 'work' on GNU/Linux.

Correct.  That will be the case with new HarfBuzz Indic shaper.


> 2) I tested a few freely (cost) available fonts that used to be or are
> now reference fonts. Two of these work correctly with both the *new* and
> *old* shapers.

It's definitely possible to develop a font that works with both.


> 3) Taking (2) into account, it may be possible for existing GNU/Linux
> fonts to support both the new and old shaper. However, I would have to
> defer that to someone who knows their way around fonts better.

When you find someone who wants to fix the font, let me know and I can tell
them exactly what to do.


> 4) The way Uniscribe has implemented this is a bit unexpected, what
> happens if Uniscribe changes/fixes this? Do we just blindly follow
> Uniscribe?

In absence of any standard, yes.  That said, Microsoft has never changed those
kinds of things in a way that would break things.


behdad
------------------------------------------

Hence, in September you clearly communicated that HarfBuzz intends to
follow Uniscribe's behaviour on this issue. However, in November without
any communication you committed a patch that overturned that decision.

You've set a rule for Harfbuzz, that you have repeated a number of
times, that you consider fonts that do not work with Uniscribe to be
broken. In this case you have made an exception to your own rule for a
font that is deprecated and I no longer maintain. Can you please explain
why? I am advocating that the Uniscribe behaviour be retained in
Harfbuzz on this issue as it was one week ago.

By the lack of transparency, in particular not communicating/discussing
direction changes, you are making it harder for those trying to support
HarfBuzz. As HarfBuzz is more widely deployed, this will become a bigger
problem.

cya,
#