[FriBidi] log2vis() misbehaving with Arabic text?

[Behdad Esfahbod]

Hi Philip,

Comments below.

On 14-10-21 11:42 AM, Philip Semanchuk wrote:
> Hi all,
> I’m a non-Arabic speaker working with Arabic text, and I sometimes see results
> from log2vis() that don’t look quite right.
> 
> I have a test where I compare the results from two Arabic text reshapers. The
> first is pyfribidi which is a thin wrapper around fribidi-0.19.6. The second
> reshaper is a Python port of this:
> https://github.com/agawish/Better-Arabic-Reshaper/
> 
> The two reshapers almost always agree on how to reshape a logical word.
> However, in some cases they don’t. For instance, when given this logical string:
> u'\u062c\u0627\u0630\u0628\u064a\u0651\u0629'
> 
> log2vis() puts the Shadda in a different place than the BAR
> (Better-Arabic-Reshaper):
> log2vis: u’\ufe94\ufef4\u0651\ufe91\ufeab\ufe8e\ufe9f'
> bar:     u’\ufe94\u0651\ufef4\ufe91\ufeab\ufe8e\ufe9f'
> 
> When I paste quoted versions of the visual representation of these strings
> into Google, Google finds 23 instances of the fribidi version and ~37k
> versions of the BAR version. To me, that’s a pretty strong argument that the
> fribidi version is incorrect.
> 
> I’m happy to file a bug if that’s appropriate, but I’d much rather learn that
> I’m simply misusing the library or making some other mistake. Can someone help
> me understand what I'm doing wrong?

Which one is correct depends on how you are going to use the results.  The
rule in question is written down here:

  http://www.unicode.org/reports/tr9/#L3

If you want, for example, to output this sequence to a non-bidi-aware
terminal, then the result that FriBidi is creating is correct and the BAR is
incorrect.

Looking at the BAR code, I'm much more confident in FriBidi being correct than
in BAR.


> Thanks in advance
> Philip
> 
> 
> _______________________________________________
> fribidi mailing list
> fribidi at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/fribidi
> 

-- 
behdad
http://behdad.org/