[FriBidi] Incorrect Arabic text rendering in maperver

[Ian Walberg]

Behdad,

I have taken a look at the code and from what I can see the relavant part of the code is below.

Can I make a test modification to this to test the cause of the problem?

Thanks

Ian



for (i = 0; i < len - 1; i++) {
    register FriBidiChar c;
    if (FRIBIDI_LEVEL_IS_RTL(embedding_levels[i]) &&
        embedding_levels[i] == embedding_levels[i+1] &&
        (c = PAIR_MATCH(table, size, str[i], str[i+1])))
      {
        str[i] = FRIBIDI_CHAR_FILL; */
        FRIBIDI_SET_BITS(ar_props[i], FRIBIDI_MASK_LIGATURED);
        str[i+1] = c; */
      }
  }


-----Original Message-----
From: Behdad Esfahbod [mailto:behdad.esfahbod at gmail.com] On Behalf Of Behdad Esfahbod
Sent: Monday, January 17, 2011 6:56 AM
To: Stephen Woodbridge
Cc: Ian Walberg; fribidi at lists.freedesktop.org
Subject: Re: [FriBidi] Incorrect Arabic text rendering in maperver

Ok, I actually have an answer!

Just filter out the character FRIBIDI_CHAR_FILL out of the output character
stream.  When FriBidi forms a ligature, it stuffs in FRIBIDI_CHAR_FILL
characters in instead.  That character is actually the Unicode ZeroWidth
NoBreak Space, so if your font has a glyph for it, there's no problem at all,
otherwise you get that box.  It's safe to not render that character.

Hope that helps.

behdad

PS.  Have you considered using Pango in mapserver?

On 01/17/11 09:40, Stephen Woodbridge wrote:
> Hi Behdad,
> 
> In mapserver, mapserver.org, all strings are run through the following function:
> 
> On my linux box, assume:
> #define USE_FRIBIDI
> #undef FRIBIDI_NO_CHARSETS
> 
> I am assuming on windows builds:
> #define USE_FRIBIDI
> #define FRIBIDI_NO_CHARSETS
> 
> Ian, are you using Windows or Linux?
> 
> After the string is encoded, it is rendered using freetype and hopefully an
> appropriate font. I have had clients on Linux use this successfully except
> that most fonts do not seem to have all the joining characters. Missing
> characters typically showed up as a square box or just unjoined characters.
> They have been able to get good results by finding a font with more of the
> joining characters included and in some cases by using a font editor to add
> the missing glyphs. I'm not sure if this is Ian's problem or not.
> 
> Also I would be interested in knowing of any good truetype fonts that work
> well with Fribidi for Arabic, especially any free ones.
> 
> Best regards,
>   -Steve
> 
> #ifdef USE_FRIBIDI
> char *msGetFriBidiEncodedString(const char *string, const char *encoding)
> {
>   FriBidiChar logical[MAX_STR_LEN];
>   FriBidiCharType base = FRIBIDI_TYPE_ON;
>   size_t len;
> 
> #ifdef FRIBIDI_NO_CHARSETS
>   iconv_t to_ucs4, from_ucs4;
> #else
>   int to_char_set_num;
>   int from_char_set_num;
> #endif
> 
>   len = strlen(string);
> 
> #ifdef FRIBIDI_NO_CHARSETS
>   to_ucs4 = iconv_open ("WCHAR_T", encoding);
>   from_ucs4 = iconv_open ("UTF-8", "WCHAR_T");
> #else
>   to_char_set_num = fribidi_parse_charset ((char*)encoding);
>   from_char_set_num = fribidi_parse_charset ("UTF-8");
> #endif
> 
> #ifdef FRIBIDI_NO_CHARSETS
>   if (to_ucs4 == (iconv_t) (-1) || from_ucs4 == (iconv_t) (-1))
> #else
>   if (!to_char_set_num || !from_char_set_num)
> #endif
>   {
>     msSetError(MS_IDENTERR, "Encoding not supported (%s).",
>                "msGetFriBidiEncodedString()", encoding);
>     return NULL;
>   }
> 
> #ifdef FRIBIDI_NO_CHARSETS
>   {
>     char *st = string, *ust = (char *) logical;
>     int in_len = (int) len;
>     len = sizeof logical;
>     iconv (to_ucs4, &st, &in_len, &ust, (int *) &len);
>     len = (FriBidiChar *) ust - logical;
>   }
> #else
>   len = fribidi_charset_to_unicode (to_char_set_num, (char*)string, len,
> logical);
> #endif
> 
>   {
>     FriBidiChar *visual;
>     char outstring[MAX_STR_LEN];
>     FriBidiStrIndex *ltov, *vtol;
>     FriBidiLevel *levels;
>     FriBidiStrIndex new_len;
>     fribidi_boolean log2vis;
> 
>     visual = (FriBidiChar *) malloc (sizeof (FriBidiChar) * (len + 1));
>     ltov = NULL;
>     vtol = NULL;
>     levels = NULL;
> 
>     /* Create a bidi string. */
>     log2vis = fribidi_log2vis (logical, len, &base,
>        /* output */
>        visual, ltov, vtol, levels);
> 
>     if (!log2vis) {
>       msSetError(MS_IDENTERR, "Failed to create bidi string.",
>              "msGetFriBidiEncodedString()");
>       return NULL;
>     }
> 
>     new_len = len;
> 
>     /* Convert it to utf-8 for display. */
> #ifdef FRIBIDI_NO_CHARSETS
>     {
>       char *str = outstring, *ust = (char *) visual;
>       int in_len = len * sizeof visual[0];
>       new_len = sizeof outstring;
>       iconv (from_ucs4, &ust, &in_len, &str, (int *) &new_len);
>       *str = '\0';
>       new_len = str - outstring;
>      }
> #else
>      new_len =
>        fribidi_unicode_to_charset (from_char_set_num,
>            visual, len, outstring);
> #endif
>      return strdup(outstring);
>   }
> }
> #endif
> 
> 
> On 1/17/2011 8:48 AM, Behdad Esfahbod wrote:
>> And how are you calling FriBidi exactly?
>>
>> On 01/15/11 11:53, Ian Walberg wrote:
>>> As requested here are the images of the problem we are seeing and the text
>>> string that is not working.
>>>
>>>
>>>
>>> This is UniversOTOArabic:- http://i52.tinypic.com/156s5ea.jpg
>>>
>>>
>>>
>>> This is ArialUni :- http://i54.tinypic.com/25ouses.jpg
>>>
>>>
>>>
>>> This is andale and as far as we can tell is correct
>>> :-http://i53.tinypic.com/hx56kn.jpg
>>>
>>>
>>>
>>> And in English :- http://i54.tinypic.com/i1jkmb.jpg
>>>
>>>
>>>
>>> arabic.htm has the Arabic text for Alexandria in it.
>>>
>>>
>>>
>>> Arabic Issues 01.xls has the English and Arabic for Alexandria.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Ian
>>>
>>>
>>>
>>>
>>>
>>> *From:*Ian Walberg
>>> *Sent:* Wednesday, January 05, 2011 8:02 PM
>>> *To:* fribidi at lists.freedesktop.org
>>> *Subject:* Incorrect Arabic text rendering in maperver
>>>
>>>
>>>
>>> Hello list and Happy New Year,
>>>
>>>
>>>
>>> I have posted this on the mapserver list and our current assumption is this is
>>> an issue related to fribidi and the font we are using.
>>>
>>>
>>>
>>> We are seeing incorrectly rendered Arabic text, typically a 'square' displayed
>>> instead of one of the characters or sometimes one or more characters missing.
>>>
>>>
>>>
>>> The data is coming from shape files and most the names appear to be displayed
>>> correctly.
>>>
>>>
>>>
>>> This is being seen on both ms4w and the target Linux installation.
>>>
>>>
>>>
>>> Any idea where to look would be greatly appreciated. Is the 'square' character
>>> what Freetype displays if it cannot find a character in the font.
>>>
>>>
>>>
>>> Specifically we get the issue with the Arabic text for the city Alexandria (in
>>> Egypt) with the font UniversOTSArabic.ttf and ArialUni.ttf. Andaltewt.ttf
>>> works ok.
>>>
>>>
>>>
>>> Many thanks
>>>
>>>
>>>
>>> Ian
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> fribidi mailing list
>>> fribidi at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/fribidi
>> _______________________________________________
>> fribidi mailing list
>> fribidi at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/fribidi
> 
>