[FriBidi] Incorrect Arabic text rendering in maperver

[Ian Walberg]

Steve,

We see the issue on both Windows and Linux.

Regards

Ian

-----Original Message-----
From: Stephen Woodbridge [mailto:woodbri at swoodbridge.com] 
Sent: Monday, January 17, 2011 6:40 AM
To: Behdad Esfahbod
Cc: Ian Walberg; fribidi at lists.freedesktop.org
Subject: Re: [FriBidi] Incorrect Arabic text rendering in maperver

Hi Behdad,

In mapserver, mapserver.org, all strings are run through the following 
function:

On my linux box, assume:
#define USE_FRIBIDI
#undef FRIBIDI_NO_CHARSETS

I am assuming on windows builds:
#define USE_FRIBIDI
#define FRIBIDI_NO_CHARSETS

Ian, are you using Windows or Linux?

After the string is encoded, it is rendered using freetype and hopefully

an appropriate font. I have had clients on Linux use this successfully 
except that most fonts do not seem to have all the joining characters. 
Missing characters typically showed up as a square box or just unjoined 
characters. They have been able to get good results by finding a font 
with more of the joining characters included and in some cases by using 
a font editor to add the missing glyphs. I'm not sure if this is Ian's 
problem or not.

Also I would be interested in knowing of any good truetype fonts that 
work well with Fribidi for Arabic, especially any free ones.

Best regards,
   -Steve

#ifdef USE_FRIBIDI
char *msGetFriBidiEncodedString(const char *string, const char
*encoding)
{
   FriBidiChar logical[MAX_STR_LEN];
   FriBidiCharType base = FRIBIDI_TYPE_ON;
   size_t len;

#ifdef FRIBIDI_NO_CHARSETS
   iconv_t to_ucs4, from_ucs4;
#else
   int to_char_set_num;
   int from_char_set_num;
#endif

   len = strlen(string);

#ifdef FRIBIDI_NO_CHARSETS
   to_ucs4 = iconv_open ("WCHAR_T", encoding);
   from_ucs4 = iconv_open ("UTF-8", "WCHAR_T");
#else
   to_char_set_num = fribidi_parse_charset ((char*)encoding);
   from_char_set_num = fribidi_parse_charset ("UTF-8");
#endif

#ifdef FRIBIDI_NO_CHARSETS
   if (to_ucs4 == (iconv_t) (-1) || from_ucs4 == (iconv_t) (-1))
#else
   if (!to_char_set_num || !from_char_set_num)
#endif
   {
     msSetError(MS_IDENTERR, "Encoding not supported (%s).",
                "msGetFriBidiEncodedString()", encoding);
     return NULL;
   }

#ifdef FRIBIDI_NO_CHARSETS
   {
     char *st = string, *ust = (char *) logical;
     int in_len = (int) len;
     len = sizeof logical;
     iconv (to_ucs4, &st, &in_len, &ust, (int *) &len);
     len = (FriBidiChar *) ust - logical;
   }
#else
   len = fribidi_charset_to_unicode (to_char_set_num, (char*)string, 
len, logical);
#endif

   {
     FriBidiChar *visual;
     char outstring[MAX_STR_LEN];
     FriBidiStrIndex *ltov, *vtol;
     FriBidiLevel *levels;
     FriBidiStrIndex new_len;
     fribidi_boolean log2vis;

     visual = (FriBidiChar *) malloc (sizeof (FriBidiChar) * (len + 1));
     ltov = NULL;
     vtol = NULL;
     levels = NULL;

     /* Create a bidi string. */
     log2vis = fribidi_log2vis (logical, len, &base,
        /* output */
        visual, ltov, vtol, levels);

     if (!log2vis) {
       msSetError(MS_IDENTERR, "Failed to create bidi string.",
              "msGetFriBidiEncodedString()");
       return NULL;
     }

     new_len = len;

     /* Convert it to utf-8 for display. */
#ifdef FRIBIDI_NO_CHARSETS
     {
       char *str = outstring, *ust = (char *) visual;
       int in_len = len * sizeof visual[0];
       new_len = sizeof outstring;
       iconv (from_ucs4, &ust, &in_len, &str, (int *) &new_len);
       *str = '\0';
       new_len = str - outstring;
      }
#else
      new_len =
        fribidi_unicode_to_charset (from_char_set_num,
            visual, len, outstring);
#endif
      return strdup(outstring);
   }
}
#endif


On 1/17/2011 8:48 AM, Behdad Esfahbod wrote:
> And how are you calling FriBidi exactly?
>
> On 01/15/11 11:53, Ian Walberg wrote:
>> As requested here are the images of the problem we are seeing and the
text
>> string that is not working.
>>
>>
>>
>> This is UniversOTOArabic:- http://i52.tinypic.com/156s5ea.jpg
>>
>>
>>
>> This is ArialUni :- http://i54.tinypic.com/25ouses.jpg
>>
>>
>>
>> This is andale and as far as we can tell is correct
>> :-http://i53.tinypic.com/hx56kn.jpg
>>
>>
>>
>> And in English :- http://i54.tinypic.com/i1jkmb.jpg
>>
>>
>>
>> arabic.htm has the Arabic text for Alexandria in it.
>>
>>
>>
>> Arabic Issues 01.xls has the English and Arabic for Alexandria.
>>
>>
>>
>> Thanks
>>
>>
>>
>> Ian
>>
>>
>>
>>
>>
>> *From:*Ian Walberg
>> *Sent:* Wednesday, January 05, 2011 8:02 PM
>> *To:* fribidi at lists.freedesktop.org
>> *Subject:* Incorrect Arabic text rendering in maperver
>>
>>
>>
>> Hello list and Happy New Year,
>>
>>
>>
>> I have posted this on the mapserver list and our current assumption
is this is
>> an issue related to fribidi and the font we are using.
>>
>>
>>
>> We are seeing incorrectly rendered Arabic text, typically a 'square'
displayed
>> instead of one of the characters or sometimes one or more characters
missing.
>>
>>
>>
>> The data is coming from shape files and most the names appear to be
displayed
>> correctly.
>>
>>
>>
>> This is being seen on both ms4w and the target Linux installation.
>>
>>
>>
>> Any idea where to look would be greatly appreciated. Is the 'square'
character
>> what Freetype displays if it cannot find a character in the font.
>>
>>
>>
>> Specifically we get the issue with the Arabic text for the city
Alexandria (in
>> Egypt) with the font UniversOTSArabic.ttf and ArialUni.ttf.
Andaltewt.ttf
>> works ok.
>>
>>
>>
>> Many thanks
>>
>>
>>
>> Ian
>>
>>
>>
>>
>>
>> _______________________________________________
>> fribidi mailing list
>> fribidi at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/fribidi
> _______________________________________________
> fribidi mailing list
> fribidi at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/fribidi