Hebrew Characters Support

[dmorissette]

In bug 858 we added multibyte encoding support, but it seems that this doesn't
work for languages written from right to left such as Hebrew:

Danny Stopel (Anywhere Ltd) wrote:
> Hello again Daniel!
> 
> Sorry to disturb you, but I succeeded to download binaries of MS4.3
> (from http://www.maptools.org/dl/)
> I also succeeded to display Hebrew, but backwards :(
> Hebrew is a right to left language. I've tried all possible encodings
> from 
> http://www.gnu.org/software/libiconv/
> but none of them worked right. The only thing I need is some property of
> label to reverse the string before displaying it. Is such thing exists
> in MS?
> 
> Sorry again for disturbing you and thanks for help!
> 
> Danny Stopel

[dmorissette]

Um... seems that it won't be a simple one to deal with. Iconv's website says
nothing about right-to-left vs left-to-right. Just to make things worse, Hebrew
characters go from right to left, but numbers and english words embedded into
the text still go right to left and need to be taken into account.

Here are a few resources I found on the web:

Intro to computing in Hebrew:
http://www.multilingual.com/FMPro?-db=archives&-format=ad%2fselected%5fresults.html&-lay=cgi&-sortfield=magazine%20number&-sortorder=descend&-op=eq&Ad%20Type=reprint&-max=1817529521&-recid=33256&-token=%5bFMP-currenttoken%5d&-find=

The Hebrew HOWTO:
http://www.ibiblio.org/mdw/HOWTO/Hebrew-HOWTO.html

IGLU FAQ (The Israeli Group of Linux Users)
http://iglu.org.il/faq/cache/8.html

[dmorissette]

Found the GNU FriBidi library, it seems that this library would be able to
convert a bidirectional string from logical to visual character ordering for
graphical presentation:
http://freedesktop.org/Software/FriBidi

[dmorissette]

Okay, more news: it seems that freetype 2.1.2 and more recent adds OpenType
Support which might be the simple solution to our current problem:
http://freetype.sourceforge.net/opentype/index.html

The maptools windows binary that Danny used was built with Freeetype 2.1.3, so
it should include this openType stuff if the information that I found is right.
Maybe the solution is simply to use an font that includes this OpenType stuff?

I don't have time to research this any further for now. Danny, can you please
try to see if you can find an Open Type font for Hebrew and test it with the
MapServer binary that you got? (Then please report your findings to this bug).

[dannys@…]

Weel, I have tried two OpenType fonts Cardo and SIL Hebrew font.
All I did is used them in a regular way in with MapServer with few different 
encodings, (HEBREW,ISO-8859-8, ..) I've also tried few Unicode encodings...
In Unicode encodings I get strange characters, in Hebrew encoding I get good 
character but backwards, like before.
I actually didn't realized why OpenFont is different than any other TT font in 
MapServer, is it some new way I can use OpenFont in MapServer to get new 
results (reversed text)? 
Anyway, thank you Daniel for your care :)

[dmorissette]

Danny Stopel (Anywhere Ltd) wrote:
> 
> I succeeded to show Hebrew by reversing the characters of the string in 
> my database.
> 
> So, the problem of Hebrew is solved, at least for me.
> 

Setting target milestone to FUTURE... we may revisit this in a future version.

Note that we can't do anything without some good way to test. It would be great
if Danny (or anyone) could attach to this bug a sample mapfile + data + fonts
and some screenshots showing the expected result.

[eshabtai@…]

Is there any updates on this bug?

It seems that Hebrew text is still displayed in the wrong direction.

[dmorissette]

No developments yet, and can't really work on this until someone attaches sample
mapfile + data + fonts and some screenshots showing the expected result

[eshabtai@…]

Quick fix for bidi support

[eshabtai@…]

I will upload some hello world example whith some Hebrew text so you can play
with (as soon as I can).

[eshabtai@…]

A simple map file which displays Hebrew text

[eshabtai@…]

An hebrew font from ​http://culmus.sourceforge.net/

[eshabtai@…]

fontset file required for the map file

[eshabtai@…]

The expected output (correct output).

[eshabtai@…]

I have uploaded all the required files to generate a simple Hebrew text.
correct.gif shows how the text should appear. Currently mapserver renders the
text in the wrong direction. When applying the patch I attached to mapserver it
will render the text correctly.

I'll be happy to wrap this one myself, but I need some review on the code I added.

[dmorissette]

Let's try to get this one done in 5.2

[dmorissette]

I had received the following email about the original patch. I will attach Endre's patch as well that fixes this issue:

Subject: MapServer Hebrew Characters Support
From: Piret Endre
Date: Mon, 12 Feb 2007 16:51:48 +0100
To:dmorissette@...

Daniel,

I had to deal with some Hebrew maps in MapServer. I found your
correspondence with Ehud Shabtai and his patch, it helped me a lot.
But the patch makes impossible to display labels with encodings not
supported by FriBidi, like Windows-1250 and so on. I have to diplay
both Hebrew and European texts, so I changed the patch a bit. It
requires the encoding to be specified in the mapfile, so one needs to
add an ENCODING "UTF-8" line to Ehud's mapfile. After this addition it
gives the same result as with the original patch, while my European
labels are still working.

Best Regards
Endre

[dmorissette]

Endre's patch for the unsupported encoding problem (e.g. Windows-1250)

[dmorissette]

AlanB has ported and tested the patches with MapServer 5.1 (trunk) and made the required configure and Makefile.in changes. I have reviewed and committed them in r7457.

Notes/Limitations:

• Not tested on Windows yet (at a minimum makefile.vc needs an update)
• Character "joining" not supported yet, this will be done in ticket #2549

[dmorissette]

Assefa, could you please test this on Windows and update the Makefile.vc, and then mark the ticket as closed? Thanks.

[assefa]

which version of the fribidi library is used?

I have downloaded the fribidi-0.10.9.tar.gz and the CVS version but when compiling, I get errors in mapstring.c such as 'fribidi_log2vis' : too few arguments for call. The first argument seems to be missing from the call comapred to the header of this function that I have RIBIDI_API fribidi_boolean fribidi_log2vis (FriBidiEnv *fribidienv,

/* input */ const FriBidiChar *str, FriBidiStrIndex len, FriBidiCharType *pbase_dirs, /* output */ FriBidiChar *visual_str, FriBidiStrIndex *position_L_to_V_list, FriBidiStrIndex *position_V_to_L_list, FriBidiLevel *embedding_level_list);

Note that by setting the first argument to NULL, I was able to build.

BTW I was not able to have the test sample working (I mean to see the hebrew characters), is it possible to provide a zip containing the full set of map/font for testing (maybe I have screwed up the sample data when I downloaded it from the attachments and I would like to give it another shot)

I will wait until the header issue is resolved and commit the windows makefiles.

[woodbri]

FYI:

From ​http://fribidi.freedesktop.org/wiki/

There is a release of the fribidi2 modules available now, which is ​http://fribidi.org/download/fribidi-0.19.1.tar.gz (md5sum, signed by Behdad Esfahbod) from January 15, 2008. This release is based on Unicode 5.0.0 character database too and is supposed to be fully API/ABI compatible with the 0.10 series and may need some testing before fully replacing the 0.10 series.

Given that this is supposed to be API/ABI compatible with the 0.10 series AND it is supposed to have support for character joining, it might make sense to consider just upgrading to this version of the library and dropping the patches. This is covered under ticket #2549.

[aboudreault]

Assefa,

The version that is supported in the MapServer patch is 0.10.9. I tryed the version 0.19.1 but no success. Both version are supposed to be compatible but i think we will need to modify the MapServer patch in mapstring.c. Is your fribidi compile correctly? If not, you can checkthe fribidi package that came with Abiword source. It contain a "win" folder that is for building fribidi under MSVC. ​http://www.abisource.com/downloads/abiword/2.4.6/source/abiword-2.4.6.tar.gz

btw, i didnt see anywhere the argument of log2vis function "FriBidiEnv? *fribidienv," in the version 0.10.9. Your probably using the CVS version for have this header? For your information, the only warning that i get during compilation with gcc is: mapstring.c: In function ‘msConvertWideStringToUTF8’: mapstring.c:1270: warning: dereferencing type-punned pointer will break strict-aliasing rules but it's not related to the log2vis function call.

Alan

[assefa]

Alan,

I was using cvs, you are right the header of 0.10.9 builds properly with Mapserver.

I am building fribidi locally with a local makefile since I was not able to fully build the library using the MSVC project files.

If possible can you send me directly a zip containing your test map/font. For some reason I am not able to view the Hebrew characters properly.

Committed windows build files r7477

[dmorissette]

FYI we had to add ENCODING "UTF-8" to the LABEL object in hebrew.map to make it work.

[assefa]

This was tested on windows with version 0.19.1 and makefiles have been updated. Closing.