Opened 15 years ago

Closed 11 years ago

#956 closed defect (fixed)

Hebrew Characters Support

Reported by: dmorissette Owned by: assefa
Priority: high Milestone: 5.2 release
Component: MapServer C Library Version: 4.3
Severity: normal Keywords:
Cc: dannys@…, eshabtai@…, aboudreault, woodbri

Description (last modified by dmorissette)

In bug 858 we added multibyte encoding support, but it seems that this doesn't
work for languages written from right to left such as Hebrew:

Danny Stopel (Anywhere Ltd) wrote:
> Hello again Daniel!
> 
> Sorry to disturb you, but I succeeded to download binaries of MS4.3
> (from http://www.maptools.org/dl/)
> I also succeeded to display Hebrew, but backwards :(
> Hebrew is a right to left language. I've tried all possible encodings
> from 
> http://www.gnu.org/software/libiconv/
> but none of them worked right. The only thing I need is some property of
> label to reverse the string before displaying it. Is such thing exists
> in MS?
> 
> Sorry again for disturbing you and thanks for help!
> 
> Danny Stopel

Attachments (6)

mapserver_bidi.patch (3.9 KB) - added by eshabtai@… 14 years ago.
Quick fix for bidi support
hebrew.map (420 bytes) - added by eshabtai@… 13 years ago.
A simple map file which displays Hebrew text
DavidCLM-Bold.pfa (60.7 KB) - added by eshabtai@… 13 years ago.
An hebrew font from http://culmus.sourceforge.net/
fontset.txt (24 bytes) - added by eshabtai@… 13 years ago.
fontset file required for the map file
correct.gif (932 bytes) - added by eshabtai@… 13 years ago.
The expected output (correct output).
mapserver-bidi.patch (3.3 KB) - added by dmorissette 11 years ago.
Endre's patch for the unsupported encoding problem (e.g. Windows-1250)

Download all attachments as: .zip

Change History (26)

comment:1 Changed 15 years ago by dmorissette

Status: newassigned
Um... seems that it won't be a simple one to deal with. Iconv's website says
nothing about right-to-left vs left-to-right. Just to make things worse, Hebrew
characters go from right to left, but numbers and english words embedded into
the text still go right to left and need to be taken into account.

Here are a few resources I found on the web:

Intro to computing in Hebrew:
http://www.multilingual.com/FMPro?-db=archives&-format=ad%2fselected%5fresults.html&-lay=cgi&-sortfield=magazine%20number&-sortorder=descend&-op=eq&Ad%20Type=reprint&-max=1817529521&-recid=33256&-token=%5bFMP-currenttoken%5d&-find=

The Hebrew HOWTO:
http://www.ibiblio.org/mdw/HOWTO/Hebrew-HOWTO.html

IGLU FAQ (The Israeli Group of Linux Users)
http://iglu.org.il/faq/cache/8.html

comment:2 Changed 15 years ago by dmorissette

Found the GNU FriBidi library, it seems that this library would be able to
convert a bidirectional string from logical to visual character ordering for
graphical presentation:
http://freedesktop.org/Software/FriBidi

comment:3 Changed 15 years ago by dmorissette

Okay, more news: it seems that freetype 2.1.2 and more recent adds OpenType
Support which might be the simple solution to our current problem:
http://freetype.sourceforge.net/opentype/index.html

The maptools windows binary that Danny used was built with Freeetype 2.1.3, so
it should include this openType stuff if the information that I found is right.
Maybe the solution is simply to use an font that includes this OpenType stuff?

I don't have time to research this any further for now. Danny, can you please
try to see if you can find an Open Type font for Hebrew and test it with the
MapServer binary that you got? (Then please report your findings to this bug).

comment:4 Changed 15 years ago by dannys@…

Weel, I have tried two OpenType fonts Cardo and SIL Hebrew font.
All I did is used them in a regular way in with MapServer with few different 
encodings, (HEBREW,ISO-8859-8, ..) I've also tried few Unicode encodings...
In Unicode encodings I get strange characters, in Hebrew encoding I get good 
character but backwards, like before.
I actually didn't realized why OpenFont is different than any other TT font in 
MapServer, is it some new way I can use OpenFont in MapServer to get new 
results (reversed text)? 
Anyway, thank you Daniel for your care :)



comment:5 Changed 15 years ago by dmorissette

Cc: dannys@… added
Milestone: FUTURE
Danny Stopel (Anywhere Ltd) wrote:
> 
> I succeeded to show Hebrew by reversing the characters of the string in 
> my database.
> 
> So, the problem of Hebrew is solved, at least for me.
> 

Setting target milestone to FUTURE... we may revisit this in a future version.

Note that we can't do anything without some good way to test. It would be great
if Danny (or anyone) could attach to this bug a sample mapfile + data + fonts
and some screenshots showing the expected result.


comment:6 Changed 14 years ago by eshabtai@…

Is there any updates on this bug?

It seems that Hebrew text is still displayed in the wrong direction.

comment:7 Changed 14 years ago by eshabtai@…

Cc: eshabtai@… added

comment:8 Changed 14 years ago by dmorissette

No developments yet, and can't really work on this until someone attaches sample
mapfile + data + fonts and some screenshots showing the expected result

Changed 14 years ago by eshabtai@…

Attachment: mapserver_bidi.patch added

Quick fix for bidi support

comment:9 Changed 14 years ago by eshabtai@…

I will upload some hello world example whith some Hebrew text so you can play
with (as soon as I can).

Changed 13 years ago by eshabtai@…

Attachment: hebrew.map added

A simple map file which displays Hebrew text

Changed 13 years ago by eshabtai@…

Attachment: DavidCLM-Bold.pfa added

An hebrew font from http://culmus.sourceforge.net/

Changed 13 years ago by eshabtai@…

Attachment: fontset.txt added

fontset file required for the map file

Changed 13 years ago by eshabtai@…

Attachment: correct.gif added

The expected output (correct output).

comment:10 Changed 13 years ago by eshabtai@…

I have uploaded all the required files to generate a simple Hebrew text.
correct.gif shows how the text should appear. Currently mapserver renders the
text in the wrong direction. When applying the patch I attached to mapserver it
will render the text correctly.

I'll be happy to wrap this one myself, but I need some review on the code I added.

comment:11 Changed 11 years ago by dmorissette

Description: modified (diff)
Milestone: FUTURE5.2 release
Owner: changed from mapserverbugs to dmorissette
Status: assignednew

Let's try to get this one done in 5.2

comment:12 Changed 11 years ago by dmorissette

Cc: aboudreault added

I had received the following email about the original patch. I will attach Endre's patch as well that fixes this issue:

Subject: MapServer Hebrew Characters Support
From: Piret Endre
Date: Mon, 12 Feb 2007 16:51:48 +0100
To:dmorissette@...

Daniel,

I had to deal with some Hebrew maps in MapServer. I found your
correspondence with Ehud Shabtai and his patch, it helped me a lot.
But the patch makes impossible to display labels with encodings not
supported by FriBidi, like Windows-1250 and so on. I have to diplay
both Hebrew and European texts, so I changed the patch a bit. It
requires the encoding to be specified in the mapfile, so one needs to
add an ENCODING "UTF-8" line to Ehud's mapfile. After this addition it
gives the same result as with the original patch, while my European
labels are still working.

Best Regards
Endre

Changed 11 years ago by dmorissette

Attachment: mapserver-bidi.patch added

Endre's patch for the unsupported encoding problem (e.g. Windows-1250)

comment:13 Changed 11 years ago by dmorissette

Cc: woodbri added
Status: newassigned

AlanB has ported and tested the patches with MapServer 5.1 (trunk) and made the required configure and Makefile.in changes. I have reviewed and committed them in r7457.

Notes/Limitations?:

  • Not tested on Windows yet (at a minimum makefile.vc needs an update)
  • Character "joining" not supported yet, this will be done in ticket #2549

comment:14 Changed 11 years ago by dmorissette

Owner: changed from dmorissette to assefa
Status: assignednew

Assefa, could you please test this on Windows and update the Makefile.vc, and then mark the ticket as closed? Thanks.

comment:15 Changed 11 years ago by assefa

Status: newassigned

which version of the fribidi library is used?

I have downloaded the fribidi-0.10.9.tar.gz and the CVS version but when compiling, I get errors in mapstring.c such as 'fribidi_log2vis' : too few arguments for call. The first argument seems to be missing from the call comapred to the header of this function that I have RIBIDI_API fribidi_boolean fribidi_log2vis (FriBidiEnv? *fribidienv,

/* input */ const FriBidiChar? *str, FriBidiStrIndex? len, FriBidiCharType? *pbase_dirs, /* output */ FriBidiChar? *visual_str, FriBidiStrIndex? *position_L_to_V_list, FriBidiStrIndex? *position_V_to_L_list, FriBidiLevel? *embedding_level_list);

Note that by setting the first argument to NULL, I was able to build.

BTW I was not able to have the test sample working (I mean to see the hebrew characters), is it possible to provide a zip containing the full set of map/font for testing (maybe I have screwed up the sample data when I downloaded it from the attachments and I would like to give it another shot)

I will wait until the header issue is resolved and commit the windows makefiles.

comment:16 Changed 11 years ago by woodbri

FYI:

From http://fribidi.freedesktop.org/wiki/

There is a release of the fribidi2 modules available now, which is http://fribidi.org/download/fribidi-0.19.1.tar.gz (md5sum, signed by Behdad Esfahbod) from January 15, 2008. This release is based on Unicode 5.0.0 character database too and is supposed to be fully API/ABI compatible with the 0.10 series and may need some testing before fully replacing the 0.10 series.

Given that this is supposed to be API/ABI compatible with the 0.10 series AND it is supposed to have support for character joining, it might make sense to consider just upgrading to this version of the library and dropping the patches. This is covered under ticket #2549.

comment:17 Changed 11 years ago by aboudreault

Assefa,

The version that is supported in the MapServer patch is 0.10.9. I tryed the version 0.19.1 but no success. Both version are supposed to be compatible but i think we will need to modify the MapServer patch in mapstring.c. Is your fribidi compile correctly? If not, you can checkthe fribidi package that came with Abiword source. It contain a "win" folder that is for building fribidi under MSVC. http://www.abisource.com/downloads/abiword/2.4.6/source/abiword-2.4.6.tar.gz

btw, i didnt see anywhere the argument of log2vis function "FriBidiEnv?? *fribidienv," in the version 0.10.9. Your probably using the CVS version for have this header? For your information, the only warning that i get during compilation with gcc is: mapstring.c: In function ‘msConvertWideStringToUTF8’: mapstring.c:1270: warning: dereferencing type-punned pointer will break strict-aliasing rules but it's not related to the log2vis function call.

Alan

comment:18 Changed 11 years ago by assefa

Alan,

I was using cvs, you are right the header of 0.10.9 builds properly with Mapserver.

I am building fribidi locally with a local makefile since I was not able to fully build the library using the MSVC project files.

If possible can you send me directly a zip containing your test map/font. For some reason I am not able to view the Hebrew characters properly.

Committed windows build files r7477

comment:19 Changed 11 years ago by dmorissette

FYI we had to add ENCODING "UTF-8" to the LABEL object in hebrew.map to make it work.

comment:20 Changed 11 years ago by assefa

Resolution: fixed
Status: assignedclosed

This was tested on windows with version 0.19.1 and makefiles have been updated. Closing.

Note: See TracTickets for help on using tickets.