Error with Japanese translation (PDF)

[strk]

The problem seems to be in release_notes.xml which taken out makes the build of pdf succeed. It can be used as the only input to the dblatex command and would result in:

postgis-out.toc:4: You can't use `macro parameter character #' in horizontal mode.
postgis-out.toc:4: leading text: ...&##x5909;\&##x66f4;}{2}{subsection.a.A.1.2}

This is a snip-off of #5438

[strk]

Hard to believe but I found out that keeping only the first sect1 of release_notes.xml and changing ulink tags attribute url to an empty string in the "Bug Fixes" section fixes the error. I've no idea what that means

[strk]

I've realized that changing the url tag (or anything else) in the source XML just effectively avoids to bring in the Japanese text, as the "msgid" changes.

Grayshade tested some more things and was able to obtain a PDF. It was a combination of passing -P latex.encoding=utf8 and installing texlive-ucs and removing translations from programlistings (-- AsText output was translated)

[strk]

The other flag used by Grayshade was -P latex.unicode.use=1

[strk]

For context &##x5909; may refer to unicode character ​変 and &##x66f4; may refer to unicode character ​更

[Sandro Santilli <strk@…>]

In 452b3a1/git:

Use lineannotation for comments inside screen tags

References #5439

[Sandro Santilli <strk@…>]

In a8b3dd9/git:

More lineannotation usage

References #5439

[strk]

While the commit using lineannotation tag temporarely fixed the japanese PDF generation, it's not a definitive solution, becuase the .po file will include that tag and still fail if any unicode is inserted in there

[strk]

To recap: building the PDF fails IF japanese unicode characters are entered in tags that are rendered as literal strings in tex. This seem to be things like <programlisting> and <screen>. The use of <lineannotation> was in hope that'd be different but is not, as long as that tag is INSIDE '<programlisting> or <screen>`. My commit resulted in a success because the japanese translators did not update the translation after introduction of the new tag so ther's no Japanese character in the translated XML.

The bug in handling utf8 in so called "listing environment" seems to be already reported upstream: ​https://sourceforge.net/p/dblatex/bugs/130/ and I saw workaround posted suggesting to add mapping for each character to be used in there, which is not very practical.

What we could do as a workaround, maybe, is to pre-process our XML to make sure <programlisting> and <screen> tags are never sent to the translation process.

[strk]

Adding \usepackage{xeCJK} to the doc/texstyle.sty makes the japanese fonts work for me, mostly: ​https://ctrl.vi/i/La0IrWNoh

[lnicola]

After some investigation, it seems that XeTeX is unmaintained these days (its previous maintainer has moved on to LuaTeX), and non-Latin support in pdfTeX is very painful, if not wholly unachievable.

I was able to get a somewhat usable Japanese PDF by dumping the TeX source (with -P latex.encoding=utf8 -P latex.unicode.use=1) and editing it to remove this:

\IfFileExists{ifxetex.sty}{%
    \usepackage{ifxetex}%
  }{%
    \newif\ifxetex
    \xetexfalse
  }
  \ifxetex
\usepackage{fontspec}
\usepackage{xltxtra}
\defaultfontfeatures{Mapping=tex-text}
\setmainfont{DejaVu Serif}
\setsansfont{DejaVu Sans}
\setmonofont{DejaVu Sans Mono}
\else
\usepackage[T2A,T2D,T1]{fontenc}
\usepackage{ucs}
\usepackage[utf8]{inputenc}
\def\hyperparamadd{unicode=true}
\fi

to replace it with this:

\usepackage{luatexja}
\setlength{\headheight}{12.06757pt}

Increasing the headheight isn't strictly necessary, but I was getting thousands of warnings about it.

luatexja supports Noto Sans CJK and related fonts, but the output looked rather bad, especially the code listings. I ended up using the default font, available as texlive-haranoaji on my distro.

Note that the dblatex style tries to load some fonts of its own when not using XeTeX. This may or may not cause issues. I originally commented them out in /usr/share/texlive/texmf-dist/tex/latex/dblatex/dbk_fonts.sty, but it might work with luatexja.

Replacing utf8x with utf8 in \lstset{inputencoding=utf8x, extendedchars=true} might also be a good idea, but I didn't test it. I did notice that the comments in code blocks are bogus. EDIT: no, utf8 doesn't help.

Also note that building the PDF is about 5x slower or so, probably because the layout calls into Lua code.

I'm attaching a couple of versions, for anyone who's curious.

EDIT: with that 5.9 MB limit, I'm not attaching anything.

[robe]

When I did an

apt search haranoaji

It returned texlive-lang-japanese which is already installed on debbie. So I assume we should be set to use that.

[lnicola]

Here's a working version with XeTeX:

% \setmainfont{DejaVu Serif}
% \setsansfont{DejaVu Sans}
% \setmonofont{DejaVu Sans Mono}
\usepackage{xeCJK}
\setCJKmainfont[BoldFont=HaranoAjiMincho-Bold]{HaranoAjiMincho}
\setCJKsansfont[BoldFont=HaranoAjiGothic-Bold]{HaranoAjiGothic}
\setCJKmonofont[BoldFont=HaranoAjiGothic-Bold]{HaranoAjiGothic}

(commenting out the DejaVu fonts might not be necessary, but I don't have them, and Noto doesn't work)

[Sandro Santilli <strk@…>]

In e00f658/git:

Use custom tex style for japanese

References #5439

[strk]

I've pushed support for per-language tex style and japanese override with those fonts, please report if it works for you guys, and we'll see debbie

[strk]

Debbie outputs a nice looking japanese pdf now: ​http://postgis.net/stuff/postgis-3.4.0dev-ja.pdf

Closing as completed