[patch] Geopublish doesnt properly send accentued chars in the title

[landry]

An issue i've been tracking down since some weeks, if my metadata title contains accentued chars they're not properly sent to GeoServer when publishing.

Espaces Naturels Sensibles - Département du Cantal - 2013

becomes this in GeoServer layer title

Espaces Naturels Sensibles - D?partement du Cantal - 2013

GeoServer itself properly supports POST/PUT'ing utf-8 encoded xml (ie curl -X -H"Content-type: text/xml" -d@/tmp/t.xml ​http://localhost:9080/wxs/rest/workspaces/public/datastores/CG15_2013_ENS/featuretypes/CG15_2013_ENS with /tmp/t.xml containing utf-8 chars sets a correctly accentued title/abstract).

So the issue is on Geonetwork side - afaict, when GET'ing geoserver.publisher (action=CREATE) upon publish, the chars are url-encoded :

GET /geocat/srv/fre/geoserver.publisher?_dc=1362049081712&metadataId=355&metadataUuid=a6a5a793-b9d1-4351-974a-fffd20644614&metadataTitle=Espaces%20Naturels%20Sensibles%20-%20%5CnD%C3%A9partement%20du%20Cantal%20-%202013

But when you're in the publisher backend in web/src/main/java/org/fao/geonet/services/publisher/Do.java Exec() the metadataTitle doesnt contain utf-8 chars anymore.

Log.debug(MODULE, "metadataTitle="+metadataTitle);

=>

DEBUG [geonetwork.GeoServerPublisher] - metadataTitle=Espaces Naturels Sensibles - D?partement du Cantal - 2013

So at some point the encoding is lost before PUT'ing/POST'ing xml to GeoServer.

[landry]

There is something definitely weird going on, since my tomcat has URIEncoding properly set to UTF-8 in server.xml, and using URLDecoder.decode on the query string returns ?? for accentued letters. A standalone example in a simple java file works.

import java.net.URLDecoder;
class t {
public static void main (String [] args) {
try {
    System.out.println( java.net.URLDecoder.decode(args[0],"UTF-8") );
} catch (Exception e) {}
}
}

java t foo=bar%C3%A9%C3%A0                                                          
foo=baréà

After adding some debug code to jeeves' ServiceRequestFactory.java:

Log.debug(Log.REQUEST,"Decoded queryString "+ URLDecoder.decode(req.getQueryString(), "UTF-8"));
Log.debug(Log.REQUEST, "Query string  : "+ req.getQueryString());

[jeeves.request] - Query string  : foo=bar%C3%A9%C3%A0
[jeeves.request] - Decoded queryString foo=bar??

[landry]

Finally nailed it, it was a tricky one - in fact after comparing the String lenght and converting it as a byte array, realized the utf chars were still properly in the string, not rendered in debug log. So it was down the call stack... and StringRequestEntity ctor was the culprit - apparently it defaults to encode as default HTTP content charset (ISO-8859-1).

So the following diff properly encodes all the xml sent by sendREST :

--- a/web/src/main/java/org/fao/geonet/services/publisher/GeoServerRest.java
+++ b/web/src/main/java/org/fao/geonet/services/publisher/GeoServerRest.java
@@ -661,7 +658,7 @@ public class GeoServerRest {
                        }
                        if (postData != null) {
                                ((PutMethod) m).setRequestEntity(new StringRequestEntity(
-                                               postData));
+                                               postData, contentType, "UTF-8"));
                        }
                } else if (method.equals(METHOD_DELETE)) {
                        m = new DeleteMethod(url);
@@ -669,7 +666,7 @@ public class GeoServerRest {
                        m = new PostMethod(url);
                        if (postData != null) {
                                ((PostMethod) m).setRequestEntity(new StringRequestEntity(
-                                               postData));
+                                               postData, contentType, "UTF-8"));
                        }
 
                } else {

And the layer definition in GeoServer correctly gets all the accentued chars - which are properly displayed in the WMS getCapability document.

[landry]

Create StringEntityRequest with the proper contentType and encoding

[ianwallen]

Commit in master ad83740d136a7f7423388b785489e3560a9b0db8