[raster]: String of 388800145 bytes is too long for encoding conversion

[robe]

This one is an odd one and may be an issue with PostgreSQL and not raster. Since I'm coming across it when loading raster I'll put it here.

Okay I managed to load all the raster pictures I have in except for the biggest one and it gives this error.

NOTICE:  CREATE TABLE will create implicit sequence "armory3_local_rid_seq" for
serial column "armory3_local.rid"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "armory3_local_pk
ey" for table "armory3_local"
CREATE TABLE
ERROR:  out of memory
DETAIL:  String of 388800145 bytes is too long for encoding conversion.
CONTEXT:  COPY armory3_local, line 1
CREATE INDEX
ANALYZE

Now this may be a windows only issue. You see I have this database set with UTF-8 encoding, but to deal with legacy apps, I have the client encoding set to :

ALTER DATABASE mydb
  SET client_encoding = 'WIN1252';

So i presume raster2pgsql is encoding as win1252, though for some reason I always thought bytea data would be untouched.

The error only happens with the biggest image I have in this set which is ~29 MB. I'll send you link to image separately.

[robe]

BTW - works fine if I add to the top of my batch script:

set PGCLIENTENCODING=UTF8

So not sure if there is much we can do here aside from maybe explicitly setting the client encoding in raster2pgsql. I still don't know why it should matter for bytea or its a bug in postgreSQL why it even tries.

[dustymugs]

For a split second I thought this was a memory issue but after seeing the word "encoding" and the size of the raster file, it definitely has something to do with PostgreSQL. raster2pgsql doesn't know anything about the database's settings for client encoding as it never (ever!) touches a database.

I found the place in PostgreSQL's source that emits that error message and the max size permitted for an encoding conversion is ~250 MB.

​http://doxygen.postgresql.org/memutils_8h_source.html

Having the client encoding be the same as the server encoding eliminates the conversion process.

I don't know what we can do about this though beyond leaving a note in the docs…

[robe]

Oh I see what you are saying I was assuming we could just say we are outputting UTF8 as part of the generated raster2pgsql. Except we have no place to denote encoding in raster2pgsql anyway and not sure what that would mean for binary data why it even needs to pass thru an encoder.

I suppose people might have filenames that require encoding care or if we start reading meta data from fiels we could encounter weird characters taht require encoding care. Huh I don't even want to think about this.

I'll just document in the FAQ. In theory you don't have to do anything until you get the error.

[robe]

I put in raster FAQ at r11105 (put pretty anchor at r11106)

​http://postgis.net/docs/manual-dev/RT_FAQ.html

should show up soon.

[robe]

I made some typos on my tags, but finally got it right — so you can see if in

​http://postgis.net/docs/manual-dev/RT_FAQ.html#qa_raster_fails_encoding_conversion

If looks okay to you I can make the same change to 2.0. I'll assume you are okay with it unless I hear otherwise.

[dustymugs]

The only suggestion I have is that instead of "String of 388800145 bytes", you should just substitute the number with N… "String of N bytes". This is because the numerical value is dependent on the length of the string to be converted.

[robe]

Done at r11110 and back ported at r11111 yeh we have more 1s than I can count now.