Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#2209 closed defect (fixed)

[raster]: String of 388800145 bytes is too long for encoding conversion

Reported by: robe Owned by: robe
Priority: medium Milestone: PostGIS 2.0.3
Component: documentation Version: master
Keywords: Cc:


This one is an odd one and may be an issue with PostgreSQL and not raster. Since I'm coming across it when loading raster I'll put it here.

Okay I managed to load all the raster pictures I have in except for the biggest one and it gives this error.

NOTICE:  CREATE TABLE will create implicit sequence "armory3_local_rid_seq" for
serial column "armory3_local.rid"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "armory3_local_pk
ey" for table "armory3_local"
ERROR:  out of memory
DETAIL:  String of 388800145 bytes is too long for encoding conversion.
CONTEXT:  COPY armory3_local, line 1

Now this may be a windows only issue. You see I have this database set with UTF-8 encoding, but to deal with legacy apps, I have the client encoding set to :

  SET client_encoding = 'WIN1252';

So i presume raster2pgsql is encoding as win1252, though for some reason I always thought bytea data would be untouched.

The error only happens with the biggest image I have in this set which is ~29 MB. I'll send you link to image separately.

Change History (9)

comment:1 by robe, 11 years ago

BTW - works fine if I add to the top of my batch script:


So not sure if there is much we can do here aside from maybe explicitly setting the client encoding in raster2pgsql. I still don't know why it should matter for bytea or its a bug in postgreSQL why it even tries.

comment:2 by Bborie Park, 11 years ago

For a split second I thought this was a memory issue but after seeing the word "encoding" and the size of the raster file, it definitely has something to do with PostgreSQL. raster2pgsql doesn't know anything about the database's settings for client encoding as it never (ever!) touches a database.

I found the place in PostgreSQL's source that emits that error message and the max size permitted for an encoding conversion is ~250 MB.

Having the client encoding be the same as the server encoding eliminates the conversion process.

I don't know what we can do about this though beyond leaving a note in the docs…

comment:3 by robe, 11 years ago

Oh I see what you are saying I was assuming we could just say we are outputting UTF8 as part of the generated raster2pgsql. Except we have no place to denote encoding in raster2pgsql anyway and not sure what that would mean for binary data why it even needs to pass thru an encoder.

I suppose people might have filenames that require encoding care or if we start reading meta data from fiels we could encounter weird characters taht require encoding care. Huh I don't even want to think about this.

I'll just document in the FAQ. In theory you don't have to do anything until you get the error.

comment:4 by robe, 11 years ago

Component: rasterdocumentation
Owner: changed from Bborie Park to robe

comment:5 by robe, 11 years ago

Resolution: fixed
Status: newclosed

I put in raster FAQ at r11105 (put pretty anchor at r11106)

should show up soon.

comment:6 by robe, 11 years ago

I made some typos on my tags, but finally got it right — so you can see if in

If looks okay to you I can make the same change to 2.0. I'll assume you are okay with it unless I hear otherwise.

comment:7 by Bborie Park, 11 years ago

The only suggestion I have is that instead of "String of 388800145 bytes", you should just substitute the number with N… "String of N bytes". This is because the numerical value is dependent on the length of the string to be converted.

comment:8 by robe, 11 years ago

Done at r11110 and back ported at r11111 yeh we have more 1s than I can count now.

comment:9 by robe, 11 years ago

Milestone: PostGIS 2.1.0PostGIS 2.0.3
Note: See TracTickets for help on using tickets.