Opened 10 years ago
Closed 10 years ago
Last modified 10 years ago
#2209 closed defect (fixed)
[raster]: String of 388800145 bytes is too long for encoding conversion
|Reported by:||robe||Owned by:||robe|
This one is an odd one and may be an issue with PostgreSQL and not raster. Since I'm coming across it when loading raster I'll put it here.
Okay I managed to load all the raster pictures I have in except for the biggest one and it gives this error.
NOTICE: CREATE TABLE will create implicit sequence "armory3_local_rid_seq" for serial column "armory3_local.rid" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "armory3_local_pk ey" for table "armory3_local" CREATE TABLE ERROR: out of memory DETAIL: String of 388800145 bytes is too long for encoding conversion. CONTEXT: COPY armory3_local, line 1 CREATE INDEX ANALYZE
Now this may be a windows only issue. You see I have this database set with UTF-8 encoding, but to deal with legacy apps, I have the client encoding set to :
ALTER DATABASE mydb SET client_encoding = 'WIN1252';
So i presume raster2pgsql is encoding as win1252, though for some reason I always thought bytea data would be untouched.
The error only happens with the biggest image I have in this set which is ~29 MB. I'll send you link to image separately.
Change History (9)
comment:1 by , 10 years ago
comment:2 by , 10 years ago
For a split second I thought this was a memory issue but after seeing the word "encoding" and the size of the raster file, it definitely has something to do with PostgreSQL. raster2pgsql doesn't know anything about the database's settings for client encoding as it never (ever!) touches a database.
I found the place in PostgreSQL's source that emits that error message and the max size permitted for an encoding conversion is ~250 MB.
Having the client encoding be the same as the server encoding eliminates the conversion process.
I don't know what we can do about this though beyond leaving a note in the docs…
comment:3 by , 10 years ago
Oh I see what you are saying I was assuming we could just say we are outputting UTF8 as part of the generated raster2pgsql. Except we have no place to denote encoding in raster2pgsql anyway and not sure what that would mean for binary data why it even needs to pass thru an encoder.
I suppose people might have filenames that require encoding care or if we start reading meta data from fiels we could encounter weird characters taht require encoding care. Huh I don't even want to think about this.
I'll just document in the FAQ. In theory you don't have to do anything until you get the error.
comment:4 by , 10 years ago
|Component:||raster → documentation|
comment:5 by , 10 years ago
|Status:||new → closed|
I put in raster FAQ at r11105 (put pretty anchor at r11106)
should show up soon.
comment:6 by , 10 years ago
I made some typos on my tags, but finally got it right — so you can see if in
If looks okay to you I can make the same change to 2.0. I'll assume you are okay with it unless I hear otherwise.
comment:7 by , 10 years ago
The only suggestion I have is that instead of "String of 388800145 bytes", you should just substitute the number with N… "String of N bytes". This is because the numerical value is dependent on the length of the string to be converted.
comment:8 by , 10 years ago
comment:9 by , 10 years ago
|Milestone:||PostGIS 2.1.0 → PostGIS 2.0.3|
BTW - works fine if I add to the top of my batch script:
So not sure if there is much we can do here aside from maybe explicitly setting the client encoding in raster2pgsql. I still don't know why it should matter for bytea or its a bug in postgreSQL why it even tries.