ST_Contour bug over the limits of tiles

[latot]

Hi all, ST_Contour works right when we have one tile, or one big raster and we execute it over GDAL, but in postgis is different, we have super big rasters!, and we handle it splitting them in small cute ones.

Well, the ST_Contour algorithm does not like it, actually the output of one big raster is pretty different from the same in small pieces, the difference is that the contour lines will be spplited/broken… D:

The distance between the lines can depends on the raster it self, is not an easy fix to get the right contour from the data, if there is contour lines that are more closer between them than the separation caused by this causes that we can't recognize the contour lines… D:

I'll upload 3 images, one from the contour generated by postgis, the other two, one with one raster, and the other one with the other, obvs the contour should follow the lines and don't cause what is in the images.

[latot]

Full Contour

[latot]

Raster 1 and contour

[latot]

images

[latot]

Hi, sorry, I uploaded two time the same image, I wanted to remove them but I didn't get how to do it u.u

[pramsey]

I would think in general any attempt to generate contours on independent raster and then match them up will be bound to fail. Most handling of things that pass over boundaries in tile problems involves divying up the raw inputs so they have generous overlap between each tile, then apply the algorithm, then clip the outputs to the actual tile bounds. That would at least MINIMIZE the errors one might expect from the algorithm, though I bet there would still be some smaller inconsistencies, maybe as a result of the algorithm differing based on where it chooses to seed the process. This is not something that is/will be changed/fixed in PostGIS.

[latot]

mm, personally is weird the bug it self….

When we store a raster in a db, we expect to be able to access to any pixel in a similar way, there should be only one raster from the "out" perspective, even if is decided to save the raster as several tiles, all the raster funcs should run over all the data, so when we run a function that works for example, in the nearest pixels, will run fine in all of them, including in the limit of the tiles (there maybe must be some constraints).

I don't think the idea should be "trick" postgis in order to get algorithms to work, we use postgis because is able to handle big amounts of spatial data and we have good access and functions.

GDAL devs tell me a little about this, is in fact similar, in order to get contours or apply any other function we should use vrt format, vrt is not a "raster" format like tiff, it just tell to GDAL where the tiff files are, and insure access too all pixels like one big raster, this is usually what we can expect from postgis, very similar. I tested it and works great. The only thing I didn't tested is in case two rasters overlap.

I'm just a little worried about: This is not something that is/will be changed/fixed in PostGIS.

If, for the contour function, the access to the pixels can't be insured in a uniform way for the function, maybe for other functions can happens too. Maybe you know, when and how this can happens, but for the ones who didn't read and know that much detail that is insternal to postgis, I think is a fair thought.

The actual contour function, without this fix is mainly broken. Think in the usual 100x100 tiles, so you will probably have more than 1 tile, we load a super raster, run contour, and ppl will think will works fine while is not the case… D:

Just as note, and avoid misunderstandings, the bug here is the lines are not connected betwen the tiles, so the contours are broken.

Sorry if I was too brute in my explanation, if I misunderstood something sorry too!

[pramsey]

The premise of the raster module is small pieces of raster, and users build them up as necessary. That's the premise of the whole thing. There's no "make it work easier" magic to wave at it. A different raster module built on a completely different premise might be more like you expect, but that's not what we have.

Should we get rid of what we have? Maybe! It certainly fills people with false hope.

But "it's in the database so it should be fast" is not actually logical reasoning. The database has to handle the same limitations as any other software, in terms of access to storage and memory and moving data between those things. With the added overhead, in the case of PgSQL of also maintaining full transactional integrity over multiple simultaneous readers and writers. It's going to end up trading off specific efficiency for general purpose utility.

Maybe explore other ways of processing your rasters, like an imperative scripting tool like python rasterio.

[robe]

Lets not forget the most limiting factor of postgis raster the 1GB memory allocation limit that PostgreSQL thrusts upon us. A limitation you don't have using GDAL direct.