Opened 6 years ago

Last modified 7 weeks ago

#1764 new defect

[raster] overviews not marked as same alignment though main table is

Reported by: robe Owned by: dustymugs
Priority: medium Milestone: PostGIS Fund Me
Component: raster Version: trunk
Keywords: Cc: mloskot

Description

I'm guessing this is a rounding issue. I loaded up some 1-meter DEMS and chunked them to 128x128 and also created over views. So my main table has about 100,000 records.

The main table got registered as same alignment = true, the level 2 got registered also as same alignment = true, but the level 3 dems came in as same alignment = false.

I would assume if the core table was same alignment, then all the overviews should be as well -- no?

This is testing on PostGIS 2.0.0 64-bit on windows 2008 R2.

Change History (23)

comment:1 Changed 6 years ago by dustymugs

Owner: changed from pracine to dustymugs
Status: newassigned

You're probably right about this being a rounding issue. We need to add a tolerance parameter for the same alignment check. I'll see if I can reproduce this using some of my rasters. Might be a good time to play with DEM datasets.

comment:2 Changed 6 years ago by robe

This is interesting. I loaded a smaller DEM that is 1000x1000. I didn't chunk it.

With -l 2,3,4,5 (levels 2,4,5) came in with same alignment = true

level 3 came in with same_alignment = false

with 333x333 block size. I guess because 1/3 goes on forever when converted to decimal. Not sure if that helps any.

comment:3 Changed 6 years ago by dustymugs

I have the same suspicion about the factor = 3 causing issues. Is this happening for a single DEM raster file or across a set?

comment:4 Changed 6 years ago by dustymugs

Curses. I'm unable to get any of my rasters to have the same alignment issue. Is it possible to share that smaller DEM file?

comment:5 Changed 6 years ago by dustymugs

I've tested a subset of my rasters in Linux 32-bit, Linux 64-bit and Windows 7 64-bit and am unable to recreate this issue.

comment:6 Changed 6 years ago by robe

I'll send in a bit. I doesn't seem to happen in the same file. So you have to test at least two files I think.

I had to run this query to find a candidate:

SELECT v1.filename As v1_filename, o1.rid As o1_rid,  v2.filename As v2_filename, o2.rid as o2_rid
FROM o_3_vhms As o1 INNER JOIN o_3_vhms As o2 ON ST_SameAlignment(o1.rast,o2.rast) = false
  INNER JOIN vhms As v1 ON ST_Within(v1.rast::geometry, o1.rast::geometry) 
   INNER JOIN vhms As v2 ON ST_Within(v2.rast::geometry, o2.rast::geometry) 
limit 1;

comment:7 Changed 6 years ago by robe

For the non-chunked version I had to change ST_Within to ST_Intersects. I assume that is because the bounding box is a bit fatter so they overlap. anyrate same query using intersects and within returns the same files at issue. Just getting permssion before I can send the files.

comment:8 Changed 6 years ago by dustymugs

As the rasters which uncovered the problem described in this ticket, the following describes the attributes of the rasters in a generic manner.

Raster 1

width: 100
height: 100
scale-x: 1
scale-y: -1
skew-x: 0
skew-y: 0
upperleft-x: 0
upperleft-y: 0

Raster 2

width: 100
height: 100
scale-x: 1
scale-y: -1
skew-x: 0
skew-y: 0
upperleft-x: 100
upperleft-y: 0

If the above two rasters are loaded in with raster2pgsql, the tiles are aligned. But for an overview factor of 3 (and probably anything else where the division of the width/height by the overview factor results in a repeating decimal), the alignment of the overview will not be aligned.

When creating overviews, raster2pgsql treats each input raster independently of all other input rasters. This does not work for situations like this. Instead, it may be better to get the overall extent for the set of input rasters and then process each individual input raster based upon the overall extent.

The one issue here is what to do if the input rasters are NOT of the same scale and skews? Use the "old" method (the current method) or fail?

comment:9 Changed 6 years ago by dustymugs

The first sentence in the prior comment should be...

As the rasters which uncovered the problem described in this ticket are publicly unavailable, the following describes the attributes of the rasters in a generic manner.

comment:10 Changed 6 years ago by dustymugs

Summary: overviews not marked as same alignment though main table is[raster] overviews not marked as same alignment though main table is

The possible solution also isn't possible in any of the input rasters have a different SRID...

comment:11 Changed 6 years ago by robe

Maybe the best thing to do is just make a note of it in the loader docs or something as expected behavior you should be cautious of. It's not really a big deal as long as you know about it because the pixel width / height chunk settings are rule of thumb anyway so the user can just ensure the dimensions they pick will be suitable dimensions for their overviews.

What we need -- which I think we kind of need anyway -- is a raster import calculator that asks questions like:

What do you want to do? and may suggest chunk size, and even overviews to pick

comment:12 Changed 6 years ago by dustymugs

Milestone: PostGIS 2.0.1PostGIS 2.1.0

I think you're right about just making a note of it in the loader docs indicating the expected behavior. I'm punting this ticket to 2.1 as it will require significant refactoring of raster2pgsql.

comment:13 Changed 6 years ago by dustymugs

Added note in using_raster_dataman.xml regarding this ticket in r9653.

comment:14 in reply to:  12 Changed 6 years ago by pracine

Replying to dustymugs:

I think you're right about just making a note of it in the loader docs indicating the expected behavior. I'm punting this ticket to 2.1 as it will require significant refactoring of raster2pgsql.

It is not clear to me what refactoring is necessary in 2.1. Could you summarize it in a couple of sentence?

comment:15 Changed 6 years ago by dustymugs

Currently, overviews are generated on a per raster basis. This works great because each raster is treated independently.

But the limitation of this method is that though the rasters may be aligned, the overviews for a particular factor may not be aligned. A possible solution is for rasters to be processed first and create overviews afterwards. This would allow the loader to check that the rasters are aligned and then determine a new alignment grid for the overview at a specific factor. Only by doing this can we guarantee that the overview at a specific factor is aligned.

I still don't know if this is worth doing though as this can ONLY be run if all source rasters are aligned.

comment:16 Changed 5 years ago by dustymugs

Status: assignednew
Version: 2.0.xtrunk

I was thinking of a possible solution to this issue...

  1. before doing anything to the set of import rasters, see if the set are aligned
  1. if set of input rasters aligned, overviews are guaranteed to be aligned

As it looks like raster2pgsql may need to move away from using VRTs for overviews, this would be an appropriate change to make at the same time.

comment:17 Changed 4 years ago by dustymugs

Milestone: PostGIS 2.1.0PostGIS Future

I plan on rewriting raster2pgsql for 2.2 due to the number of enhancement tickets...

comment:18 Changed 3 years ago by strk

As 2.2.0 has a milestone now, should this be retargetted from "PostGIS Future" to 2.2.0 ?

I stumbled on this ticket as I'm also having troubles with alignment, particularly aligned tiles that becomes unaligned after a call to ST_Resize() with the _same_ parameters (puzzling).

comment:19 Changed 3 years ago by strk

Milestone: PostGIS FuturePostGIS 2.2.0

I'm getting the same problem with "round" scale factors of 2,4,8,16. This is again with scales 1,-1 and offsets 0,0 -- that's the identity matrix, which could very well be the (or one) cause of it. See also #2914

comment:20 Changed 3 years ago by strk

Sorry, I'm taking it back, for 2,4,8,16 it's not failing even with identity matrix. Not when using my new ST_CreateOverview and ST_Resize for resizing.

comment:21 Changed 2 years ago by dustymugs

Milestone: PostGIS 2.2.0PostGIS Future

comment:22 Changed 2 months ago by robe

Milestone: PostGIS FuturePostGIS Fund Me

Milestone renamed

comment:23 Changed 7 weeks ago by mloskot

Cc: mloskot added
Note: See TracTickets for help on using tickets.