Cracking problem with raster data.

[omry@…]

The problem is that for cetrain map requests, I can see a white gap between two
adjacent tiles.
I have reproduced the problem with two tiles, both in ECW format and in GeoTiff
format.

GDALINFO for ECW tile 1:
[c:\projects\nat_ecw]gdalinfo ntn_b.ecw
Driver: ECW/ERMapper Compressed Wavelets
Size is 15804, 11058
Origin = (195706.200000,700001.600000)
Pixel Size = (0.904480,-0.904467)
Corner Coordinates:
Upper Left  (  195706.200,  700001.600)
Lower Left  (  195706.200,  690000.000)
Upper Right (  210000.600,  700001.600)
Lower Right (  210000.600,  690000.000)
Center      (  202853.400,  695000.800)
Band 1 Block=15804x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary
Band 2 Block=15804x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary
Band 3 Block=15804x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary

GDALINFO for ECW tile 2:
[c:\projects\nat_ecw]gdalinfo ntn_d.ecw
Driver: ECW/ERMapper Compressed Wavelets
Size is 15804, 11058
Origin = (195706.200000,690000.000000)
Pixel Size = (0.904480,-0.904467)
Corner Coordinates:
Upper Left  (  195706.200,  690000.000)
Lower Left  (  195706.200,  679998.400)
Upper Right (  210000.600,  690000.000)
Lower Right (  210000.600,  679998.400)
Center      (  202853.400,  684999.200)
Band 1 Block=15804x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary
Band 2 Block=15804x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary
Band 3 Block=15804x1 Type=Byte, ColorInterp=Undefined
  Overviews: arbitrary

its easy to see that the lower left of the first tile and the upper left of the
second are the same point, so the tiles are adjacent.


The map file I use is 
MAP
	NAME "Israel orthophoto 2000"
	IMAGETYPE JPEG
	WEB
		METADATA
			"wms_title"                  "WMS Demo Server"
		END
	END

	PROJECTION
		"init=epsg:2039"
	END
	
	LAYER
		PROJECTION
			"init=epsg:2039"
		END	
	
		NAME "Israel orthophoto 2000"
		STATUS DEFAULT
		TILEINDEX "tindex.shp"
		TILEITEM "location"
		TYPE RASTER
	END

END # Map File

The request that generated a tile with the 'crack' is
http://omry/cgi-bin/mapserv.exe?map=/projects/nat_ecw/israel-itm.map&REQUEST=GetMap&VERSION=1&bbox=193943.77,702052.0,211426.34,677569.06&width=477&height=668


I have converted the ecw data to GeoTIFF, and using a different map file, with
the same request parameters, I got the exact same crack using the GeoTIFF data.
so this problem appears to be format independent.

I would be happy to provide any required additional info about the problem.

[fwarmerdam]

Omry, 

I have tried setting up something essentially the same as what you had
but I do *not* get any cracking using the current MapServer code base from
CVS.  I would suggest:

 o Try reproducing the problem with MapServer from CVS. 
 o Try reproducing the problem with a complete configuration you can provide
   to me. For instance, using much smaller files.  

You might also want to attach the image you got at the url you mention so
I can see the problem visually.  

[omry@…]

I am not a win32 programmer, and it will take very much effort to compile an
appropriate map server from CVS.
could you possibly provide me with a win32 binary of the lastest map server with
WMS and GDAL (ECW, specifically) so I can test it?

[fwarmerdam]

Omry, 

I have prepared a new release of my OpenEV_FW tools, and thrown in a current
MapServer build.  Download it from:

  ftp://ftp.remotesensing.org/gdal/openev/OpenEV_FW_163.zip 

Put openev_fw\bin in your path and you should have a new mapserv with GDAL,
ECW support, and WMS support.  

[omry@…]

I tried the server, and I think its compiled with strict error checking, which
prevented me from checkign the real problem:
The last problem I encountered was wms invalid layer or something alike.
the error was not informative enough for me to get over it.
could you possibly send me a map server which strict error checking off?
or help me figure out whats the problem with my map file:

MAP
	NAME "Israel orthophoto 2000"
	
	IMAGETYPE PNG24
	#EXTENT ? ? ? ?
	

	WEB
		METADATA
			"wms_title"                  "WMS Demo Server"
			"**wms_onlineresource"  "http://omry/cgi-bin/mapserv?map=nat.map&"
		END
	END

	PROJECTION
		"init=epsg:2039"
	END


	LAYER
		PROJECTION
			"init=epsg:2039"
		END	
	
		NAME "Israel orthophoto 2000"
		STATUS DEFAULT
		TILEINDEX "tindex.shp"
		TILEITEM "location"
		TYPE RASTER
	END

END # Map File

[fwarmerdam]

Omry,

Sorry for not getting back to you on this sooner.  I lost track of it in the
christmas rush. 

I am not aware of a method to turn off error checking.  As far as I know, most
WMS metadata is optional, and even so called "required" items usually just result
in loud comments being emitted in the capabilities about what is missing.

You mentioned in seperate email that the problem persists with ms401_gif_png.
But I am not really prepared to pursue the problem unless you can reproduce it
in the development version.  While things have moved on since I prepared the
development cut in December, I would still like you to try and reproduce the
problem with that.  

If the problem does remain in the development version then I would like to get
it fixed before the soon to come MapServer release.  

[omry@…]

Where can I obtain a development version?
I looked at http://mapserver.gis.umn.edu/win32nightlies.html but there is
nothing there.
the version you gave me three months ago is older that 4.01, isnt it? 
do you want me to try it again (OpenEV_FW_163.zip) ?

[fwarmerdam]

View from Omry demonstrating cracking effect

[fwarmerdam]

The version in OpenEV_FW 1.6.3 is off the development trunk, and so distinctly
different than 4.0.1 which is just a minor bug fixes release on the 4.0
branch. 

However, if you can get a nightly build of the development version to work
from then that would be even better.  I don't know where you would get a
development version from though. 

[omry@…]

I tried to compile the server from the nighly build.
had to get the libraries:
gd-1.8.4
gdal-1.2.0
proj-4.4.7
regex-0.12

and finally, I got stuck with this error:
I am building on Win2K, using visual C++ 6.0.
what do I do now?


[c:\development\cpp\rastermapserver\mapserver_dev]nmake /f makefile.vc

Microsoft (R) Program Maintenance Utility   Version 6.00.8168.0
Copyright (C) Microsoft Corp 1988-1998. All rights reserved.

        cl /nologo /Zi /W3 /DDEBUG -DWIN32 -D_WIN32 -IC:\development\cpp\RasterM
apServer\gd-1.8.4.tar.gz\gd-1.8.4\gd-1.8.4    -IC:\development\cpp\RasterMapServ
er\proj-4.4.7\proj-4.4.7/src -IC:\development\cpp\RasterMapServer\regex-0.12 -IC
:\development\cpp\RasterMapServer\gdal-1.2.0/gcore -IC:\development\cpp\RasterMa
pServer\gdal-1.2.0/alg   -Ic:/projects/curl-7.10.2/include   -Ic:/projects/libpq
/interfaces/libpq -Ic:/projects/libpq/include    -DHAVE_STRING_H -DREGEX_MALLOC
-DNEED_STRCASECMP -DNEED_STRNCASECMP -DUSE_POSTGIS   -DUSE_PROJ -DUSE_PROJ_API_H
   -DUSE_GD_PNG -DUSE_GD_JPEG -DUSE_GD_WBMP -DUSE_GD_GIF -DGD_HAS_GDIMAGEGIFPTR
-DUSE_GD_FT  -DUSE_WMS_SVR  -DUSE_WMS_LYR  -DIGNORE_MISSING_DATA -DUSE_GDAL  -DF
EATURE_INFO_HTML -DUSE_WFS_SVR -DUSE_WFS_LYR    -DUSE_GD_ANTIALIAS /c mapbits.c
/Fomapbits.obj
mapbits.c
mapows.h(52) : fatal error C1083: Cannot open include file: 'sys/time.h': No suc
h file or directory
NMAKE : fatal error U1077: 'cl' : return code '0x2'
Stop.

[omry@…]

Guys, I really want to help, but I need some assistence from you:
Either give me a win32 binary that has WMS and GDAL support, or help me compile one.

[dmorissette]

There's a win32 binary of 4.1 (dated April 8th) with a recent GDAL at 
http://maptools.org/php_mapscript/index.phtml?page=downloads.html

A new win32 binary of the 4.2 beta should be posted there as well in the next
few days.

[omry@…]

I tried the server there, but I get the following error:

msWMSLoadGetMapParams(): Invalid layer(s) given in LAYERS parameter

I use this URL :
http://omry/cgi-bin/mapserv.exe?map=/projects/israel/israel-itm.map&REQUEST=GetMap&VERSION=1&bbox=125327.375,646532.75,244672.62,723467.25&width=802&height=517
which works with my previous server.
maybe something is missing in the URL, which the old server didn't mind and the
new one does?
any suggestions?

[omry@…]

Okay, managed to get a map from the new server (Added LAYERS=MyLayerName to the URL)
the cracking problem exists in the new server as well.
I got the exact same map as with the previous server, the cracks even appears in
the same places exactly.

[omry@…]

NOTE : the previous post refers to MapServer ver 4.1

[fwarmerdam]

Omry,

I have reviewed what I did to try and reproduce your problem, and I don't
see what I can constructively change to get your results. 

First, are you *absolutely* sure that the white strip isn't in your actual
image files?  If not, the only way to proceed seems to be for you to make
your data available for download (ecw + index file).  I have been unable to 
reproduce your problem with simulated data.

[fwarmerdam]

Note to self: 

All work so far is in ~/mapserver/bug493 on gdal2200. 

[omry@…]

I am certain that the white strip is not in the data, it appears and disappears
when I move the map, and when I zoom onto its it disappears.

I have created a minimal data-set to reproduce the problem (only two tiles).
the data-set is available at:
http://polaris.telmap.com:8080/omry/nat_ecw.zip

This request which generates a map with a 'crack':
http://SERVER_HOST/cgi-bin/mapserv.exe?map=/projects/nat_ecw/israel-itm.map&REQUEST=GetMap&VERSION=1&bbox=125327.375,646532.75,244672.62,723467.25&width=802&height=517&LAYERS=Israel%20orthophoto%202000

Note: in this data set the problem is harder to reproduce.
when I use my real data it happens more often.
since the real data weight 4.5gb, I think we better focus on this sample.
the url I specified above reproduce the problem 100% of the time, on MapServer 4.1

[fwarmerdam]

Omry,

Your perseverence has won out against my sloth.

I downloaded your sample data that demonstrates the problem and was able to
reproduce the problem.  On digging in further it turns out the issue is a 
subtle matter of rounding when computing the "destination" raster window
in mapdrawgdal.c.  I have corrected it so that any map pixel whose center falls
within the available source raster will get set.  Before this rule was used
for determining the top left corner, but the width and height of the area to be
set was determined without regard to whether the top left corner rounding had
effectively moved the whole window. 

Well, anyways, the test render now works properly, and I think that proper
symmetric rules are now applied when rendering raster data (tiled or otherwise)
from GDAL supported files.  

The change in the trunk is revision 1.26.  I think it is important enough
to go into the 4.2 branch if that hasn't been released yet.  I'll raise the
issue with Daniel. 

[omry@…]

Thank you, Frank.
I would like to test the change against my actual data - as I said, the problem
appeared there more often than in the sample.
where can I get a compiled binary with the fix?

[fwarmerdam]

Folks,

I botched the last fix.  In some cases the "output window" was going one pixel
off the output raster and causing memory corruption.  I am committing a fix
for this in 4.2 and 4.3.  

Omry ... you will need to wait for a new build to test the fix.  Perhaps
an OpenEV_FW 1.7.5 shortly.

[omry@…]

Okay.
please let me know when its available...

[fwarmerdam]

Omry,

I hope you aren't still waiting for this!  I am closing this bug report
since the bug is fixed, but if you have had a chance to confirm it please
add a note here. 

Thanks,

[omry@…]

its fixed in version 4.2.2? (the current public version at
http://mapserver.gis.umn.edu/)

[fwarmerdam]

Omry, 

Yes, the problem has been corrected since 4.2.0 beta3, so the 4.2.2 release
should certainly include the fix.