memory overwrite in mapxbase.c

[woodbri@…]

I have been able to track down a problem I was having working with Navtech data.
It seems that they generate shapefiles for some layers that appear to have valid
dbf files that do not have any attribute fields defined in them.

I looks like when these are accessed via a tileindex there is some kind of
memory overwrite happening. I had a streets layer that was defined after this
offending layer that was acting weird like all that attributes were off by one
or were being pulled from the wrong column. When I commented out this layer
everything when back to normal.

OK, reading some code looks like mapxbase.c is not checking for the case where
nFields = 0 and is doing a bunch of malloc on zero bytes and then stuffing
values into those zero byte locations.

I'll write a bug on this also and while you might argue that I should not have a
dbf without attributes, I don't see any reason I shouldn't but this is a nasty
memory overwrite condidtion that I have been fighting with off and on for 3
weeks and needs to be fixed one way or another.

[dmorissette]

Steve, do you have a small sample dataset and mapfile that you could attach to
reproduce this?

[woodbri@…]

Daniel, I has stripped the problem down to a few layers and two DCAs, but upon 
some additional investigation the layers did have attributes (the dbfdump I was 
using was not from shapelib, so passing it *.shp was bad), so I don't think 
that is the problem. There is definitely some problem if adding or not a 
polygon layer before a strets layer screws up the streets layer.

It would be easiest to give you access to the system if you want to chase this 
down. 

I plan to just dump the polygon layer for the time being. It is possible that 
the polygon layer from Navtech is somehow damaged. I have brought it into 
ArcExplorer with no problems.

[dmorissette]

Duh... I had written a reply to your email this morning describing what I
thought the problem was but decided to not send it since you seemed convinced
that it was due to the 0-fields dbfs.

So here it is again: you should check that all the dbfs in your tile index share
the exact same number of attributes and that they are all in the same order. 
MapServer only reads the header of the first DBF file in the tile index to
figure the indexes of each of the fields that it needs to read.  If any of the
DBF in the tiled layer has a different set of attributes then you get exactly
the symptoms that you described.

[woodbri@…]

I actually thought of that one also and did check that and everything looks the 
same. But now that I think about it I only check the layer that I removed and 
fixed my problem, it is possible that some other layer is bad. Remove that 
layer may have shifted the problem to another place. I will check them also.

Hmmm, I thinking that a utility that validates a tileindex, but checking all 
this stuff would be a good tool to have around.

[dmorissette]

Steve (Lime): what would you think of adding some tests in the tiled layer code
when layer->debug==TRUE to validate the DBF attributes when opening a new tile
and report errors if any inconsistencies are found?

[sdlime]

I was thinking the same thing, especially since I've been hanging around in 
that code for the last few days. I suspect all I need is a simple function to 
compare item indexes extracted from the template to the new tile when opened. 
I'll take care of it as part of the mods I'm doing to tiling to allow filtering.

Steve

[woodbri@…]

Daniel, this can be closed. You were right, Navtech changed the field 
definitions on DCA1. 

I emailed you a modified version of tile4ms.c that checks the DBF structure of 
all the shapefiles being included into a tileindex to verify that they all have 
the same structure. This adds a little bit of extra processing, but it should 
save a LOT of headaches for when you do run into this situation. Please look 
over code changes I made against 4.0.1 and check it into cvs for me.

[dmorissette]

I have committed Steve's tile4ms.c changes in both the 4.1 and 4.0.x branches in
CVS.

Before we mark this fixed, I think a clarification should be added to the
TILEINDEX parameter docs in the mapfile reference to explain that all DBFs in a
tile index must share the exact same set of attributes and in the same order.

I'll reassign the bug to JF Doyon to add this to the mapfile reference.

[dmorissette]

JF: any chance you could add this clarification to TILEINDEX in the mapfile
reference before the 4.2 release? (see comment #8) If not then please reassign
to Jeff.

[dmorissette]

Jeff, please see comments #8 and #9

[jmckenna@…]

i will add a note on this in the mapfile-reference doc.

[jmckenna@…]

added note in mapfile-reference doc in the main CVS for docs.  Can this bug be
closed?

[woodbri@…]

I'm good with it. thanks for everyones effort.

[dmorissette]

If the docs are updated then we can close. Marking fixed. Thanks.