AVCbin bug on Linux Kernel 2.6

[Markus Neteler]

Frank,

the AVCbin driver in OGR seems to have troubles, especially under
Linux 2.6 Kernel (Mandrake 10):

[E00 file uploaded]

1. avcimport olson-carbon.e00 olson
2. export CPL_DEBUG=ON
3. ogrinfo -summary olson ARC
OGR: OGROpen(olson) failed.
OGR: OGROpen(olson) succeeded (0x804d408).
Had to open data source read-only.
INFO: Open of `olson'
using driver `AVCBin' successful.

Layer name: ARC
Geometry: Line String
Feature Count: 15750
Extent: (-170128861896635013341904755760727851008.000000,
-170140544564566716704068949377468596224.000000) - (0.000000, 0.000000)
Layer SRS WKT:
(unknown)
UserId: Integer (0.0)
FNODE#: Integer (0.0)
TNODE#: Integer (0.0)
LPOLY#: Integer (0.0)
RPOLY#: Integer (0.0)

uname -a
Linux dandre2.itc.it 2.6.3-13mdk #1 Tue May 18 18:49:17 EDT 2004 i686 unknown
unknown GNU/Linux

gcc -v
Reading specs from /usr/lib/gcc-lib/i586-mandrake-linux-gnu/3.3.2/specs
Configured with: ../configure --prefix=/usr --libdir=/usr/lib
--with-slibdir=/lib --mandir=/usr/share/man --infodir=/usr/share/info
--enable-shared --enable-threads=posix --disable-checking --enable-long-long
--enable-__cxa_atexit --enable-clocale=gnu
--enable-languages=c,c++,ada,f77,objc,java,pascal --host=i586-mandrake-linux-gnu
--with-system-zlib
Thread model: posix
gcc version 3.3.2 (Mandrake Linux 10.0 3.3.2-6mdk)

So GRASS 5.7 doesn't read any Arc Coverage.

---------------
On a Linux 2.4 Kernel (Redhat9) the extent is correct but 
the column lengths are not:

ogrinfo -summary olson ARC
Had to open data source read-only.
INFO: Open of `olson'
using driver `AVCBin' successful.

Layer name: ARC
Geometry: Line String
Feature Count: 15750
Extent: (-180.000000, -89.500000) - (179.999969, 90.499985)
Layer SRS WKT:
(unknown)
UserId: Integer (0.0)
FNODE#: Integer (0.0)
TNODE#: Integer (0.0)
LPOLY#: Integer (0.0)
RPOLY#: Integer (0.0)

As this was working earlier, might it be related to bugfix:
2004-02-28 07:39  warmerda
        * avc.h (1.3), avc_bin.c (1.6), avc_mbyte.c (1.3): updated
?

Best regards

 Markus Neteler

[neteler@…]

Olson carbon dataset

[neteler@…]

Frank,

I have already tried to track this bug down, but
I'm unable to insert correct debug output (CPL_DEBUG) into
the AVC driver - I don't understand where to add it
(which files? All tries failed so far).

Suggestions welcome, then I'll try again to identify
the problem.

Best regards

 Markus

[warmerdam]

Markus,

I have reproduced the "redhat" case on my redhat 8 system.  The column lengths
of zero just indicates that there is no apriori known column width and is not
an error.  As far as I know it was always like this for this driver. 

The extent on your 2.6 system is indeed an issue.  The 2.6 system is still
just a conventional 32bit intel system, right?  No 64 bit issues?  Not
linux on a big endian system? 

First, can you verify that the problem in ogrinfo as opposed to avcimport?  
That is, if you do the avcimport on 2.6 and then run ogrinfo against it on
2.4 do things work or not? 

I believe the extent is computed by scanning through the whole layer and
collecting the extent of the individual geometries.  Do the geometries look
ok if you do a full ogrinfo on the AVC layer (as opposed to just getting
the summary)?  If the problem is the geometries themselves then you will need
to track down through lots of stuff. 

The arce00 library arc geometries are converted into OGRGeometries circa
line 252 in ogravclayer.cpp.  At a lower level the ARC geometries are read
in the _AVCBinReadNextArc() function in avc_bin.c. 

Good luck!

[neteler@…]

Frank,

> The extent on your 2.6 system is indeed an issue.  The 2.6 system is still
> just a conventional 32bit intel system, right?  No 64 bit issues?  Not
> linux on a big endian system?

It's a plain laptop, 32bit. I just updated from Mandrake 9.1 to Mandrake 10
changing the Linux Kernel from 2.4 to 2.6.

> First, can you verify that the problem in ogrinfo as opposed to avcimport?
> That is, if you do the avcimport on 2.6 and then run ogrinfo against it on
> 2.4 do things work or not?

OK, I'll do that.

> I believe the extent is computed by scanning through the whole layer and
> collecting the extent of the individual geometries.  Do the geometries look
> ok if you do a full ogrinfo on the AVC layer (as opposed to just getting
> the summary)?  If the problem is the geometries themselves then you will need
> to track down through lots of stuff.

It's difficult because due to the huge extent the v.in.ogr segfaults.
So I cannot import it into GRASS. Also QGIS fails, stretching the map
abnormous.
If I recall correctly, also
E00 -> avcimport -> OGR -> SHAPE produced a SHAPE file with
huge extent so that v.in.ogr failed.

> The arce00 library arc geometries are converted into OGRGeometries circa
> line 252 in ogravclayer.cpp.  At a lower level the ARC geometries are read
> in the _AVCBinReadNextArc() function in avc_bin.c.
>
> Good luck!

Inserting fprintf(stderr,...) should work?
Or, maybe, you could add some CPL debug output where appropriate?

Best regards

 Markus

[neteler@…]

Frank,

I have recompiled AVCE00 on the Linx 2.6 kernel. Then used avcimport
and created the AVCbin cover. Then ogrinfo used there, huge extent.

Then I transferred the directories to the other machine with 
Linux 2.4 kernel. Finally used the OGR compiled there, and
this is the output:

ogrinfo -summary olson ARC
Had to open data source read-only.
INFO: Open of `olson'
using driver `AVCBin' successful.

Layer name: ARC
Geometry: Line String
Feature Count: 15750
Extent: (-180.000000, -89.500000) - (179.999969, 90.499985)
Layer SRS WKT:
(unknown)
UserId: Integer (0.0)
FNODE#: Integer (0.0)
TNODE#: Integer (0.0)
LPOLY#: Integer (0.0)
RPOLY#: Integer (0.0)

This may suggest that 'avcimport' works fine, but OGR somewhat
fails under Linux 2.6.

----------------
I have added debug output here:
diff -u -r1.6 avc_bin.c
--- avc_bin.c   28 Feb 2004 06:39:50 -0000      1.6
+++ avc_bin.c   18 Aug 2004 20:15:23 -0000
@@ -794,6 +794,8 @@
     {
         for(i=0; i<numVertices; i++)
         {
+       fprintf(stderr,"F x %f y%f \n", psArc->pasVertices[i].x,
psArc->pasVertices[i].y);
+
             psArc->pasVertices[i].x = AVCRawBinReadFloat(psFile);
             psArc->pasVertices[i].y = AVCRawBinReadFloat(psFile);
         }
@@ -802,12 +804,13 @@
     {
         for(i=0; i<numVertices; i++)
         {
+       fprintf(stderr,"D x %f y%f \n", psArc->pasVertices[i].x,
psArc->pasVertices[i].y);
+
             psArc->pasVertices[i].x = AVCRawBinReadDouble(psFile);
             psArc->pasVertices[i].y = AVCRawBinReadDouble(psFile);
         }

     }
-
     /*-----------------------------------------------------------------
      * Record size may be larger than number of vertices.  Skip up to
      * start of next object.
---------------
Result (2.6 kernel):

ogrinfo -summary olson ARC 2> /tmp/tt
Had to open data source read-only.
INFO: Open of `olson'
using driver `AVCBin' successful.

Layer name: ARC
Geometry: Line String
Feature Count: 15750
Extent: (-170128861896635013341904755760727851008.000000,
-170140544564566716704068949377468596224.000000) - (0.000000, 0.000000)
Layer SRS WKT:
(unknown)
UserId: Integer (0.0)
FNODE#: Integer (0.0)
TNODE#: Integer (0.0)
LPOLY#: Integer (0.0)
RPOLY#: Integer (0.0)

cat /tmp/tt
[...]
F x 0.000000 y0.000000 
F x 0.000000 y0.000000 
F x 0.000000 y0.000000 
F x 0.000000 y0.000000 
F x 224.563480 y0.000000 
F x
191643330841768927342084085033433925451552693687906722345527511880200385024792998748352174193514894236997257005748407145027529354124648554083522870967005114092307857204220633569495810823890727975398052277063986710577152.000000
y2319211256025804177408.000000 
F x 0.000000 y0.000000 
[...]
F x 0.000000 y0.000000 
F x 0.000000 y224.563480 
F x 0.000000 y0.000000 
F x 0.000000 y0.000000 
F x 0.000000 y0.000000 
F x 0.000000 y0.000000 
F x 224.563480 y0.000000 
[...]
F x 0.000000 y0.000000 
F x 0.000000 y0.000000 
F x 0.000000 y-169941949350932561373112869866501570560.000000 
F x 0.000000 y-169944545499361828786927135114666180608.000000 
[...]
F x 0.000000 y-169895218679205747924456095399538589696.000000 
F x 0.000000 y-169892622530776480510641830151373979648.000000 
F x -169791372742035051371885485472954187776.000000
y-169892622530776480510641830151373979648.000000 
F x -169791372742035051371885485472954187776.000000
y-169895218679205747924456095399538589696.000000 
F x -169786180445176516544256954976624967680.000000
y-169895218679205747924456095399538589696.000000 
F x -169780988148317981716628424480295747584.000000
y-169895218679205747924456095399538589696.000000 
F x -169775795851459446888999893983966527488.000000
y-169895218679205747924456095399538589696.000000 
F x -169770603554600912061371363487637307392.000000
y-169895218679205747924456095399538589696.000000 
F x -169973103132083770338884052844476891136.000000
y-169895218679205747924456095399538589696.000000 
F x -169975699280513037752698318092641501184.000000
y-169895218679205747924456095399538589696.000000 
F x -169978295428942305166512583340806111232.000000
y-169895218679205747924456095399538589696.000000 
F x -169980891577371572580326848588970721280.000000
y-169895218679205747924456095399538589696.000000 
F x -169980891577371572580326848588970721280.000000
y-169892622530776480510641830151373979648.000000 
F x 0.000000 y-169892622530776480510641830151373979648.000000 
[...]

So the output of AVCRawBinReadFloat() (or AVCRawBinReadDouble() )
is not as it should be.

> Do the geometries look ok if you do a full ogrinfo on the AVC layer

Had to open data source read-only.
INFO: Open of `olson'
using driver `AVCBin' successful.

Layer name: ARC
Geometry: Line String
Feature Count: 15750
Extent: (-170128861896635013341904755760727851008.000000,
-170140544564566716704068949377468596224.0
00000) - (0.000000, 0.000000)
Layer SRS WKT:
(unknown)
UserId: Integer (0.0)
FNODE#: Integer (0.0)
TNODE#: Integer (0.0)
LPOLY#: Integer (0.0)
RPOLY#: Integer (0.0)
OGRFeature(ARC):1
  UserId (Integer) = 0
  FNODE# (Integer) = 2
  TNODE# (Integer) = 1
  LPOLY# (Integer) = 1
  RPOLY# (Integer) = 2

OGRFeature(ARC):2
  UserId (Integer) = 0
  FNODE# (Integer) = 3
  TNODE# (Integer) = 4
  LPOLY# (Integer) = 2
  RPOLY# (Integer) = 4
  LINESTRING (0.000 -169908199421352084993527421640361639936.000,0.000
-1699081994213520849935274216
40361639936.000,0.000 -169908199421352084993527421640361639936.000,0.000
-16990819942135208499352742
1640361639936.000,0.000 -169908199421352084993527421640361639936.000,0.000
-169908199421352084993527
421640361639936.000,0.000 -169908199421352084993527421640361639936.000,0.000
-1699081994213520849935
27421640361639936.000,0.000 -169908199421352084993527421640361639936.000,0.000
-16990819942135208499
...

No, looking less than good.

... I'm completely in the dark how to track this down.
Further debug suggestions welcome.

Best regards

 Markus

[warmerdam]

Markus, 

I think you put the fprintf() calls before the values in the point arrays
are set, instead of just after. 

However, the conclusion from seeing the linestring output of ogrinfo is that
the geometries read are corrupt.  

At this point it would be nice to be able to debug into AVCRawBinReadBytes()
(used by AVCRawBinReadDouble).  However, adding fprintf() calls there is likely
to produce a lot of noisy output.  Any chance of my getting ssh login access
to this machine for an hour or two?

[neteler@…]

Frank,

yes, I am obviously too tired (0:55).

diff -u -r1.6 avc_bin.c
--- ogr/ogrsf_frmts/avc/avc_bin.c       28 Feb 2004 06:39:50 -0000      1.6
+++ ogr/ogrsf_frmts/avc/avc_bin.c       18 Aug 2004 23:17:28 -0000
@@ -794,20 +794,23 @@
     {
         for(i=0; i<numVertices; i++)
         {
+
             psArc->pasVertices[i].x = AVCRawBinReadFloat(psFile);
             psArc->pasVertices[i].y = AVCRawBinReadFloat(psFile);
+       fprintf(stderr,"F x %f y%f \n", psArc->pasVertices[i].x,
psArc->pasVertices[i].y);
         }
     }
     else
     {
         for(i=0; i<numVertices; i++)
         {
+
             psArc->pasVertices[i].x = AVCRawBinReadDouble(psFile);
             psArc->pasVertices[i].y = AVCRawBinReadDouble(psFile);
+       fprintf(stderr,"D x %f y%f \n", psArc->pasVertices[i].x,
psArc->pasVertices[i].y);
         }

     }
-
     /*-----------------------------------------------------------------
      * Record size may be larger than number of vertices.  Skip up to
      * start of next object.

Here again:

F x 0.000000 y-169941949350932561373112869866501570560.000000
F x 0.000000 y-169944545499361828786927135114666180608.000000
F x 0.000000 y-169944545499361828786927135114666180608.000000
F x 0.000000 y-169944545499361828786927135114666180608.000000

So the problem remains.

Of course you can have a login. But right now I am over telephone
(56k). Tomorrow I can take the laptop to the institute and
provide access to you (if possible in your morning time).

To give an idea about the time shift:
date --utc
Mi Aug 18 23:21:25 UTC 2004
date
Do Aug 19 01:21:34 CEST 2004

Best regards

 Markus

[warmerdam]

I have tried this on a Linux 2.6 system and wasn't able to reproduce the 
problem:

warmerda@cvs.remotesensing% gcc --version
gcc (GCC) 3.3.3 20040412 (Red Hat Linux 3.3.3-7)
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

warmerda@cvs.remotesensing% uname -a
Linux cvs.flids.com 2.6.6-1.435 #1 Mon Jun 14 09:09:07 EDT 2004 i686 i686 i386
GNU/Linux

[warmerdam]

The problem is one that only appears in optimized mode with modern GCC versions.

In particular, there is a warning issued about 'type punning' and aliases
with the following code:

        *(GUInt32*)(&fValue) = CPL_SWAP32(*(GUInt32*)(&fValue));

Changing this to the following corrects the problem (as does building without
optimization):
        CPL_SWAP32PTR( &fValue );


I can't say I entirely understand why the first approach is a problem, but
apparently it fools the optimizer into not realizing that the value of
fValue is changed and so the unswapped value from a register is returned 
instead of the swapped value really in the stack location. 

I am correcting this upstream (in two places in avc_rawbin.c) and will 
incorporate the change into GDAL.  I'm also running a complete build on 
Markus' machine so I can search the log for other similar build warnings
that should be looked into.  I've seen the warnings before on occation but
not thought them too serious. 

[warmerdam]

For reference, the warnings look like this:

avc_rawbin.c: In function `AVCRawBinReadFloat':
avc_rawbin.c:468: warning: dereferencing type-punned pointer will break
strict-aliasing rules
avc_rawbin.c:468: warning: dereferencing type-punned pointer will break
strict-aliasing rules

[neteler@…]

Frank,

thanks so much for fixing it. Was a pleasure
to "host" you here in my laptop via phone!

Markus

[warmerdam]

Finished upstream changes in Daniel's avce00 CVS tree and pulled latest
version of this code back into GDAL and committed it. 

Closing bug report.

Thanks for all your assistance on this Markus!