Perl tests crash on telascience-stable builder

[Mateusz Łoskot]

Recently, Perl tests have been seriously crashing on the telascience-stable, here is one of recent logs:

​http://buildbot.osgeo.org:8500/telascience-stable/builds/42/step-perl-test/0

In particular:

                                                                             t/ogr.....ok 170/0
                                                                             
t/ogr.....ok 171/0*** glibc detected *** /usr/bin/perl: free(): invalid next size (fast): 0x084f3578 ***

command timed out: 1200 seconds without output, killing pid 17269

I tried to reproduce this problem on my Linux box (Ubuntu 7.04) but tests output is surprisingly short:

./configure (...)
make clean && make
cd swig/perl
perl Makefile.PL
make build
make test
...
mloskot@dog:~/dev/gdal/_svn/branches/1.4/gdal/swig/perl$ make test
make -f Makefile_Geo__GDAL
make[1]: Entering directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make[1]: Leaving directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make -f Makefile_Geo__GDAL__Const
make[1]: Entering directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make[1]: Leaving directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make -f Makefile_Geo__OGR
make[1]: Entering directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make[1]: Leaving directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make -f Makefile_Geo__OSR
make[1]: Entering directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make[1]: Leaving directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
make -f Makefile_Geo__GDAL test
make[1]: Entering directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/gdal....ok 268/0                                                           
Available drivers were AAIGrid, AIG, AirSAR, BMP, BSB, BT, CEOS, CPG, DIPEx, DOQ1, DOQ2, DTED, EHdr, ELAS, ENVI, ESAT, FAST, FIT, FujiBAS, GIF, GMT, GSC, GTiff, GXF, HFA, IDA, ILWIS, ISIS2, JDEM, JP2KAK, JPEG, L1B, LAN, Leveller, MEM, MFF, MFF2, MrSID, NDF, NITF, PAux, PCIDSK, PCRaster, PDS, PNG, PNM, RIK, RMF, RS2, RST, SAR_CEOS, SDTS, SGI, USGSDEM, VRT, WCS, XPM, netCDF
Drivers used in tests were: GTiff, MEM, EHdr
t/gdal....ok                                                                 
t/ogr.....ok 1/0                                                             
Available drivers were AVCBin, CSV, DGN, ESRI Shapefile, GML, Interlis 1, Interlis 2, KML, MapInfo File, Memory, MySQL, OCI, PostgreSQL, REC, S57, SDTS, SQLite, TIGER, UK .NTF, VRT
Drivers used in tests were: ESRI Shapefile, MapInfo File, Memory
t/ogr.....ok                                                                 
All tests successful.
Files=2, Tests=674,  1 wallclock secs ( 0.38 cusr +  0.12 csys =  0.50 CPU)
make[1]: Leaving directory `/home/mloskot/dev/gdal/_svn/branches/1.4/gdal/swig/perl'
mloskot@dog:~/dev/gdal/_svn/branches/1.4/gdal/swig/perl$ 

Any idea what's going wrong with the crashing tests and the short output?

[warmerdam]

Added Ari in case he has thoughts.

Can you try this under valgrind?

[Ari Jolma]

All tests are succesful with r12521 and ./configure;make;make install;cd swig/perl;rm *wrap*;make generate;make build;make test in my Fedora Moonshine

Ari

[warmerdam]

I see similar brief results like mloskot. I tried running these under valgrind and there were no apparent problems.

Ari, any thoughts on why the buildbot is running hundreds of tests, and we only seem to be running a few?

[Ari Jolma]

The text that stays on the console may be like

t/gdal....ok                                                                 
t/ogr.....ok 1/0 
             ***

but later you see

Files=2, Tests=700, 1 wallclock secs ( 0.63 cusr +  0.16 csys =  0.79 CPU)
               ***

The Test module I use does not linefeed in interactive use and while it shows each of the test result, it goes so fast that you really don't see them.

[warmerdam]

Ari - understood, thanks.

I see the gdal 1.4 stable python tests are also failing similarly:

​http://buildbot.osgeo.org:8500/telascience-stable/builds/43/step-test/0

So I suspect the issue is not related to the bindings at all.

Mateusz, I'm sorry to do this, but I think you will need to isolate this problem on buildbot.osgeo.org. You might want to start by verifying you have a clean build and no junk lying around.

The problem seems to be only in 1.4, so I think we need to isolate it before 1.4.3 release.

[Mateusz Łoskot]

Frank, no problem.

I'm testing it on the telascience-stable (from command line, as buildbot user, env configured in the same way as for buildbot instance) and surprisingly, it works without any error:

[buildbot@xblade14-2 perl]$ make test
make -f Makefile_Geo__GDAL
make[1]: Entering directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make[1]: Leaving directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make -f Makefile_Geo__GDAL__Const
make[1]: Entering directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make[1]: Leaving directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make -f Makefile_Geo__OGR
make[1]: Entering directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make[1]: Leaving directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make -f Makefile_Geo__OSR
make[1]: Entering directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make[1]: Leaving directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
make -f Makefile_Geo__GDAL test
make[1]: Entering directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/gdal....ok 333/0                                                           
Available drivers were AAIGrid, AIG, AirSAR, BMP, BSB, BT, CEOS, CPG, DIPEx, DOQ1, DOQ2, DTED, EHdr, ELAS, ENVI, ESAT, FAST, FIT, FujiBAS, GIF, GMT, GSC, GTiff, GXF, HFA, IDA, ILWIS, ISIS2, JDEM, JPEG, L1B, LAN, Leveller, MEM, MFF, MFF2, NDF, NITF, PAux, PCIDSK, PCRaster, PDS, PNG, PNM, RIK, RMF, RS2, RST, SAR_CEOS, SDTS, SGI, USGSDEM, VRT, WCS, XPM, netCDF
Drivers used in tests were: GTiff, MEM, EHdr
t/gdal....ok                                                                 
t/ogr.....ok 337/0                                                           
Available drivers were AVCBin, CSV, DGN, ESRI Shapefile, GML, KML, MapInfo File, Memory, ODBC, PGeo, PostgreSQL, REC, S57, SDTS, SQLite, TIGER, UK .NTF, VRT
Drivers used in tests were: ESRI Shapefile, MapInfo File, Memory
t/ogr.....ok                                                                 
All tests successful.
Files=2, Tests=670,  2 wallclock secs ( 0.86 cusr +  0.25 csys =  1.11 CPU)
make[1]: Leaving directory `/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl'

[buildbot@xblade14-2 perl]$ pwd
/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal/swig/perl

[buildbot@xblade14-2 perl]$ echo $LD_LIBRARY_PATH 
/osgeo/buildbot/gdal/usr/lib:/osgeo/buildbot/gdal/buildslave/telascience-stable/build/gdal

[Mateusz Łoskot]

It's a bit mysterious, but I removed the src tree completely, restarted the buildbot and it's green now :-)

I'm closing this bug assuming there is no problem in Perl bindings. Sorry, for the false alarm.