crash loading big shapefile

[msieczka]

QGIS crashes loading a 382 MB shapefile, 180514 polygons:

$ ogrinfo -al -so neg_patch_poly_bnd_area_splt

INFO: Open of `robota/fucha/2008_javier/neg_patch_poly_bnd_area_splt'
      using driver `ESRI Shapefile' successful.

Layer name: neg_patch_poly_bnd_area_splt
Geometry: Polygon
Feature Count: 180514
Extent: (-180.000000, -90.000000) - (180.000000, 90.000000)
Layer SRS WKT:
GEOGCS["GCS_WGS_1984",
    DATUM["WGS_1984",
        SPHEROID["WGS_1984",6378137,298.257223563]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
cat: Real (11.0)

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f6abcda2710 (LWP 21711)]
0x00007f6abb8f80fc in ?? () from /usr/lib/libQtGui.so.4
(gdb) bt
#0  0x00007f6abb8f80fc in ?? () from /usr/lib/libQtGui.so.4
#1  0x00007f6abb8c1295 in ?? () from /usr/lib/libQtGui.so.4
#2  0x00007f6abb8bde4f in QStroker::processCurrentSubpath ()
   from /usr/lib/libQtGui.so.4
#3  0x00007f6abb8beee0 in QStrokerOps::strokePath ()
   from /usr/lib/libQtGui.so.4
#4  0x00007f6abb8f758a in ?? () from /usr/lib/libQtGui.so.4
#5  0x00007f6abc6d38ef in QgsVectorLayer::drawPolygon (this=0x2b2b980, 
    feature=0x7f6a69433010 "\001\003", p=0x7fffc4edde70, mtp=0x1fa2c78, 
    ct=0x0, drawingToEditingCanvas=true)
    at /home/shoofi/src/straight/qgis-trunk/src/core/qgsvectorlayer.cpp:649
#6  0x00007f6abc6d4908 in QgsVectorLayer::drawFeature (this=0x2b2b980, 
    p=0x7fffc4edde70, fet=@0x7fffc4edd8d0, theMapToPixelTransform=0x1fa2c78, 
    ct=0x0, marker=0x7fffc4edd920, widthScale=3.8582677165354333, 
    rasterScaleFactor=1, drawingToEditingCanvas=true)
    at /home/shoofi/src/straight/qgis-trunk/src/core/qgsvectorlayer.cpp:3389
#7  0x00007f6abc6d5043 in QgsVectorLayer::draw (this=0x2b2b980, 
    rendererContext=@0x1fa2c38)
    at /home/shoofi/src/straight/qgis-trunk/src/core/qgsvectorlayer.cpp:767
#8  0x00007f6abc685776 in QgsMapRenderer::render (this=0x1fa2ba0, 
    painter=0x7fffc4edde70)
    at /home/shoofi/src/straight/qgis-trunk/src/core/qgsmaprenderer.cpp:340
#9  0x00007f6abca85049 in QgsMapCanvasMap::render (this=0x1fc0630)
    at /home/shoofi/src/straight/qgis-trunk/src/gui/qgsmapcanvasmap.cpp:84
#10 0x00007f6abca7fc72 in QgsMapCanvas::refresh (this=0x1f99700)
    at /home/shoofi/src/straight/qgis-trunk/src/gui/qgsmapcanvas.cpp:365
#11 0x00000000004c517e in QgisApp::addVectorLayers (this=0x1eee610, 
    theLayerQStringList=@0x7fffc4ede1a0, enc=@0x7fffc4ede190)
    at /home/shoofi/src/straight/qgis-trunk/src/app/qgisapp.cpp:2171
#12 0x00000000004c6c64 in QgisApp::addVectorLayer (this=0x1eee610)
    at /home/shoofi/src/straight/qgis-trunk/src/app/qgisapp.cpp:2109
#13 0x000000000069b636 in QgisApp::qt_metacall (this=0x1eee610, 
    _c=QMetaObject::InvokeMetaMethod, _id=86, _a=0x7fffc4ede3b0)
    at /home/shoofi/src/straight/qgis-trunk/build/src/app/moc_qgisapp.cxx:328
#14 0x00007f6abc27c6d4 in QMetaObject::activate () from /usr/lib/libQtCore.so.4
#15 0x00007f6abb786d77 in QAction::triggered () from /usr/lib/libQtGui.so.4
#16 0x00007f6abb787540 in QAction::activate () from /usr/lib/libQtGui.so.4
#17 0x00007f6abba97cca in ?? () from /usr/lib/libQtGui.so.4
#18 0x00007f6abba97f65 in QAbstractButton::mouseReleaseEvent ()
   from /usr/lib/libQtGui.so.4
#19 0x00007f6abbb5f74a in QToolButton::mouseReleaseEvent ()
   from /usr/lib/libQtGui.so.4
#20 0x00007f6abb7df139 in QWidget::event () from /usr/lib/libQtGui.so.4
#21 0x00007f6abb78ca5d in QApplicationPrivate::notify_helper ()
   from /usr/lib/libQtGui.so.4
#22 0x00007f6abb79504a in QApplication::notify () from /usr/lib/libQtGui.so.4
#23 0x00007f6abc638ebe in QgsApplication::notify (this=0x7fffc4edfc20, 
    receiver=0x1f6f710, event=0x7fffc4eded40)
    at /home/shoofi/src/straight/qgis-trunk/src/core/qgsapplication.cpp:78
#24 0x00007f6abc268381 in QCoreApplication::notifyInternal ()
   from /usr/lib/libQtCore.so.4
#25 0x00007f6abb7943e8 in QApplicationPrivate::sendMouseEvent ()
   from /usr/lib/libQtGui.so.4
#26 0x00007f6abb7f8909 in ?? () from /usr/lib/libQtGui.so.4
#27 0x00007f6abb7f77ff in QApplication::x11ProcessEvent ()
   from /usr/lib/libQtGui.so.4
#28 0x00007f6abb81ec84 in ?? () from /usr/lib/libQtGui.so.4
#29 0x00007f6ab80c778b in g_main_context_dispatch ()
   from /usr/lib/libglib-2.0.so.0
#30 0x00007f6ab80caf5d in ?? () from /usr/lib/libglib-2.0.so.0
#31 0x00007f6ab80cb11b in g_main_context_iteration ()
   from /usr/lib/libglib-2.0.so.0
#32 0x00007f6abc29070f in QEventDispatcherGlib::processEvents ()
   from /usr/lib/libQtCore.so.4
#33 0x00007f6abb81e44f in ?? () from /usr/lib/libQtGui.so.4
#34 0x00007f6abc266ca2 in QEventLoop::processEvents ()
   from /usr/lib/libQtCore.so.4
#35 0x00007f6abc266e2d in QEventLoop::exec () from /usr/lib/libQtCore.so.4
#36 0x00007f6abc2692dd in QCoreApplication::exec ()
   from /usr/lib/libQtCore.so.4
#37 0x00000000004b5639 in main (argc=1, argv=0x7fffc4ee0318)
    at /home/shoofi/src/straight/qgis-trunk/src/app/main.cpp:683

Debian testing amd64, QGIS SVN trunk r9770, QT 4.4.3, GDAL 1.6 SVN trunk r15902 (05.12.2008).

[hdus]

Works for me. I just loaded a shape with size of 1.2GB and more than 1000000 polygons without any problems regardless whether I use on the fly projection to WGS84 or not. Did you check your data?

[msieczka]

Replying to hdus:

Works for me. I just loaded a shape with size of 1.2GB and more than 1000000 polygons without any problems regardless whether I use on the fly projection to WGS84 or not. Did you check your data?

The data is OK AFAICT. I can easily process it with OGR tools, export/import in GRASS. Will soon try rendering with MapServer and I'll post the result if I remember to.

Interesting - the Shapefile crashes whole Xorg (!!!) if "Fix problems..." is set ON in Options>Rendering, and only crashes QGIS alone if the option is set OFF.

I can't share the data in public. It's for sale. I could share it with an interested QGIS developer in private if he promises not to redistribute it.

[hdus]

Replying to msieczka:

Replying to hdus:

Works for me. I just loaded a shape with size of 1.2GB and more than 1000000 polygons without any problems regardless whether I use on the fly projection to WGS84 or not. Did you check your data?

The data is OK AFAICT. I can easily process it with OGR tools, export/import in GRASS. Will soon try rendering with MapServer and I'll post the result if I remember to.

Interesting - the Shapefile crashes whole Xorg (!!!) if "Fix problems..." is set ON in Options>Rendering, and only crashes QGIS alone if the option is set OFF.

That's the difference to my attempt. I had swiched off this option. Now I switched "Fix problems ..." on, but every works fine for me.

I can't share the data in public. It's for sale. I could share it with an interested QGIS developer in private if he promises not to redistribute it.

[msieczka]

I have isolated the smalest single polygon that crashes Xorg when loaded to QGIS. It's 2.5 MB. You can download it at ​http://www.sieczka.org/tmp/feat_8.7z (0.5 MB). It's a subset of the big Shapefile I originaly reported about. I have 4 more such polygons (larger though) if needed.

ogr2ogr can process the polygon just fine. It also renders OK with MapServer's shp2img. As well as QGIS with "Fix problems..." set ON in Options>Rendering set OFF.

Yet it constantly crashes Xorg when loaded to QGIS with "Fix problems..." set ON. Differently than with the original big shapefile, if "Fix problems..." is set OFF there's no crash with this single polygon. After looking more into this, it showed that the original big shapefile crashes QGIS because the memory runs out.

So probably two issues here - QGIS crashes when it runs out of memory with big shapefiles AND it makes Xorg crash rendering certain shapes.

[pcav]

The shp causing the crash is no longer available. Is the bug still valid? Please check, and if it is, make the shp available (if not, just close the ticket). Thanks

[pcav]

According to our new bug policy, I'm closing this pending feedback from the user. In case it is still valid, please reopen it. Sorry for the additional trouble.

[(none)]

Milestone Version 1.0.2 deleted

[sieczka]

Replying to pcav:

The shp causing the crash is no longer available. Is the bug still valid? Please check, and if it is, make the shp available (if not, just close the ticket).

I can again reproduce the crash with GSSHS 2.0 Shapefiles: ​ftp://ftp.soest.hawaii.edu/pwessel/gshhs/GSHHS_shp_2.0.zip.

QGIS trunk r11495, GDAL 1.6.2, QT 4.4.3, GEOS 3.1.0, Debian stable amd64.

[lutra]

Hi,

I have loaded all the shapes included in the archive you linked (including a a 180000 polygon one) and I have no crashes. Qgis trunk compiled under ubuntu 9.04 32 bit with 512mb of ram.

[lutra]

Tested also under windows xp 32 bit, 512mb of ram with qgis 1.2 (osgeo4w). No problems at all.

Can anyone on a 64bit platform make a test and report back? Thanks.

[trapanator]

Big shapefile loading works on a x64 system

[trapanator]

Works to me on x64 system, with QGIS trunk. See my attachment.

​https://trac.osgeo.org/qgis/attachment/ticket/1452/Schermata.jpg

closed fixed.