Opened 7 years ago

Last modified 7 years ago

#3365 closed defect

r.in.gdal: crash on large dataset — at Version 2

Reported by: neteler Owned by: grass-dev@…
Priority: normal Milestone: 7.2.2
Component: Raster Version: svn-releasebranch72
Keywords: r.in.gdal Cc:
CPU: x86-64 Platform: Linux

Description (last modified by neteler)

Trying to import a large VRT dataset, I get a crash:

GRASS 7.2.2svn (latlong):~/software/grass72_svn_debugmode/bin.x86_64-pc-linux-gnu > r.in.gdal input=/mnt/kvm7-storage/geodata/global_forest/GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015/loss_global_forest_cover_loss_2000_2015.vrt output=loss_global_forest_cover_loss_2000_2015
Proceeding with import of 1 raster bands...
Importing raster map <loss_global_forest_cover_loss_2000_2015>...
Segmentation fault (core dumped)

Debugging:

GRASS 7.2.2svn (latlong):~/software/grass72_svn_debugmode/bin.x86_64-pc-linux-gnu > gdb r.in.gdal
GNU gdb (GDB) Fedora 7.12.1-48.fc25
Copyright (C) 2017 Free Software Foundation, Inc.
[...]
Reading symbols from r.in.gdal...done.
(gdb) r input=/mnt//GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015.vrt output=loss_global_forest_cover_loss_2000_2015
Starting program: /home/mundialis/software/grass72_svn_debugmode/dist.x86_64-pc-linux-gnu/bin/r.in.gdal input=/mnt//GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015.vrt output=loss_global_forest_cover_loss_2000_2015
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.24-8.fc25.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Proceeding with import of 1 raster bands...
Importing raster map <loss_global_forest_cover_loss_2000_2015>...

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff77a1ab9 in put_data (fd=0, null_buf=0x7fffffe9b020 "", cell=0x7fffc62ac010, row=0, n=1440000, zeros_r_nulls=0) at put_row.c:370
370		compressed_buf[0] = work_buf[0] = nbytes;



(gdb) bt full
#0  0x00007ffff77a1ab9 in put_data (fd=0, null_buf=0x7fffffe9b020 "", cell=0x7fffc62ac010, row=0, n=1440000, zeros_r_nulls=0) at put_row.c:370
        wk = 0x7fffff91cb31 ""
        nbytes = 1
        compressed_buf = 0x7fffff7bd220 <error: Cannot access memory at address 0x7fffff7bd220>
        total = 1440000
        fcb = 0x691080
        compressed = 1
        len = 4
        work_buf = 0x7fffff91cb30 "\001"
        wk = 0x7fffff91cb31 ""
        nwrite = 0
#1  0x00007ffff77a1f98 in put_raster_data (fd=0, null_buf=0x7fffffe9b020 "", rast=0x7fffc62ac010, row=0, n=1440000, zeros_r_nulls=0, map_type=0)
    at put_row.c:476
        fcb = 0x691080
#2  0x00007ffff77a2bdb in put_raster_row (fd=0, buf=0x7fffc62ac010, data_type=0, zeros_r_nulls=0) at put_row.c:699
        convert_and_write_FtypeOtype = {{0x0, 0x7ffff77a23d9 <convert_and_write_if>, 0x7ffff77a25f5 <convert_and_write_id>}, {
            0x7ffff77a280f <convert_and_write_fi>, 0x0, 0x7ffff77a26f3 <convert_and_write_fd>}, {0x7ffff77a2928 <convert_and_write_di>, 
            0x7ffff77a24d7 <convert_and_write_df>, 0x0}}
        fcb = 0x691080
        null_buf = 0x7fffffe9b020 ""
#3  0x00007ffff77a0f8b in Rast_put_row (fd=0, buf=0x7fffc62ac010, data_type=0) at put_row.c:67
No locals.
#4  0x0000000000406e4d in ImportBand (hBand=0x7dbcb0, output=0x627dc0 "loss_global_forest_cover_loss_2000_2015", group_ref=0x0) at main.c:1141
        data_type = 0
        eGDT = GDT_Int32
        eRawGDT = GDT_Byte
        row = 1
        nrows = 560000
        ncols = 1440000
        complex = 0
        cf = 0
        cfR = 32767
        cfI = -145625496
        bNoDataEnabled = 0
        indx = 32767
        cell = 0x7fffc62ac010
        cellReal = 0x77
        cellImg = 0x7fffffffac40
        bufComplex = 0x7fffffffab40
        dfNoData = -10000
        outputReal = "0\341\377\367\377\177\000\000\260\254\377\377\377\177\000\000\071\031@\000\000\000\000\000\243\366\251\242\000\000\000\000\377\377\377\377", '\000' <repeats 12 times>, "H\017\064\366\377\177\000\000\000\240\374\367\377\177\000\000\000\000\000\000\000\000\360?", '\000' <repeats 14 times>, "\360?", '\000' <repeats 30 times>, "\360?\000\000\000\000\000\000\000\000\220\062@\000\000\000\000\000\314\345w\367\377\177\000\000P\255\377\377\377\177\000\000\350\262`\000\000\000\000\000\220\062@\000\000\000\000\000\320\332\377\377\377\177", '\000' <repeats 18 times>, "\360\331\377\377\377\177\000\000"...
        outputImg = '\000' <repeats 32 times>, "\377\000\000\000\000\000\377\377\000\000\000\000\000\000\377\377\000\000\000\000\000\000\377\377\377\377\377\377\000\377\377\377", '\000' <repeats 22 times>, "\360?", '\000' <repeats 14 times>, "\360?", '\000' <repeats 14 times>, "\360?\000\000\000\000\000\000\000\000\210\345w\367\377\177", '\000' <repeats 26 times>, "\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\200\256\377\377\377\177\000\000\240\224z\367\377\177\000\000\000\273}", '\000' <repeats 13 times>...
        nullFlags = 0x0
        history = {fields = {0x7ffff7fca000 "", 0x7fffffffabe8 "", 0x7fffffffabe4 "", 0x7ffff7ff7b78 "\350\343\377\367\377\177", 
            0x7fffffffaba0 "\300\254\377\377\377\177", 0x401939 "GDALGetRasterBand", 0x400be8 "\261\a", 0x7fffffffabe8 ""}, nlines = -1565919581, 
          lines = 0x28aa7da}
        GDALmetadata = 0x7ffff77aa094
        have_colors = 0
        gdal_rat = 0x7ffff752f25f <find_file1+198>
#5  0x0000000000405231 in main (argc=3, argv=0x7fffffffdad8) at main.c:619
        nBand = 1
        input = 0x627d20 "/mnt/GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015.vrt"
        output = 0x627dc0 "loss_global_forest_cover_loss_2000_2015"
        title = 0x0
        cellhd = {format = 0, compressed = 0, rows = 560000, rows3 = 560000, cols = 1440000, cols3 = 1440000, depths = 1, proj = 3, zone = 0, 
          ew_res = 0.00025000000000000017, ew_res3 = 0.00025000000000000017, ns_res = 0.00025000000000000017, ns_res3 = 0.00025000000000000017, tb_res = 1, 
          north = 80, south = -60.000000000000085, east = 180.00000000000023, west = -180, top = 1, bottom = 0}
        loc_wind = {format = 0, compressed = -1, rows = 1, rows3 = 1, cols = 1, cols3 = 1, depths = 1, proj = 3, zone = 0, ew_res = 1, ew_res3 = 1, 
          ns_res = 1, ns_res3 = 1, tb_res = 1, north = 1, south = 0, east = 1, west = 0, top = 1, bottom = 0}
        cur_wind = {format = 0, compressed = 0, rows = 0, rows3 = 0, cols = 0, cols3 = 0, depths = 0, proj = 0, zone = 0, ew_res = 0, ew_res3 = 0, 
          ns_res = 0, ns_res3 = 0, tb_res = 0, north = 0, south = 0, east = 0, west = 0, top = 0, bottom = 0}
        proj_info = 0x68ced0
        proj_units = 0x68cbf0
        loc_proj_info = 0x68e5f0
        loc_proj_units = 0x7db4f0
        hDS = 0x7dbb00
        hDriver = 0x62ef90
        hBand = 0x7dbcb0
        adfGeoTransform = {-180, 0.00025000000000000017, 0, 80, 0, -0.00025000000000000017}
        n_bands = 0
        force_imagery = 0
        error_msg = "\253\000\000\000\022", '\000' <repeats 19 times>, "i\000\000\000\022", '\000' <repeats 19 times>, "\222\000\000\000\022", '\000' <repeats 19 times>, "\231\000\000\000\022", '\000' <repeats 19 times>, "\001\000\000\000 \000\000\000@\323\377\377\377\177\000\000\260\327\377\377\377\177\000\000\350
Y\372\367\377\177\000\000\060U\372\367\377\177\000\000\001\000\000\000\000\000\000\000\235\066\313\335\377\177\000\000\260Q\336\367\377\177\000\000\000\000\000\000\000\000\000\000D|\336\367\377\177\000\000`O\372\367\377\177\000\000\210\337\377\367\377\177\000\000\000\000\000\000\000\000\000\000"...
        projcomp_error = 1
        overwrite = 0
        offset = 0
        suffix = 0x629130 ""
        num_digits = 0
        module = 0x7ffff777ea28 <state+40>
        parm = {input = 0x7ffff777ea88 <state+136>, output = 0x626ea0, target = 0x6270e0, title = 0x6271a0, outloc = 0x6274a0, band = 0x626f60, 
          memory = 0x627020, offset = 0x627260, num_digits = 0x627320, map_names_file = 0x6273e0, rat = 0x627560}
        flag_o = 0x7ffff777ea58 <state+88>
        flag_e = 0x626840
        flag_k = 0x627ab0
        flag_f = 0x627a10
        flag_l = 0x627a60
        flag_c = 0x627b00
        flag_p = 0x627b50
        flag_j = 0x626550
(gdb) 

Doing a convert operation with GDAL, no apparent issues (I CTRL-C'd it then):

gdal_translate /mnt/geodata/global_forest/GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015/loss_global_forest_cover_loss_2000_2015.vrt /scratch/bla.tif
Input file size is 1440000, 560000
0..

Data source: http://earthenginepartners.appspot.com/science-2013-global-forest/download_v1.3.html

I created a VRT from the "loss" layer tiles which I then tried to import as above:

gdalbuildvrt loss_global_forest_cover_loss_2000_2015.vrt *.tif

gdalinfo loss_global_forest_cover_loss_2000_2015.vrt
Driver: VRT/Virtual Raster
[...]
Size is 1440000, 560000
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-180.000000000000000,80.000000000000000)
Pixel Size = (0.000250000000000,-0.000250000000000)
Corner Coordinates:
Upper Left  (-180.0000000,  80.0000000) (180d 0' 0.00"W, 80d 0' 0.00"N)
Lower Left  (-180.0000000, -60.0000000) (180d 0' 0.00"W, 60d 0' 0.00"S)
Upper Right ( 180.0000000,  80.0000000) (180d 0' 0.00"E, 80d 0' 0.00"N)
Lower Right ( 180.0000000, -60.0000000) (180d 0' 0.00"E, 60d 0' 0.00"S)
Center      (   0.0000000,  10.0000000) (  0d 0' 0.00"E, 10d 0' 0.00"N)
Band 1 Block=128x128 Type=Byte, ColorInterp=Gray

Test with GRASS GIS 7.3:

GRASS 7.3.svn (latlong):~/software/grass73_svn > r.in.gdal input=/mnt/geodata/global_forest/GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015/loss_global_forest_cover_loss_2000_2015.vrt output=loss_global_forest_cover_loss_2000_2015
Importing raster map <loss_global_forest_cover_loss_2000_2015>...
0% Segmentation fault (core dumped)

System:

gcc -v
...
gcc version 6.3.1 20161221 (Red Hat 6.3.1-1) (GCC) 

uname -a 
Linux fedora-calc 4.9.12-200.fc25.x86_64 #1 SMP Thu Feb 23 19:31:49 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Change History (2)

comment:1 by mmetz, 7 years ago

I have experienced problems with importing GDAL VRT raster maps before, therefore decided to gdal_translate them before import.

Results of

  1. running the import again with valgrind --tool=memcheck and posting the valgrind log
  1. gdal_translate'ing the VRT to tif, then import the tif with r.in.gdal in order to test if this causes also a segfault

would help a lot.

comment:2 by neteler, 7 years ago

Description: modified (diff)

Using valgrind:

GRASS 7.3.svn (latlong):~ > valgrind $CMD
==30211== Memcheck, a memory error detector
==30211== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==30211== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==30211== Command: r.in.gdal input=/mnt/GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015/loss_global_forest_cover_loss_2000_2015.vrt output=loss_global_forest_cover_loss_2000_2015
==30211== 
Importing raster map <loss_global_forest_cover_loss_2000_2015>...
==30211== Warning: client switching stacks?  SP change: 0xffee9d140 --> 0xffe91ed30
==30211==          to suppress, use: --max-stackframe=5760016 or greater
==30211== Invalid write of size 8
==30211==    at 0x5273D89: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x52744B4: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x4063DE: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==    by 0x404F64: main (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==  Address 0xffe91ed28 is on thread 1's stack
==30211== 
==30211== Invalid read of size 8
==30211==    at 0x6D63E3F: lseek (in /usr/lib64/libc-2.24.so)
==30211==    by 0x5273D8D: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x52744B4: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x4063DE: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==    by 0x404F64: main (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==  Address 0xffe91ed28 is on thread 1's stack
==30211==  in frame #0, created by lseek (???:)
==30211== 
==30211== Invalid write of size 1
==30211==    at 0x527378C: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x52744B4: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x4063DE: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==    by 0x404F64: main (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==  Address 0xffe91ed34 is on thread 1's stack
==30211== 
==30211== Invalid write of size 1
==30211==    at 0x5273798: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x52744B4: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x4063DE: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==    by 0x404F64: main (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==  Address 0xffe91ed32 is on thread 1's stack
==30211== 
==30211== Invalid read of size 1
==30211==    at 0x527384E: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x52744B4: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==    by 0x4063DE: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==    by 0x404F64: main (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/bin/r.in.gdal)
==30211==  Address 0xffe91ed31 is on thread 1's stack
==30211== 
==30211== 
==30211== More than 10000000 total errors detected.  I'm not reporting any more.
==30211== Final error counts will be inaccurate.  Go fix your program!
==30211== Rerun with --error-limit=no to disable this cutoff.  Note
==30211== that errors may occur in your program without prior warning from
==30211== Valgrind, because errors are no longer being displayed.
==30211== 
==30211== 
==30211== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==30211==  Access not within mapped region at address 0xFFE7BF420
==30211==    at 0x5273CBA: ??? (in /home/mundialis/software/grass73_svn/dist.x86_64-pc-linux-gnu/lib/libgrass_raster.7.3.svn.so)
==30211==  If you believe this happened as a result of a stack
==30211==  overflow in your program's main thread (unlikely but
==30211==  possible), you can try to increase the size of the
==30211==  main thread stack using the --main-stacksize= flag.
==30211==  The main thread stack size used in this run was 8388608.
==30211== 
==30211== Process terminating with default action of signal 11 (SIGSEGV)
==30211==  Access not within mapped region at address 0xFFE7BF418
==30211==    at 0x4A286C0: _vgnU_freeres (vg_preloaded.c:59)
==30211==  If you believe this happened as a result of a stack
==30211==  overflow in your program's main thread (unlikely but
==30211==  possible), you can try to increase the size of the
==30211==  main thread stack using the --main-stacksize= flag.
==30211==  The main thread stack size used in this run was 8388608.
==30211== 
==30211== HEAP SUMMARY:
==30211==     in use at exit: 106,603,369 bytes in 11,066 blocks
==30211==   total heap usage: 78,802 allocs, 67,736 frees, 136,593,845 bytes allocated
==30211== 
==30211== LEAK SUMMARY:
==30211==    definitely lost: 134 bytes in 5 blocks
==30211==    indirectly lost: 353 bytes in 20 blocks
==30211==      possibly lost: 0 bytes in 0 blocks
==30211==    still reachable: 106,602,882 bytes in 11,041 blocks
==30211==         suppressed: 0 bytes in 0 blocks
==30211== Rerun with --leak-check=full to see details of leaked memory
==30211== 
==30211== For counts of detected and suppressed errors, rerun with: -v
==30211== ERROR SUMMARY: 10000000 errors from 5 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

Next I converted the VRT to GeoTIFF and tried to import that:

GRASS 7.3.svn (latlong):~ > nice gdal_translate -co "COMPRESS=DEFLATE" /mnt/global_forest/GFC-2015-v1.3/loss_global_forest_cover_loss_2000_2015/loss_global_forest_cover_loss_2000_2015.vrt /scratch/loss_global_forest_cover_loss_2000_2015.tif
Input file size is 1440000, 560000
0..10...20...30...40...50...60...70...80...90...100 - done.

GRASS 7.3.svn (latlong):~ > r.in.gdal /scratch/loss_global_forest_cover_loss_2000_2015.tif out=loss_global_forest_cover_loss_2000_2015
Importing raster map <loss_global_forest_cover_loss_2000_2015>...
Segmentation fault (core dumped)

The debug results are the same as before:

GRASS 7.2.2svn (latlong):~/software/grass72_svn_debugmode/bin.x86_64-pc-linux-gnu > gdb r.in.gdal
r GNU gdb (GDB) Fedora 7.12.1-48.fc25
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from r.in.gdal...done.
(gdb) r /scratch/loss_global_forest_cover_loss_2000_2015.tif out=loss_global_forest_cover_loss_2000_2015
Starting program: /home/mundialis/software/grass72_svn_debugmode/dist.x86_64-pc-linux-gnu/bin/r.in.gdal /scratch/loss_global_forest_cover_loss_2000_2015.tif out=loss_global_forest_cover_loss_2000_2015
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.24-8.fc25.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Proceeding with import of 1 raster bands...
Importing raster map <loss_global_forest_cover_loss_2000_2015>...

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff77a1ab9 in put_data (fd=0, null_buf=0x7fffffe9b080 "", cell=0x7fffc57fd010, row=0, n=1440000, zeros_r_nulls=0) at put_row.c:370
370		compressed_buf[0] = work_buf[0] = nbytes;

Note: See TracTickets for help on using tickets.