Opened 12 years ago

Last modified 6 years ago

#1752 new defect

Disabling Run Length Encoding (RLE) in 3D raster library

Reported by: huhabla Owned by: huhabla
Priority: major Milestone: 7.0.7
Component: Raster3D Version: svn-trunk
Keywords: RLE, 3D raster, compression Cc: grass-dev@…
CPU: All Platform: All

Description

This ticket refers to a discussion at the GRASS development mailing list about the RLE compression in the 3D raster library: "Grass SVN in Android, display issue" ​http://osgeo-org.1560.n6.nabble.com/Grass-SVN-in-Android-display-issue-td5002221i20.html

Request:

I would like to disable the use of Run Length Encoding (RLE) in the 3D raster library since there is a serious bug (wrong processing of strings larger than ~130000 chars).

Besides of that is the compression effect of the RLE in conjunction with a LZ77 compression not worth the computational effort and using RLE alone seems to create strange file sizes to me (see tests below). The existing LZ77 compression is IMHO sufficient.

The RLE code should not be removed for backward compatibility reasons. Maybe it can be removed in GRASS8?

Proof:

I have implemented the generation of 3D raster maps with different sizes and compression methods in the 3D raster library test suite to compare the RLE and LZ77 compression methods (r53296). The script to generate the comparison and the resulting text file are attached.

I generated 4 different 3D raster maps with 5 different sizes: 

a) 3D raster map with increasing values in x, y, and z direction
b) 3D raster map with zeros
c) 3D raster map with random values
d) 3D raster map with sparse random values: 70% 0, 10% 1, 10% 2 and 10% 3

The following sizes are computed (x * y * z): 

1.) 25 x 25 x 25     ->    15625 cells
2.) 50 x 50 x 50     ->   125000 cells
3.) 100 x 100 x 100  ->  1000000 cells
4.) 150 x 150 x 150  ->  3375000 cells
5.) 250 x 250 x 250  -> 15625000 cells

The tile size was set to 64 kilo byte.

Here the summary of the test run: 

1.) 15625 cells

No compression:
  a) 125056 bytes
  b) 125056 bytes
  c) 125056 bytes
  d) 125056 bytes

LZ77 compression only:
  a)  9588 bytes
  b)   349 bytes
  c) 69410 bytes
  d)  6746 bytes  

LZ77 and RLE compression:
  a)  9599 bytes
  b)   384 bytes
  c) 69358 bytes
  d)  6777 bytes  

RLE compression only:
  a) 109495 bytes
  b) 109495 bytes
  c) 109495 bytes
  d) 109495 bytes 

2.) 125000 cells

No compression:
  a) 1000148 bytes
  b) 1000148 bytes
  c) 1000148 bytes
  d) 1000148 bytes

LZ77 compression only:
  a)  49172 bytes
  b)   1701 bytes
  c) 538023 bytes
  d)  52754 bytes  

LZ77 and RLE compression:
  a)  49087 bytes
  b)   1764 bytes
  c) 537367 bytes
  d)  52679 bytes  

RLE compression only:
  a) 875372 bytes
  b) 875372 bytes
  c) 875372 bytes
  d) 875372 bytes 

3.) 1000000 cells

No compression:
  a) 8000518 bytes
  b) 8000518 bytes
  c) 8000518 bytes
  d) 8000518 bytes

LZ77 compression only:
  a)  208253 bytes
  b)   11108 bytes
  c) 4324676 bytes
  d)  415237 bytes  

LZ77 and RLE compression:
  a)  207288 bytes
  b)   10731 bytes
  c) 4319077 bytes
  d)  413469 bytes  

RLE compression only:
  a) 7001734 bytes
  b) 7001734 bytes
  c) 7001734 bytes
  d) 7001734 bytes 

4.) 3375000 cells

No compression:
  a) 27002901 bytes
  b) 27002901 bytes
  c) 27002901 bytes
  d) 27002901 bytes

LZ77 compression only:
  a)   997588 bytes
  b)    39644 bytes
  c) 14572465 bytes
  d)  1408884 bytes  

LZ77 and RLE compression:
  a)   993839 bytes
  b)    39427 bytes
  c) 14553084 bytes
  d)  1403304 bytes  

RLE compression only:
  a) 23631895 bytes
  b) 23631895 bytes
  c) 23631895 bytes
  d) 23631895 bytes 

5.) 15625000 cells

No compression:
  a) 125008331 bytes
  b) 125008331 bytes
  c) 125008331 bytes
  d) 125008331 bytes

LZ77 compression only:
  a)  4806063 bytes
  b)   179118 bytes
  c) 67723205 bytes
  d)  6507044 bytes  

LZ77 and RLE compression:
  a)  4787982 bytes
  b)   175862 bytes
  c) 67638694 bytes
  d)  6483027 bytes  

RLE compression only:
  a) 109404748 bytes
  b) 109404748 bytes
  c) 109404748 bytes
  d) 109404748 bytes

Attachments (5)

run_compression_analysis.sh​ (3.4 KB ) - added by huhabla 12 years ago.
Running the 3D raster test suite and analysing the file sizes for different compressions and region sizes
compression_run.txt​ (96.6 KB ) - added by huhabla 12 years ago.
Result of the test run
rast3d.diff​ (31.0 KB ) - added by huhabla 11 years ago.
raster3dlib.diff​ (61.1 KB ) - added by huhabla 11 years ago.
Patch for 3D raster library and include files
raster3dmodules.diff​ (11.7 KB ) - added by huhabla 11 years ago.
Patch for 3D raster modules

Download all attachments as: .zip

Change History (17)

by huhabla, 12 years ago

Attachment: run_compression_analysis.sh​ added

Running the 3D raster test suite and analysing the file sizes for different compressions and region sizes

by huhabla, 12 years ago

Attachment: compression_run.txt​ added

Result of the test run

in reply to:  description comment:1 by glynn, 12 years ago

Replying to huhabla:

Besides of that is the compression effect of the RLE in conjunction with a LZ77 compression not worth the computational effort and using RLE alone seems to create strange file sizes to me (see tests below). The existing LZ77 compression is IMHO sufficient.

The file sizes for RLE alone don't seem to make much sense. It consistently results in output files which are almost precisely 7/8ths the size of the "no compression" case, and which are entirely unaffected by the actual data.

The test d) ("sparse" values) isn't representative. A representative test would have variable-sized runs of a constant value, e.g. irregular but contiguous "blobs" of a single specific value against a single background value.

The RLE code should not be removed for backward compatibility reasons. Maybe it can be removed in GRASS8?

There's no reason to retain the code for encoding using the existing RLE algorithm (with or without LZ77). The code to read it needs to be retained for compatibility.

There might be some value in adding code to read and write RLE+LZ77 using a corrected RLE implementation, but that would require implementing a corrected RLE algorithm then performing representative tests. The effort may not be worthwhile.

comment:2 by huhabla, 11 years ago

Just for the record and to document the implementation progress:

I have created a patch that removes the LZW compression and some RLE related code so that now zip compression or no compression of tiles is supported. Compression can be switched on and off. Backward compatibility should be granted.

All module tests were performed successfully. The library module tests are working too.

Unfortunately the implementation will not work with large 3D raster maps neither for compressed nor for uncompressed maps.

This is the error i got for 1000000000 cells using no compression: 

test.g3d.lib unit=large depth=1000 rows=1000 cols=1000

Testing DCELL put function for large files
Creating 3D raster map
 100%
Verifying 3D raster map
ERROR: Rast3d_get_double_region: error in Rast3d_get_tile_ptr.Region
       coordinates x 0 y 0 z 0 tile index 0 offset 0

This is the error i got for 1000000000 cells using zip compression and a tile size of 4000Byte: 

# Using compression and different tile size
test.g3d.lib unit=large depth=1000 rows=1000 cols=1000 -l tile_size=4

Testing DCELL put function for large files filled with zeros
Creating 3D raster map filled with zeros
 100%
Verifying 3D raster map filled with zeros
ERROR: Rast3d_get_double_region: error in Rast3d_get_tile_ptr.Region
       coordinates x 976 y 960 z 88 tile index 186997 offset 0

Using valgrind to detect possible memory leaks leads to this log: 

GRASS 7.0.svn (Test XY):~/src/grass7.0/grass_trunk > valgrind --tool=memcheck test.g3d.lib unit=large depth=1000 rows=1000 cols=1000 -l tile_size=4
==13114== Memcheck, a memory error detector
==13114== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==13114== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==13114== Command: test.g3d.lib unit=large depth=1000 rows=1000 cols=1000 -l tile_size=4
==13114== 

++ Running g3d put/get value large file unit tests ++
Testing DCELL put function for large files
Creating 3D raster map
 100%
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508AE8B: Rast3d_readIndex (index.c:68)
==13114==    by 0x508B153: Rast3d_flush_index (index.c:127)
==13114==    by 0x508286E: close_cell_new (close.c:83)
==13114==    by 0x5082A8A: Rast3d_close (close.c:152)
==13114==    by 0x403AA5: test_large_file (test_put_get_value_large_file.c:104)
==13114==    by 0x403850: unit_test_put_get_value_large_file (test_put_get_value_large_file.c:38)
==13114==    by 0x402039: main (test_main.c:160)
==13114== 
Verifying 3D raster map
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508AE8B: Rast3d_readIndex (index.c:68)
==13114==    by 0x508B3B8: Rast3d_init_index (index.c:188)
==13114==    by 0x508A378: Rast3d_fill_header (header.c:446)
==13114==    by 0x508D90C: Rast3d_open_cell_old (open.c:162)
==13114==    by 0x403AD8: test_large_file (test_put_get_value_large_file.c:108)
==13114==    by 0x403850: unit_test_put_get_value_large_file (test_put_get_value_large_file.c:38)
==13114==    by 0x402039: main (test_main.c:160)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1C5: indexSortCompare (index.c:146)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1C5: indexSortCompare (index.c:146)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1C5: indexSortCompare (index.c:146)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1C5: indexSortCompare (index.c:146)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1D6: indexSortCompare (index.c:148)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1D6: indexSortCompare (index.c:148)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1D6: indexSortCompare (index.c:148)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== Conditional jump or move depends on uninitialised value(s)
==13114==    at 0x508B1D6: indexSortCompare (index.c:148)
==13114==    by 0x52D76D9: msort_with_tmp.part.0 (msort.c:66)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75D6: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114==    by 0x52D75C0: msort_with_tmp.part.0 (msort.c:46)
==13114== 
==13114== 
==13114== More than 10000000 total errors detected.  I'm not reporting any more.
==13114== Final error counts will be inaccurate.  Go fix your program!
==13114== Rerun with --error-limit=no to disable this cutoff.  Note
==13114== that errors may occur in your program without prior warning from
==13114== Valgrind, because errors are no longer being displayed.
==13114== 
 100%
Testing DCELL put function for large files filled with zeros
Creating 3D raster map filled with zeros
 100%
Verifying 3D raster map filled with zeros
ERROR: Rast3d_get_double_region: error in Rast3d_get_tile_ptr.Region
       coordinates x 976 y 960 z 88 tile index 186997 offset 0
==13114== 
==13114== HEAP SUMMARY:
==13114==     in use at exit: 50,047,490 bytes in 111 blocks
==13114==   total heap usage: 61,374,556 allocs, 61,374,445 frees, 1,210,299,148,216 bytes allocated
==13114== 
==13114== LEAK SUMMARY:
==13114==    definitely lost: 2,607 bytes in 35 blocks
==13114==    indirectly lost: 150 bytes in 4 blocks
==13114==      possibly lost: 0 bytes in 0 blocks
==13114==    still reachable: 50,044,733 bytes in 72 blocks
==13114==         suppressed: 0 bytes in 0 blocks
==13114== Rerun with --leak-check=full to see details of leaked memory
==13114== 
==13114== For counts of detected and suppressed errors, rerun with: -v
==13114== Use --track-origins=yes to see where uninitialised values come from
==13114== ERROR SUMMARY: 10000000 errors from 10 contexts (suppressed: 2 from 2)

by huhabla, 11 years ago

Attachment: rast3d.diff​ added

comment:3 by huhabla, 11 years ago

Just for the record and to document the implementation progress:

I have hopefully found the reported errors. There were several offset and file position variables with insufficient sizes (32Bit instead of 64Bit) resulting in wrong indexing and tile pointer. In addition i discovered that the tile index was compressed using the 3D raster RLE implementation and then written to the 3D raster map file. I have removed the usage of RLE since the compression is buggy for large indices. Now the tile index is written uncompressed to the map file. The reported errors should be fixed now.

The changes that i have made in the code should assure backward compatibility, so that older versions of grass (6, 7) are able to read 3D raster maps created with this version and vice verse.

The patches for the library and include files as well as for the modules are attached.

Library unit test results using zip compression: 

GRASS 7.0.svn (Test XY):~ > test.g3d.lib -u depths=1000 rows=1000 cols=1000 -l

++ Running g3d coordinate transform unit tests ++
Test the upper right corner, coordinates must be col = 9, row = 0, depth =
4
Rast3d_location2coord col 9 row 0 depth 4
Test the lower left corner, coordinates must be col = 0 row = 14 depth = 0
Rast3d_location2coord col 0 row 14 depth 0
Test the center, coordinates must be col = 4 row = 7 depth = 2
Rast3d_location2coord col 4 row 7 depth 2
Test the n=3000.1, e=7000.1 and t=800.1, coordinates must be col = 4 row =
10 depth = 4
Rast3d_location2coord col 4 row 10 depth 4
Test the n=2999.9, e=6999.9 and t=799.9, coordinates must be col = 3 row =
11 depth = 3
Rast3d_location2coord col 3 row 11 depth 3

-- g3d coordinate transform unit tests finished successfully --

++ Running g3d put/get value unit tests ++
Testing put get resample value functions
Get the value of the upper left corner -> 0
Value 0 == 0 == 0 == 0
Get the value of x == y == z == 1 -> x + y + z == 3
Value 3 == 3 == 3 == 3
Get the value of x == 7 y == 9 z == 3 -> x + y + z == 19
Value 19 == 19 == 19 == 19
Get the value of x == 9 y == 14 z == 4 -> x + y + z == 27
Value 27 == 27 == 27 == 27
Value -nan == -nan == -nan == -nan

-- g3d put/get value unit tests finished successfully --

++ Running g3d put/get value large file unit tests ++
Testing DCELL put function for large files filled with random values
Creating 3D raster map filled with random values
 100%
Verifying 3D raster map filled with random values
 100%
Testing DCELL put function for large files filled with sparse random values
Creating 3D raster map filled with sparse random values
 100%
Verifying 3D raster map filled with sparse random values
 100%
Testing DCELL put function for large files
Creating 3D raster map
 100%
Verifying 3D raster map
 100%
Testing DCELL put function for large files
Creating 3D raster map
 100%
Verifying 3D raster map
 100%
Testing DCELL put function for large files
Creating 3D raster map
 100%
Verifying 3D raster map
 100%
Testing DCELL put function for large files filled with zeros
Creating 3D raster map filled with zeros
 100%
Verifying 3D raster map filled with zeros
 100%

-- g3d put/get value large file unit tests finished successfully --

-- g3d lib tests finished successfully --

by huhabla, 11 years ago

Attachment: raster3dlib.diff​ added

Patch for 3D raster library and include files

by huhabla, 11 years ago

Attachment: raster3dmodules.diff​ added

Patch for 3D raster modules

comment:4 by huhabla, 11 years ago

In addition to my previous post, here the 3D raster library unit test for uncompressed large 3D raster maps: 

GRASS 7.0.svn (Test XY):~ > test.g3d.lib -u rows=1000 cols=1000 depths=1000

++ Running g3d coordinate transform unit tests ++
Test the upper right corner, coordinates must be col = 9, row = 0, depth =
4
Rast3d_location2coord col 9 row 0 depth 4
Test the lower left corner, coordinates must be col = 0 row = 14 depth = 0
Rast3d_location2coord col 0 row 14 depth 0
Test the center, coordinates must be col = 4 row = 7 depth = 2
Rast3d_location2coord col 4 row 7 depth 2
Test the n=3000.1, e=7000.1 and t=800.1, coordinates must be col = 4 row =
10 depth = 4
Rast3d_location2coord col 4 row 10 depth 4
Test the n=2999.9, e=6999.9 and t=799.9, coordinates must be col = 3 row =
11 depth = 3
Rast3d_location2coord col 3 row 11 depth 3

-- g3d coordinate transform unit tests finished successfully --

++ Running g3d put/get value unit tests ++
Testing put get resample value functions
Get the value of the upper left corner -> 0
Value 0 == 0 == 0 == 0
Get the value of x == y == z == 1 -> x + y + z == 3
Value 3 == 3 == 3 == 3
Get the value of x == 7 y == 9 z == 3 -> x + y + z == 19
Value 19 == 19 == 19 == 19
Get the value of x == 9 y == 14 z == 4 -> x + y + z == 27
Value 27 == 27 == 27 == 27
Value -nan == -nan == -nan == -nan

-- g3d put/get value unit tests finished successfully --

++ Running g3d put/get value large file unit tests ++
Testing DCELL put function for large files filled with random values
Creating 3D raster map filled with random values
 100%
Verifying 3D raster map filled with random values
 100%
Testing DCELL put function for large files filled with sparse random values
Creating 3D raster map filled with sparse random values
 100%
Verifying 3D raster map filled with sparse random values
 100%
Testing DCELL put function for large files
Creating 3D raster map
 100%
Verifying 3D raster map
 100%
Testing DCELL put function for large files
Creating 3D raster map
 100%
Verifying 3D raster map
 100%
Testing DCELL put function for large files
Creating 3D raster map
 100%
Verifying 3D raster map
 100%
Testing DCELL put function for large files filled with zeros
Creating 3D raster map filled with zeros
 100%
Verifying 3D raster map filled with zeros
 100%

-- g3d put/get value large file unit tests finished successfully --

-- g3d lib tests finished successfully --

comment:5 by neteler, 9 years ago

Type: task → defect

I have rerun the test: Looks almost ok except for

ERROR: Rast3d_cache_hash_remove_name: name 170656 not in hashtable 

cd grass70/lib/raster3d/test/

make

# run in NC or Spearfish or XY location:
test.raster3d.lib -u rows=1000 cols=1000 depths=1000

++ Running raster3d coordinate transform unit tests ++
Test the upper right corner, coordinates must be col = 9, row = 0, depth =
4
Rast3d_location2coord col 9 row 0 depth 4
Test the lower left corner, coordinates must be col = 0 row = 15 depth = 0
Rast3d_location2coord col 0 row 15 depth 0
Test the center, coordinates must be col = 4 row = 7 depth = 2
Rast3d_location2coord col 4 row 7 depth 2
Test the n=3000.1, e=7000.1 and t=800.1, coordinates must be col = 4 row =
10 depth = 4
Rast3d_location2coord col 4 row 10 depth 4
Test the n=2999.9, e=6999.9 and t=799.9, coordinates must be col = 3 row =
11 depth = 3
Rast3d_location2coord col 3 row 11 depth 3

-- raster3d coordinate transform unit tests finished successfully --

++ Running raster3d put/get value unit tests ++
Testing DCELL put get value functions
Get the value of the upper left corner -> 0
Value 0 == 0 == 0 == 0
Get the value of x == y == z == 1 -> x + y + z == 3
Value 3 == 3 == 3 == 3
Get the value of x == 4 y == 3 z == 2 -> x + y + z = 9
Value 9 == 9 == 9 == 9
Get the value of x == 9 y == 14 z == 4 -> x + y + z = 27
Value 27 == 27 == 27 == 27
Get the value of x == 10 y == 15 z == 5 -> x + y + z = NAN
Value -nan == -nan
Testing FCELL put get value functions
Get the value of the lower left corner -> 0
Value 0 == 0 == 0 == 0
Get the value of x == y == z == 1 -> x + y + z == 3
Value 3 == 3 == 3 == 3
Get the value of x == 4 y == 3 z == 2 -> x + y + z = 9
Value 9 == 9 == 9 == 9
Get the value of x == 9 y == 14 z == 4 -> x + y + z = 27
Value 27 == 27 == 27 == 27
Get the value of x == 10 y == 15 z == 5 -> x + y + z = NAN
Value -nan == -nan
Testing put get resample value functions
Get the value of the upper left corner -> 0
Value 0 == 0 == 0 == 0
Get the value of x == y == z == 1 -> x + y + z == 3
Value 3 == 3 == 3 == 3
Get the value of x == 7 y == 9 z == 3 -> x + y + z == 19
Value 19 == 19 == 19 == 19
Get the value of x == 9 y == 14 z == 4 -> x + y + z == 27
Value 27 == 27 == 27 == 27
Value -nan == -nan == -nan == -nan

-- raster3d put/get value unit tests finished successfully --

++ Running raster3d put/get value large file unit tests ++
Testing DCELL put function for large files
Creating 3D raster map
ERROR: Rast3d_cache_hash_remove_name: name 170656 not in hashtable

comment:6 by neteler, 8 years ago

Cc: grass-dev@… added

comment:7 by neteler, 8 years ago

Milestone: 7.0.0 → 7.0.4

Tried again with current trunk (r67424): segfault... 

test.raster3d.lib -u rows=1000 cols=1000 depths=1000
[...]
 100%
Verifying 3D raster map
 100%
WARNING: Unable to move temp raster map
         </grassdata/nc_spm_08_grass7/user1/.tmp/pgis_north/30340.8> to 3D
         raster map
         </grassdata/nc_spm_08_grass7/user1/grid3/test_put_get_value_dcell_large/cell>
WARNING: Unable to create 3D raster map <test_put_get_value_dcell_large>
WARNING: Unable to create 3D raster map <test_put_get_value_dcell_large>
Testing DCELL put function for large files filled with random values
[...]
Verifying 3D raster map filled with random values
Segmentation fault (core dumped)

comment:8 by martinl, 8 years ago

Milestone: 7.0.4 → 7.0.5

comment:9 by wenzeslaus, 8 years ago

I don't get a segfault with 7.2 r69389... but well, they can be random. I don't get any warnings from in comment:7. 

> test.raster3d.lib -u rows=1000 cols=1000 depths=1000
...
Verifying 3D raster map
 100%

-- raster3d put/get value large file unit tests finished successfully --

-- raster3d lib tests finished successfully --

comment:10 by neteler, 8 years ago

Milestone: 7.0.5 → 7.0.6

comment:11 by neteler, 6 years ago

Milestone: 7.0.6 → 7.0.7

comment:12 by martinl, 6 years ago

What is the state of this ticket?

Note: See TracTickets for help on using tickets.