Memory limit issues with r.univar (when used within i.segment.stats)

[jpduffy]

Hello,

I am trying to use i.segment.stats as part of an OBIA workflow on high resolution raster orthomosaics. So far I have used i.segment.uspo, i.segment and r.clump to create a layer called gp_seg_optimum_clump. Using r.clump means that the clumps are sequentially numbered, whereas they are not directly after i.segment.

I try to run the following command:

i.segment.stats --overwrite --verbose map=gp_seg_optimum_clump@gp1 \
rasters=gp_ortho.1@gp1,gp_ortho.2@gp1,gp_ortho.3@gp1,gp_ortho.4@gp1 \
raster_statistics=min,max,mean,stddev,variance,sum \
csvfile=/home/jpd205/Wales_GRASS/GarronPill/gp_seg_stats \
separator=comma

And I get the following error:

Calculating geometry statistics
Calculating statistics for raster gp_ortho.1@gp1
ERROR: G_realloc: unable to allocate 572000 bytes of memory at
       raster/r.univar/r.univar_main.c:324
Traceback (most recent call last):
  File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 251, in <module>
    main()
  File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 176, in main
    quiet=True)
  File "/usr/lib/grass70/etc/python/grass/script/core.py", line 395, in run_command
    return handle_errors(returncode, returncode, args, kwargs)
  File "/usr/lib/grass70/etc/python/grass/script/core.py", line 313, in handle_errors
    returncode=returncode)
grass.exceptions.CalledModuleError: Module run None ['r.univar', '--o', '--q', '-et', 'zones=gp_seg_optimum_clump@gp1', 'output=/home/jpd205/Wales_GRASS/GarronPill/gp1/.tmp/osgeobeefy/6462.0', 'map=gp_ortho.1@gp1'] ended with error
Process ended with non-zero return code 1. See errors in the (error) output.

Some details about my region settings and data:

g.region -p
projection: 99 (OSGB 1936 / British National Grid)
zone:       0
datum:      osgb36
ellipsoid:  airy
north:      208007.00931776
south:      207952.59780698
west:       200993.90853302
east:       201097.28911076
nsres:      0.00430914
ewres:      0.00430914
rows:       12627
cols:       23991
cells:      302934357

r.info map=gp_seg_optimum_clump
 +----------------------------------------------------------------------------+
 | Map:      gp_seg_optimum_clump           Date: Thu Oct 27 14:28:30 2016    |
 | Mapset:   gp1                            Login of Creator: jpd205          |
 | Location: GarronPill                                                       |
 | DataBase: /home/jpd205/Wales_GRASS                                         |
 | Title:    gp_seg_optimum_clump ( gp_seg_optimum_clump )                    |
 | Timestamp: none                                                            |
 |----------------------------------------------------------------------------|
 |                                                                            |
 |   Type of Map:  raster               Number of Categories: 0               |
 |   Data Type:    CELL                                                       |
 |   Rows:         12627                                                      |
 |   Columns:      23991                                                      |
 |   Total Cells:  302934357                                                  |
 |        Projection: OSGB 1936 / British National Grid                       |
 |            N: 208007.00931776    S: 207952.59780698   Res: 0.00430914      |
 |            E: 201097.28911076    W: 200993.90853302   Res: 0.00430914      |
 |   Range of data:    min = 1  max = 1264957                                 |
 |                                                                            |
 |   Data Source:                                                             |
 |    gp_seg_optimum@gp1                                                      |
 |                                                                            |
 |                                                                            |
 |   Data Description:                                                        |
 |    generated by r.clump                                                    |
 |                                                                            |
 |   Comments:                                                                |
 |    r.clump --overwrite --verbose -d input="gp_seg_optimum@gp1" output="\   |
 |    gp_seg_optimum_clump" title="gp_seg_optimum_clump"                      |
 |                                                                            |
 +----------------------------------------------------------------------------+

r.info map=gp_ortho.1
 +----------------------------------------------------------------------------+
 | Map:      gp_ortho.1                     Date: Mon Oct 17 09:06:32 2016    |
 | Mapset:   gp1                            Login of Creator: jd              |
 | Location: GarronPill                                                       |
 | DataBase: /home/jpd205/Wales_GRASS                                         |
 | Title:     ( gp_ortho.1 )                                                  |
 | Timestamp: none                                                            |
 |----------------------------------------------------------------------------|
 |                                                                            |
 |   Type of Map:  raster               Number of Categories: 0               |
 |   Data Type:    CELL                                                       |
 |   Rows:         12627                                                      |
 |   Columns:      23991                                                      |
 |   Total Cells:  302934357                                                  |
 |        Projection: OSGB 1936 / British National Grid                       |
 |            N: 208007.00931776    S: 207952.59780698   Res: 0.00430914      |
 |            E: 201097.28911076    W: 200993.90853302   Res: 0.00430914      |
 |   Range of data:    min = 0  max = 255                                     |
 |                                                                            |
 |   Data Description:                                                        |
 |    generated by r.in.gdal                                                  |
 |                                                                            |
 |   Comments:                                                                |
 |    r.in.gdal -o input="/home/jd/GarronPill_R_BNG_PS2_O_C_RGB_TEX.tif" o\   |
 |    utput="gp_ortho" memory=300 offset=0                                    |
 |                                                                            |
 +----------------------------------------------------------------------------+

Moritz Lennert believes that this is a memory issue limit with r.univar. He suggested I submit this bug report.

Kind regards

James

[zarch]

Replying to jpduffy:

I get the following error:

Calculating geometry statistics
Calculating statistics for raster gp_ortho.1@gp1
ERROR: G_realloc: unable to allocate 572000 bytes of memory at
[...]

I had the same problem when I was working OBIA some years ago and I wrote r.univar2 that it is available as grass-addons. r.univa2 extract some general statistics from raster using the zones (consume much less memory than r.univar, and compute more general statistics like:skewness, kurtosis, and mode. It is available here: ​https://trac.osgeo.org/grass/browser/grass-addons/grass7/raster/r.univar2

It was not backported to trunk because it was used only by me and not tested enough (imho), and If I remember correctly is working only with zones.

[martinl]

Still relevant?

[mmetz]

Replying to martinl:

Still relevant?

Maybe there is a way to reduce memory consumption per zone, but still, memory consumption increases with the number of zones. This will probably not fix.

The addon r.univar2 needs to be reviewed and if regarded as an improvement, modified to work also without zones.