Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#2574 closed defect (fixed)

v.surf.icw - not working with the last version of grass7

Reported by: bhlevca Owned by: grass-dev@…
Priority: normal Milestone: 7.2.0
Component: Addons Version: svn-trunk
Keywords: v.surf.icw Cc:
CPU: Unspecified Platform: Unspecified

Description (last modified by neteler)

I updated my grass7 not thinking about compatibility issues. Several scripts used by v.surf.icw have changed among which g.remove

For example line 322

grass.run_command('g.remove', flags = 'f', rast = cost_site_name, quiet = True)

must be changed to.

 grass.run_command('g.remove', type='raster', flags = 'f', name = cost_site_name, quiet = True)

Now the module works partially but I struggled to find the final solution and I was wondering if someone who knows the code better can fix it.

I get the following errors after the successful creation of the : temporary files (tmp_icw*):

Creating partial weights ...
Site 1 of 51,  cat = 1, data value = 6.281
Site 2 of 51,  cat = 2, data value = 6.378843
Site 3 of 51,  cat = 3, data value = 6.324808
ERROR: Error reading raster data for row 281 of <MASK>
ERROR: Error reading raster data for row 2343 of <MASK>
ERROR: Error reading raster data for row 2109 of <MASK>

and at the end probably due to the former errors:

Calculating final values ...
Reading raster map <tmp_icw_25595_partial.00001> using weight 1.000000...
ERROR: Raster map <tmp_icw_25595_partial.00001> not found
ERROR: Problem running r.series
Cleanup..

This worked fine in grass 7 several months ago.

Attachments (2)

v.surf.icw (17.2 KB) - added by bhlevca 4 years ago.
modified v.surf.icw that works with grass7 trunk
v_surf_icw_rseries.diff (1.0 KB) - added by mlennert 3 years ago.

Download all attachments as: .zip

Change History (15)

comment:1 Changed 4 years ago by bhlevca

Sorry I meant v.surf.icw.

I tried to modify the subject but I could not figure out how.

comment:2 Changed 4 years ago by bhlevca

I found the problem I had a dangling raster map named MASK that was screwing up.

I deleted that and now it works properly.

I am attaching my version and hopefully someone with access will update svn

Thanks

comment:3 Changed 4 years ago by annakrat

I just fixed the g.remove calls in r64399.

Changed 4 years ago by bhlevca

Attachment: v.surf.icw added

modified v.surf.icw that works with grass7 trunk

comment:4 Changed 4 years ago by bhlevca

I have a large number of points (~20000) and the script runs fine until it reaches the point of summation of cost weights.

Then Python gives up because there are too many items in the arguments list. I am wondering if the script could be changed to work around Python's limitation

Here is the error:

Removing anomalies at site positions ... Applying radial decay ... Finding sum of squares ... Summation of cost weights ... ... omitted lines ...

File "/usr/lib64/python2.7/subprocess.py", line 1327, in

_execute_child

raise child_exception

OSError: [Errno 7] Argument list too long

comment:5 Changed 4 years ago by neteler

Component: DefaultAddons
Description: modified (diff)
Keywords: v.surf.icw added
Summary: r.surf.icw - not working with the last version of grass7v.surf.icw - not working with the last version of grass7

comment:6 in reply to:  4 Changed 4 years ago by lucadelu

Replying to bhlevca:

I have a large number of points (~20000) and the script runs fine until it reaches the point of summation of cost weights.

Then Python gives up because there are too many items in the arguments list. I am wondering if the script could be changed to work around Python's limitation

Here is the error:

Removing anomalies at site positions ... Applying radial decay ... Finding sum of squares ... Summation of cost weights ... ... omitted lines ...

File "/usr/lib64/python2.7/subprocess.py", line 1327, in

_execute_child

raise child_exception

OSError: [Errno 7] Argument list too long

Could you please send us a command line to test it?

Possibly with north carolina dataset.

Thanks Luca

comment:7 Changed 4 years ago by bhlevca

Sorry I don't have the North Carolina dataset but I can explain the command line I used below.

 v.surf.icw input=TorontoHarbourBathymetryWaterSimplifyedPoints column=Contour output=TorontoHarbourBathyInterpRast cost_map=TorontoHarbourWaterMaskCorrRast friction=3.5 --verbose workers=3
TorontoHarbourBathymetryWaterSimplifyedPoints has ~24000 points

The values attribute is Contour
The mask and the costmap is TorontoHarbourWaterMaskCorrRast just a plain polygons converted into a raster.

The problem is that when putting all the temporary maps together on the Python subprocess call it exceeds the maximum allowable size handled by Python or by the OS and I was wondering if there is another way of sequentially doing this.

see http://www.in-ulm.de/~mascheck/various/argmax/

comment:8 Changed 3 years ago by neteler

Milestone: 7.1.07.2.0

Milestone renamed

comment:9 Changed 3 years ago by mlennert

AFAIU, this comes from the r.series calls which use the input parameter with a list of maps. The solution is to use the file parameter instead, i.e. to write all map names to a file and to use that as input.

I'm attaching an untested diff which does this for the first r.series call. It might be necessary to the same for other r.series calls.

Please try with this patch to see if this is the right direction towards a solution.

Changed 3 years ago by mlennert

Attachment: v_surf_icw_rseries.diff added

comment:10 Changed 3 years ago by mlennert

Could someone else test the proposed patch ?

comment:11 Changed 3 years ago by annakrat

In 69618:

v.surf.icw: write temporary maps into file instead of passing through command line (by mlennert), see #2574

comment:12 Changed 3 years ago by annakrat

Resolution: fixed
Status: newclosed

comment:13 in reply to:  4 Changed 3 years ago by hamish

Replying to bhlevca:

I have a large number of points (~20000) and the script runs fine until it reaches the point of summation of cost weights.

Hi,

I'm just curious how this job went? The module was originally written for labor intensive water sample data (CTD) with maybe a couple hundred vector points max. Running r.cost 20000 times can be quite computationally intensive!

My usual approach for making a bathy surface interpolation -after r.in.xyz gridding is to use the r.surf.nnbathy module which makes a simple first pass mask (very similar to r.hull) then also run r.buffer on the input points at an appropriate mean distance between points *3 or so and use that as a better mask. v.to.rast area conversion of the coastline helps clip it too.

happy 2017, Hamish

Note: See TracTickets for help on using tickets.