v.kernel is very slow

[arencambre]

A v.kernel run on a dataset with over 700,000 points abruptly crashes after several hours of processing: v.kernel input=geocoding_after_highway_name_fix@PERMANENT output=mockingbird_and_central_high_resolution stddeviation=100 STDDEV: 100.000000 RES: 11.176766 ROWS: 4710 COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188. closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0 to cell file C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution Maximum value in output: 2.703527e-002. (Fri Nov 16 14:26:46 2012) Command finished (59981 sec)

[arencambre]

Sorry, the formatting got messed up. Trying again:

v.kernel input=geocoding_after_highway_name_fix@PERMANENT output=mockingbird_and_central_high_resolution stddeviation=100
STDDEV: 100.000000
RES: 11.176766	ROWS: 4710	COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188.
closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0
to cell file C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution
Maximum value in output: 2.703527e-002.
(Fri Nov 16 14:26:46 2012) Command finished (59981 sec)

[mmetz]

Replying to arencambre:

v.kernel input=geocoding_after_highway_name_fix@PERMANENT output=mockingbird_and_central_high_resolution stddeviation=100
STDDEV: 100.000000
RES: 11.176766	ROWS: 4710	COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188.
closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0
to cell file C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution
Maximum value in output: 2.703527e-002.
(Fri Nov 16 14:26:46 2012) Command finished (59981 sec)

Is there enough free disk space? You could test with creating a dummy raster with the same region settings using e.g

r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)"

This should tell you if the problem is specific to v.kernel or more general.

BTW, v.kernel is about twice as fast in trunk r53896.

Markus M

[arencambre]

I have over 100 GB disk space. I didn't detect much disk utilization during the run.

Not clear what r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)" is going to do? mockingbird_and_central_high_resolution is the bad output from the v.kernel run that crashed.

[mmetz]

Replying to arencambre:

I have over 100 GB disk space. I didn't detect much disk utilization during the run.

Over 100 GB of free disk space?

Not clear what r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)" is going to do? mockingbird_and_central_high_resolution is the bad output from the v.kernel run that crashed.

This should tell you if the problem is specific to v.kernel or more general, because the crash was caused by the GRASS library handling raster IO, not by v.kernel itself. Therefore the idea is to create a raster map with the same type and name for the same region settings, but with a different module in order to find out if the crash in the GRASS library is specific to v.kernel or more general.

Markus M

[arencambre]

Sorry, meant that I have over 100GB free.

Running r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)" appeared to be successful:

C:\Program Files (x86)\Quantum GIS Lisboa>r.mapcalc "mockingbird_and_central_hig
h_resolution = rand(0.0, 1.0)"
 100%

C:\Program Files (x86)\Quantum GIS Lisboa>

[mmetz]

Replying to arencambre:

Sorry, meant that I have over 100GB free.

Running

r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)"

appeared to be successful

Strange. I tried to reproduce this error with the current release branch (6.4.svn) and with trunk on a laptop with Fedora Linux 16 64bit, Intel core i5 at 2.4 GHz and 4 GB RAM. I followed this tutorial:

​http://qgis.spatialthoughts.com/2012/07/tutorial-making-heatmaps-using-qgis-and.html

and used this dataset:

​http://data.london.gov.uk/datafiles/crime-community-safety/police-uk-crime-data-surrey.zip

with region settings:

g.region n=318266 s=117316 e=593618 w=363772 res=100 -p

No success, i.e. v.kernel finishes successfully in a reasonable amount of time:

GRASS 6.4.svn

42 MB RAM used by v.kernel

uniform kernel
real	1m8.270s
user	0m28.876s
sys	0m37.474s

gaussian kernel
real	19m37.177s
user	9m4.927s
sys	9m59.545s


GRASS 7

19 MB RAM used by v.kernel

uniform kernel
real	1m8.286s
user	0m43.967s
sys	0m22.599s

gaussian kernel
real	5m24.772s
user	4m0.915s
sys	1m15.982s

I suggest to update to GRASS 6.4.3RC1.

Markus M

[arencambre]

I just emailed you offlist with a scenario to reproduce with 6.4.2. I will install 6.4.3RC1 here and see if that makes a difference.

[arencambre]

CORRECTION: I am reproducing the very, very long execution with the scenario I emailed you. I am not clear if it still crashes. Regardless, it appears that the very, very long execution may be the key problem here.

[arencambre]

Even with 6.4.3RC1, it's still very, very slow. On the progress bar's 2nd pass, it's been a few minutes since it has budged.

[arencambre]

On 6.4.3RC1, the status bar (% complete indicator at bottom of v.kernel window) actually hasn't budged at all since the 2nd pass started. So we either have a bug--the status bar doesn't update--or v.kernel is taking even longer to run than with 6.4.2.

[arencambre]

Refocusing the issue on the core problem. Crashing could simply be because something else is wrong that causes it to run for too long.

[arencambre]

r53983 of 6.4.3 is MUCH faster. An almost all day v.kernel run is now only 25 minutes.

[neteler]

Replying to arencambre:

r53983 of 6.4.3 is MUCH faster. An almost all day v.kernel run is now only 25 minutes.

Can we consider this as fixed?

[arencambre]

Yes.