Opened 12 years ago

Closed 12 years ago

#1800 closed defect (fixed)

v.kernel is very slow

Reported by: arencambre Owned by: grass-dev@…
Priority: normal Milestone: 6.4.3
Component: Vector Version: 6.4.2
Keywords: Cc: aren@…
CPU: x86-64 Platform: MSWindows 7

Description

A v.kernel run on a dataset with over 700,000 points abruptly crashes after several hours of processing: v.kernel input=geocoding_after_highway_name_fix@PERMANENT output=mockingbird_and_central_high_resolution stddeviation=100 STDDEV: 100.000000 RES: 11.176766 ROWS: 4710 COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188. closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0 to cell file C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution Maximum value in output: 2.703527e-002. (Fri Nov 16 14:26:46 2012) Command finished (59981 sec)

Change History (14)

comment:1 by arencambre, 12 years ago

Sorry, the formatting got messed up. Trying again:

v.kernel input=geocoding_after_highway_name_fix@PERMANENT output=mockingbird_and_central_high_resolution stddeviation=100
STDDEV: 100.000000
RES: 11.176766	ROWS: 4710	COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188.
closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0
to cell file C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution
Maximum value in output: 2.703527e-002.
(Fri Nov 16 14:26:46 2012) Command finished (59981 sec)

in reply to:  1 comment:2 by mmetz, 12 years ago

Replying to arencambre:

v.kernel input=geocoding_after_highway_name_fix@PERMANENT output=mockingbird_and_central_high_resolution stddeviation=100
STDDEV: 100.000000
RES: 11.176766	ROWS: 4710	COLS: 4495

Writing output raster map using smooth parameter=100.000000.

Normalising factor=64826.350188.
closecell: can't move C:\temp/dallascounty/PERMANENT/.tmp/7128.0
to cell file C:\temp/dallascounty/PERMANENT/fcell/mockingbird_and_central_high_resolution
Maximum value in output: 2.703527e-002.
(Fri Nov 16 14:26:46 2012) Command finished (59981 sec)

Is there enough free disk space? You could test with creating a dummy raster with the same region settings using e.g

r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)"

This should tell you if the problem is specific to v.kernel or more general.

BTW, v.kernel is about twice as fast in trunk r53896.

Markus M

comment:3 by arencambre, 12 years ago

I have over 100 GB disk space. I didn't detect much disk utilization during the run.

Not clear what r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)" is going to do? mockingbird_and_central_high_resolution is the bad output from the v.kernel run that crashed.

in reply to:  3 comment:4 by mmetz, 12 years ago

Replying to arencambre:

I have over 100 GB disk space. I didn't detect much disk utilization during the run.

Over 100 GB of free disk space?

Not clear what r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)" is going to do? mockingbird_and_central_high_resolution is the bad output from the v.kernel run that crashed.

This should tell you if the problem is specific to v.kernel or more general, because the crash was caused by the GRASS library handling raster IO, not by v.kernel itself. Therefore the idea is to create a raster map with the same type and name for the same region settings, but with a different module in order to find out if the crash in the GRASS library is specific to v.kernel or more general.

Markus M

comment:5 by arencambre, 12 years ago

Sorry, meant that I have over 100GB free.

Running r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)" appeared to be successful:

C:\Program Files (x86)\Quantum GIS Lisboa>r.mapcalc "mockingbird_and_central_hig
h_resolution = rand(0.0, 1.0)"
 100%

C:\Program Files (x86)\Quantum GIS Lisboa>

in reply to:  5 comment:6 by mmetz, 12 years ago

Replying to arencambre:

Sorry, meant that I have over 100GB free.

Running

r.mapcalc "mockingbird_and_central_high_resolution = rand(0.0, 1.0)"

appeared to be successful

Strange. I tried to reproduce this error with the current release branch (6.4.svn) and with trunk on a laptop with Fedora Linux 16 64bit, Intel core i5 at 2.4 GHz and 4 GB RAM. I followed this tutorial:

http://qgis.spatialthoughts.com/2012/07/tutorial-making-heatmaps-using-qgis-and.html

and used this dataset:

http://data.london.gov.uk/datafiles/crime-community-safety/police-uk-crime-data-surrey.zip

with region settings:

g.region n=318266 s=117316 e=593618 w=363772 res=100 -p

No success, i.e. v.kernel finishes successfully in a reasonable amount of time:

GRASS 6.4.svn

42 MB RAM used by v.kernel

uniform kernel
real	1m8.270s
user	0m28.876s
sys	0m37.474s

gaussian kernel
real	19m37.177s
user	9m4.927s
sys	9m59.545s


GRASS 7

19 MB RAM used by v.kernel

uniform kernel
real	1m8.286s
user	0m43.967s
sys	0m22.599s

gaussian kernel
real	5m24.772s
user	4m0.915s
sys	1m15.982s

I suggest to update to GRASS 6.4.3RC1.

Markus M

comment:7 by arencambre, 12 years ago

I just emailed you offlist with a scenario to reproduce with 6.4.2. I will install 6.4.3RC1 here and see if that makes a difference.

comment:8 by arencambre, 12 years ago

CORRECTION: I am reproducing the very, very long execution with the scenario I emailed you. I am not clear if it still crashes. Regardless, it appears that the very, very long execution may be the key problem here.

comment:9 by arencambre, 12 years ago

Even with 6.4.3RC1, it's still very, very slow. On the progress bar's 2nd pass, it's been a few minutes since it has budged.

comment:10 by arencambre, 12 years ago

On 6.4.3RC1, the status bar (% complete indicator at bottom of v.kernel window) actually hasn't budged at all since the 2nd pass started. So we either have a bug--the status bar doesn't update--or v.kernel is taking even longer to run than with 6.4.2.

comment:11 by arencambre, 12 years ago

Summary: v.kernel crashes after long executionv.kernel is very slow

Refocusing the issue on the core problem. Crashing could simply be because something else is wrong that causes it to run for too long.

comment:12 by arencambre, 12 years ago

r53983 of 6.4.3 is MUCH faster. An almost all day v.kernel run is now only 25 minutes.

in reply to:  12 comment:13 by neteler, 12 years ago

Replying to arencambre:

r53983 of 6.4.3 is MUCH faster. An almost all day v.kernel run is now only 25 minutes.

Can we consider this as fixed?

comment:14 by arencambre, 12 years ago

Resolution: fixed
Status: newclosed

Yes.

Note: See TracTickets for help on using tickets.