Join attributes by location crash

[bderstine]

Target layer has 2.3 million points. Join layer has a couple hundred polygons.

Process ran very slowly up to 27% completeness and then QGIS stopped responding and crashed.

[lutra]

please attach/link a sample dataset to try replicate the issue.

[cfarmer]

A point of clarification:

Process ran very slowly up to 27% completeness and then QGIS stopped responding and crashed.

Does this mean that QGIS stopped responding and you closed/terminated the application, or did it indeed crash on it's own? If crash, is there any output that you can report?

Note: The Join by location tool is not particularly efficient (not optimized for large datasets), and for large datasets, you might be better off using postgis, or even a spatialite database, which are more appropriate for these types of operations. This is obviously not a 'solution' to this problem, but rather a viable workaround. I would also recommend using the 'Check geometry validity' tool to ensure that your polygons are all properly defined and not causing problems themselves.

Carson

[bderstine]

lutra: I cannot attach the dataset as it is proprietary.

cfarmer: This means that QGIS stopped responding, the window turned slightly transparent, and I eventually shut it down. I may have been able to let it keep running, but it had already been an hour with no apparent signs of life (the progress bar ceased updating). No output to report, sadly.

I ran the same join (and a follow-up) in ArcGIS, just to test whether the files were ok, and it ran fine there (took a long time as well, but not as long and no crash).

[bderstine]

lutra: I imagine a similar query would be to load the GNIS national file: ​http://geonames.usgs.gov/domestic/download_data.htm and join attributes by spatial location to some US Census boundary file such as the Census Incorporated Places.

[lutra]

Replying to bderstine:

lutra: I imagine a similar query would be to load

Hi, can you please test if the datasets you say are enough to replicate the problem? I'm also not familiar with that datasets, so I guess it would be easier for you to try.

[bderstine]

More detail: I tried again today with 1.8.0-10. Attribute Summary set to "Take attributes of fist located feature" Output table: Keep all records (including non-matching target records)

Process started at 8:35am, progress bar jumped to 15% immediately and then doesn't move, and I get the little "thinking" mouse pointer (likely due to all the processing that's happening behind the scenes, though this is not ideal UI design).

A spatial join in ArcGIS constantly displays a running count of # processed, in chunks, so you know that it's still actually doing something. Something like that would vastly improve the UI here.

At 9:14am, I get the pop-up window telling me that it "created output shapefile: ..." So I guess I was wrong. I guess I'll change this to "enhancement" and "major". I feel strongly that either (a) the progress bar needs to be fixed so that it actually does update to show progress, or (b) some other indicator of progress should be added to inform the user that the process is still working.

[lutra]

I suggest then to close this ticket and open a more general one, like "improve handling of processes in ftools". My suggestion would be to manage the processes as in the "raster" tools menu (gdal tools). In this case launching a process does not "block" qgis as is run in the background, as a user can still work with the main program.