break in importing and cleaning very large vector datasets

[gisboa]

I try to use Grass with large vector files, with sizes up to 0,5GB, this usually results in errors during import, or if not during import, then at least while cleaning them. Both on Mac (Leopard intel) and Linux (openSuse 11 (32 and 64bit). It doesn't seem to be a problem with the data, as I can process all data when splitting the import, and handle the various slices separately.

Files are too large to attach...

Wouter

Mac OSX ====== GRASS 6.4.0RC3 (nl-rdn):/data/grassdb/nl-rdn > v.clean --o input=top_top10vec_gebouwen output=top_top10vec_geb2 tool=bpol

---

Tool: Threshold Break polygons: 0.000000e+00

---

Copying vector lines... Rebuilding parts of topology... Building topology for vector map <top_top10vec_geb2>... Registering primitives... 6041168 primitives registered 25000849 vertices registered Number of nodes: 6037532 Number of primitives: 6041168 Number of points: 0 Number of lines: 0 Number of boundaries: 3021055 Number of centroids: 3020113 Number of areas: - Number of isles: -

---

Tool: Break polygons v.clean(22861) malloc: *** mmap(size=275042304) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug ERROR: G_realloc: unable to allocate 275040036 bytes at

break_polygons.c:188

Linux 64bit ======= Rebuilding parts of topology... Building topology for vector map <top_top10vec_geb2>... Registering primitives... 6041168 primitives registered 25000849 vertices registered Number of nodes: 6037532 Number of primitives: 6041168 Number of points: 0 Number of lines: 0 Number of boundaries: 3021055 Number of centroids: 3020113 Number of areas: - Number of isles: -

---

Tool: Break polygons ERROR: G_realloc: unable to allocate 34400040 bytes at break_polygons.c:188

[mmetz]

Replying to gisboa:

I try to use Grass with large vector files, with sizes up to 0,5GB, this usually results in errors during import, or if not during import, then at least while cleaning them.

...

Tool: Break polygons ERROR: G_realloc: unable to allocate 34400040 bytes at break_polygons.c:188

Cleaning vector files of this size (about 0.5GB) in v.in.ogr or v.clean can require a lot of memory, up to 8GB. Is it possible that you ran out of memory? Physical memory plus swap space smaller than 8GB? If yes, increase physical memory and/or swap space, and try develbranch_6 instead of 6.4.0RC3.

[marisn]

As there has been no feedback in 16 months, closing this ticket, as insufficent RAM is most likely issue.