GML -> SHP Conversion does not return

Where can I upload my testcase? I have a ZIP with 19MB ...

Alex,

Please, use my FTP server:

Hostname: ftp.loskot.net Username: gdal@loskot.net Password: gdal2007

One additional note about browsing the FTP using web browser. Here is the URL to use:

ftp://gdal@loskot.net@ftp.loskot.net

and you will be asked for password.

I've uploaded a zip file ogr2ogr.zip. There are 2 directories packed: 10000 and 20000. Converting the gml in 10000 (with 10000 geometries) costs 2 minutes, converting the one in 20000 (with 20000) costs 9 minutes.

I cannot reproduce the problem with 10000 or 20000, neither with GDAL 1.4.1 nor with GDAL 1.5dev. Translation to shape of 20000 takes around 10 seconds on my PC (AMD Athlon(tm) 64 Processor 3200+ / 512 MB RAM)

Have you tried it on windows or on linux? My PC should be fast enough: AMD Athlon 64 X2 Dual Core 4200+ with 2G RAM running Windows XP (32bit) Prof. SP 2. I've installed FWTools 2.0.0 but as I've seen, there is the same gdal version (1.5dev) included as in 1.4.1 so the runtime is the same. Do you have any idea, what I should test to get the problem fixed?

Matuesz,

Could you try and reproduce this with a normal windows build, and if that doesn't show the problem, then with fwtools? If the problem only occurs with fwtools, then bounce it back to me. It may be that I need a newer xerces or something.

Just for records, I tested both datasets, 10000 and 20000, on Mac OS X 10.4 with success. It takes 4-6 secs to translate the GML files to ESRI Shapefile with success (all features are included in output shapefile).

I confirm the problem exists on Windows. It takes about 15 minutes to translate GML from 20000 to Shapefile (24MB output file) on Virtual Machine with 512 MB RAM hosted on Mac Pro with 2x2.66 GHz Intel Xeon, 5 GB RAM.

There are two problems being a reason of the bad performance:

1. The first issue is inefficient OGR datasource format detection in OGRSFDriverRegistrar::Open()
2. The second issue is a big difference in performance of CPLReadLine() function between Unix and Windows

• Ad. 1.

The function iterates through available drivers and performs try-to-open action. The problem is that the test for AVC driver takes ~10 minutes. The AVC driver does not filter out passed datasource name by file extension (i.e to automatically fail for potentially big but unsupported files .gml, .xml, .kml, etc.), nor it filters by testing first bits of file content (ie. to check if there is XML prolog).

However, this issue is nothing strange and should be possible to fix easily.

• Ad. 2.

This problem is directly related to the 1st one. The AVC driver, in private function _AVCE00ReadScanE00() iterates through lines of a file with subsequent calls of CPLReadLine(). The test file 20000/thuer_l_f.gml contains only 10 lines but length of the last line is 44628396 characters (see attached thuer_l_f-longest-line.png).

The CPLReadLine() call for this long line behaves very differently on Windows than on Unix. On Unix (Mac OS X) complete read of 44628396 characters takes ~3 seconds, but on Windows it takes ~900 seconds (see gdal-avc-cplreadline-timing.png), regardless if GDAL build in optimized or debug mode.

Fast judgement tells me that there are two bottlenecks: big number of memory allocations and I/O operations.

After the AVC finishes all test actions, translation of the test GML file 20000/thuer_l_f.gml takes a few seconds.

As a simple solution, I'd propose to add simple datasource type filtering by file extension and header content as pointed in 1. Also, if time permits it's a good idea to debug and profile CPLReadLine() for huge lines.

Looking forward your opinion guys.

Longest line (10th) of 20000/thuer_l_f.gml counted in Vim

Simple timing of CPLReadLine() calls from _AVCE00ReadScanE00() under Windows XP

The problem with poor detection in the AVC driver was already recently reported as #1989.

The problem with CPLReadLine() being slow on windows may be related to the logic at:

http://trac.osgeo.org/gdal/browser/trunk/gdal/port/cpl_conv.cpp#L365

Please create another bug report for the CPLReadLine() slowness on windows, and see if you can come up with suggestions without breaking the CPLReadLine() support for text mode files. At that point I think we can close this report and pursue the two issues in their distinct tickets.

Alex - thanks for the report leading us to these problems!

I reported new ticket about CPLReadLine issue:

#1999 - Slowness of CPLReadLine() function on Windows

I'm closing this ticket as a duplicate of tickets #1989 and #1999.