[PATCH] gdal2tiles.py: parallelize base tile generation

[stefantalpalaru]

Attached is a patch to make gdal2tiles.py use all the available cores when generating the base tiles.

[stefantalpalaru]

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

[dave]

Replying to stefantalpalaru:

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

Hi, The first patch works well, but I encountered a problem with the second patch. It generated the base layer ok but then crashed immediately before beginning the first overview layer.

Here's the output:

...[1800000 lines removed]...
Generating Overview Tiles:
Traceback (most recent call last):
 File "./gdal2tiles.py", line 2283, in <module>
   for tz in range(gdal2tiles.tmaxz-1, gdal2tiles.tminz-1, -1):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' 

[stefantalpalaru]

Replying to dave:

What's the command line? Looks like you're missing the zoom levels so tmaxz and tminz default to None.

[dave]

Replying to stefantalpalaru:

Thats it! If I specify zoom levels on the command line ( --zoom 1-11 ) everything works as expected. I was not specifying the zoom levels - I wanted all levels and gdal-tiles.py normally defaults to that. It appears that this patch requires --zoom to be specified, which is fine but perhaps should be documented.

I am very impressed with with speedup obtained form this. Thanks!

[toolboy]

I'm trying to get this running on Windows 7 Ultimate. This update looks like it's for UNIX\LINIX. Is there any chance this can get ported to Windows?

Thanks.

[stefantalpalaru]

Replying to toolboy:

The multiprocessing module should run fine on Windows. What errors are you getting?

[toolboy]

Stefan, Thanks you for the quick response. The first thing I did was rename gdal2tiles_parallelize_base_and_overview_tiles.patch to gdal2tiles.py. Then I ran the following in OSGeo4W:

J:\>gdal2tiles Nord.tif

File "C:\OSGeo4W\bin\gdal2tiles.py", line 1

--- /usr/bin/gdal2tiles.py 2011-11-24 16:20:02.340068830 +0100

SyntaxError: invalid syntax

I'm sure I've done something stupid.

Thanks kyle

[stefantalpalaru]

You're supposed to apply the patch like this (in a directory with the original gdal2tiles.py): patch -p0 -i gdal2tiles_parallelize_base_and_overview_tiles.patch

Here's the patched script, in case you don't have access to the 'patch' program: ​http://pastebin.com/JrRLcyjA

[toolboy]

Ah, I was doing something stupid. That would explain that. Thanks.

I have FWTools 2.4.7, OSGeo4W with GDAL 1.7.3, and Python 2.7.2. The non-patched gdal2tiles was working fine with this setup.

When I run the patched version through the OSGeo shell I get this:

J:\>gdal2tiles.py -zoom 1-11 Nord.tif Traceback (most recent call last):

File "C:\OSGeo4W\bin\gdal2tiles.py", line 44, in <module>

import gdal

File "C:\OSGeo4W\apps\Python27\lib\site-packages\gdal.py", line 2, in <module>

from osgeo.gdal import deprecation_warn

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 21,

in <module>

_gdal = swig_import_helper()

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 17,

in swig_import_helper

_mod = imp.load_module('_gdal', fp, pathname, description)

ImportError: DLL load failed: %1 is not a valid Win32 application.

It seems to not like my GDAL version. Is there a version you can recommend?

Thanks again, Kyle

[stefantalpalaru]

I'm using the latest stable release: gdal-1.9.0

[myp]

Hi, is there currently any reason why this patch was not applied to the current gdal version ? It seems to run pretty well. It speeds up the processing of my files significantly.

[michalis]

Tried using this one on Windows 7.

Getting a Windows Error 32: cannot access the temp file because it is used by another process.

Made sure there are no leftover temp files, rerun, getting the same error. Any ideas?

[Jukka Rahkonen]

I made a quick test on Windows 7. A modified script does not feel like production ready without some further work.

c:\temp\gdal2tiles_2.py --s_srs epsg:3067 c:\temp\poista\tile\uk3l.png
Process Process-1:
Traceback (most recent call last):
  File "C:\Program Files\QGIS Dufour\apps\Python27\lib\multiprocessing\process.p
y", line 258, in _bootstrap
    self.run()
  File "C:\Program Files\QGIS Dufour\apps\Python27\lib\multiprocessing\process.p
y", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "C:\temp\gdal2tiles_2.py", line 2243, in worker_metadata
    gdal2tiles.open_input()
  File "C:\temp\gdal2tiles_2.py", line 878, in open_input
    os.unlink(tempfilename)
WindowsError: [Error 32] Process can't file because it is used by another process 'c:\\users\\rahkonju\\appdata\\local\\temp\\tmpjsdkl7-gdal2tiles.vrt'
Generating Base Tiles:
Generating Overview Tiles:
Traceback (most recent call last):
  File "C:\temp\gdal2tiles_2.py", line 2283, in <module>
    for tz in range(gdal2tiles.tmaxz-1, gdal2tiles.tminz-1, -1):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

[bnorman]

updated patch for GDAL 1.11.1, removes requirement to specify zoom level

[bnorman]

I have added an updated version of the patch that works against GDAL 1.11.1 . It also removes the requirement for manually setting the zoom levels on the command line (so the behavior is the same as the unmodified script).

It would be great if this was mainlined.

[knutole]

@bnorman, I'm not getting valid tiles with this script.

With unmodified gdal2tiles.py, the common output is: Generating Base Tiles: 0...10...20...30...40...50...60...70...80...90...100 - done. Generating Overview Tiles: 0...10...20...30...40...50...60...70...80...90...100 - done.

With the parallel version you committed 10 days ago, the output is this:

Generating Base Tiles: 0...10...20...Generating Overview Tiles:

...and the tiles are not created. Run with simple flags: gdal2tiles.py -w none -p mercator --no-kml [...]

Any idea why? Thanks!

[knutole]

I'm getting tiles like these.

[knutole]

Running with flag --processes=1, everything works well... Hit me back on @kosjoli on twitter if you need more debug. Thanks.

[pvanbosgeo]

Same problem here. Only when using the flag --process=1 (setting it to old behaviour I guess) it works

[knutole]

Hey, pvanbosgeo. I finally found a version that seems to work alright. Have a try with this one: ​https://github.com/systemapic/geopackage-python/blob/master/Tiling/gdal2tiles_parallel.py

Note that TMS = true.

[pvanbosgeo]

Thanks Knutole, does this also create kml superoverlays?

[sfkeller]

A parallel version of 'gdal2tiles_parallel.py' would be really useful. Could even keep the name gdal2tiles.py plus some options.

AFAIK there are no bugs to resolve anymore. Otherwise there seem to exist working forks in the wild wich work, like the one mentioned above or e.g. ​https://github.com/GitHubRGI/geopackage-python/wiki/Usage-Instructions-for-gdal2tiles_parallel.py

=> What's missing to accept this patch and to close this ticket?

[Even Rouault]

What's missing to accept this patch and to close this ticket?

The patch in this ticket you mean ? Well, make sure it works correctly (not obvious from the comments in this ticket), applies on top of current version (that has received recent changes by Gregory, CC'ed), new tests for the regression test suite that demonstrate it works currently and will continue to work in the future, and a volunteer to make that happen. I'd like someone to take over maintaince of gdal2tiles in the long term Note that there's also ongoing work by Gregory in ​https://github.com/OSGeo/gdal/pull/201 in this area. I've not looked at the various merits of the different attempts in that area

[sfkeller]

Even Rouault wrote:

Note that there's also ongoing work by Gregory in ​​https://github.com/OSGeo/gdal/pull/201 in this area.

That's cool. Good you're referencing this. I have not been aware of this before.

[gbataille]

Hey, I'm working on this. But I have chosen to rewrite it a lot, to make it more modular, testable, ... and to do the parallelization in a clean way (because gdal2tiles_parallel.py is really a hack)

So it's taking a bit of time (and I have spent less than I intended). I'm hopeful to have a brand new gdal2tiles.py with a lot of unit test and parallel processing in 1-2 months

[Even Rouault]

In 40194:

gdal2tiles: add --processes=intval option to parallelize processing (patch by Gregory Bataille, ​https://github.com/OSGeo/gdal/pull/201, fixes #4379)