gdal2tiles.py: parallelize base tile generation

[stefantalpalaru]

Attached is a patch to make gdal2tiles.py use all the available cores when generating the base tiles.

[stefantalpalaru]

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

[dave]

Replying to stefantalpalaru:

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

Hi, The first patch works well, but I encountered a problem with the second patch. It generated the base layer ok but then crashed immediately before beginning the first overview layer.

Here's the output:

...[1800000 lines removed]...
Generating Overview Tiles:
Traceback (most recent call last):
 File "./gdal2tiles.py", line 2283, in <module>
   for tz in range(gdal2tiles.tmaxz-1, gdal2tiles.tminz-1, -1):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' 

[stefantalpalaru]

Replying to dave:

What's the command line? Looks like you're missing the zoom levels so tmaxz and tminz default to None.

[dave]

Replying to stefantalpalaru:

Thats it! If I specify zoom levels on the command line ( --zoom 1-11 ) everything works as expected. I was not specifying the zoom levels - I wanted all levels and gdal-tiles.py normally defaults to that. It appears that this patch requires --zoom to be specified, which is fine but perhaps should be documented.

I am very impressed with with speedup obtained form this. Thanks!

[toolboy]

I'm trying to get this running on Windows 7 Ultimate. This update looks like it's for UNIX\LINIX. Is there any chance this can get ported to Windows?

Thanks.

[stefantalpalaru]

Replying to toolboy:

The multiprocessing module should run fine on Windows. What errors are you getting?

[toolboy]

Stefan, Thanks you for the quick response. The first thing I did was rename gdal2tiles_parallelize_base_and_overview_tiles.patch to gdal2tiles.py. Then I ran the following in OSGeo4W:

J:\>gdal2tiles Nord.tif

File "C:\OSGeo4W\bin\gdal2tiles.py", line 1

--- /usr/bin/gdal2tiles.py 2011-11-24 16:20:02.340068830 +0100

SyntaxError?: invalid syntax

I'm sure I've done something stupid.

Thanks kyle

[stefantalpalaru]

You're supposed to apply the patch like this (in a directory with the original gdal2tiles.py): patch -p0 -i gdal2tiles_parallelize_base_and_overview_tiles.patch

Here's the patched script, in case you don't have access to the 'patch' program:  http://pastebin.com/JrRLcyjA

[toolboy]

Ah, I was doing something stupid. That would explain that. Thanks.

I have FWTools 2.4.7, OSGeo4W with GDAL 1.7.3, and Python 2.7.2. The non-patched gdal2tiles was working fine with this setup.

When I run the patched version through the OSGeo shell I get this:

J:\>gdal2tiles.py -zoom 1-11 Nord.tif Traceback (most recent call last):

File "C:\OSGeo4W\bin\gdal2tiles.py", line 44, in <module>

import gdal

File "C:\OSGeo4W\apps\Python27\lib\site-packages\gdal.py", line 2, in <module>

from osgeo.gdal import deprecation_warn

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 21,

in <module>

_gdal = swig_import_helper()

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 17,

in swig_import_helper

_mod = imp.load_module('_gdal', fp, pathname, description)

ImportError?: DLL load failed: %1 is not a valid Win32 application.

It seems to not like my GDAL version. Is there a version you can recommend?

Thanks again, Kyle

[stefantalpalaru]

I'm using the latest stable release: gdal-1.9.0

[myp]

Hi, is there currently any reason why this patch was not applied to the current gdal version ? It seems to run pretty well. It speeds up the processing of my files significantly.