Ticket #4379 (new enhancement)

Opened 3 years ago

Last modified 2 months ago

gdal2tiles.py: parallelize base tile generation

Reported by: stefantalpalaru Owned by: warmerdam
Priority: normal Milestone:
Component: Utilities Version: 1.8.1
Severity: normal Keywords: gdal2tiles
Cc:

Description

Attached is a patch to make gdal2tiles.py use all the available cores when generating the base tiles.

Attachments

gdal2tiles_parallelize_base_tiles.patch Download (5.9 KB) - added by stefantalpalaru 3 years ago.
gdal2tiles_parallelize_base_and_overview_tiles.patch Download (12.5 KB) - added by stefantalpalaru 3 years ago.

Change History

Changed 3 years ago by stefantalpalaru

Changed 3 years ago by stefantalpalaru

follow-up: ↓ 2   Changed 3 years ago by stefantalpalaru

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

in reply to: ↑ 1 ; follow-up: ↓ 3   Changed 3 years ago by dave

Replying to stefantalpalaru:

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

Hi, The first patch works well, but I encountered a problem with the second patch. It generated the base layer ok but then crashed immediately before beginning the first overview layer.

Here's the output:

...[1800000 lines removed]...
Generating Overview Tiles:
Traceback (most recent call last):
 File "./gdal2tiles.py", line 2283, in <module>
   for tz in range(gdal2tiles.tmaxz-1, gdal2tiles.tminz-1, -1):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' 

in reply to: ↑ 2 ; follow-up: ↓ 4   Changed 3 years ago by stefantalpalaru

Replying to dave:

What's the command line? Looks like you're missing the zoom levels so tmaxz and tminz default to None.

in reply to: ↑ 3   Changed 3 years ago by dave

Replying to stefantalpalaru:

Thats it! If I specify zoom levels on the command line ( --zoom 1-11 ) everything works as expected. I was not specifying the zoom levels - I wanted all levels and gdal-tiles.py normally defaults to that. It appears that this patch requires --zoom to be specified, which is fine but perhaps should be documented.

I am very impressed with with speedup obtained form this. Thanks!

follow-up: ↓ 6   Changed 3 years ago by toolboy

I'm trying to get this running on Windows 7 Ultimate. This update looks like it's for UNIX\LINIX. Is there any chance this can get ported to Windows?

Thanks.

in reply to: ↑ 5   Changed 3 years ago by stefantalpalaru

Replying to toolboy:

The multiprocessing module should run fine on Windows. What errors are you getting?

  Changed 3 years ago by toolboy

Stefan, Thanks you for the quick response. The first thing I did was rename gdal2tiles_parallelize_base_and_overview_tiles.patch to gdal2tiles.py. Then I ran the following in OSGeo4W:

J:\>gdal2tiles Nord.tif

File "C:\OSGeo4W\bin\gdal2tiles.py", line 1

--- /usr/bin/gdal2tiles.py 2011-11-24 16:20:02.340068830 +0100

SyntaxError?: invalid syntax

I'm sure I've done something stupid.

Thanks kyle

  Changed 3 years ago by stefantalpalaru

You're supposed to apply the patch like this (in a directory with the original gdal2tiles.py): patch -p0 -i gdal2tiles_parallelize_base_and_overview_tiles.patch

Here's the patched script, in case you don't have access to the 'patch' program:  http://pastebin.com/JrRLcyjA

  Changed 3 years ago by toolboy

Ah, I was doing something stupid. That would explain that. Thanks.

I have FWTools 2.4.7, OSGeo4W with GDAL 1.7.3, and Python 2.7.2. The non-patched gdal2tiles was working fine with this setup.

When I run the patched version through the OSGeo shell I get this:

J:\>gdal2tiles.py -zoom 1-11 Nord.tif Traceback (most recent call last):

File "C:\OSGeo4W\bin\gdal2tiles.py", line 44, in <module>

import gdal

File "C:\OSGeo4W\apps\Python27\lib\site-packages\gdal.py", line 2, in <module>

from osgeo.gdal import deprecation_warn

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 21,

in <module>

_gdal = swig_import_helper()

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 17,

in swig_import_helper

_mod = imp.load_module('_gdal', fp, pathname, description)

ImportError?: DLL load failed: %1 is not a valid Win32 application.

It seems to not like my GDAL version. Is there a version you can recommend?

Thanks again, Kyle

  Changed 3 years ago by stefantalpalaru

I'm using the latest stable release: gdal-1.9.0

  Changed 2 years ago by myp

Hi, is there currently any reason why this patch was not applied to the current gdal version ? It seems to run pretty well. It speeds up the processing of my files significantly.

  Changed 17 months ago by michalis

Tried using this one on Windows 7.

Getting a Windows Error 32: cannot access the temp file because it is used by another process.

Made sure there are no leftover temp files, rerun, getting the same error. Any ideas?

  Changed 2 months ago by hamish

  • keywords gdal2tiles added
Note: See TracTickets for help on using tickets.