Ticket #4379 (new enhancement)

Opened 3 years ago

Last modified 11 days ago

gdal2tiles.py: parallelize base tile generation

Reported by: stefantalpalaru Owned by: warmerdam
Priority: normal Milestone:
Component: Utilities Version: 1.8.1
Severity: normal Keywords: gdal2tiles
Cc:

Description

Attached is a patch to make gdal2tiles.py use all the available cores when generating the base tiles.

Attachments

gdal2tiles_parallelize_base_tiles.patch Download (5.9 KB) - added by stefantalpalaru 3 years ago.
gdal2tiles_parallelize_base_and_overview_tiles.patch Download (12.5 KB) - added by stefantalpalaru 3 years ago.

Change History

Changed 3 years ago by stefantalpalaru

Changed 3 years ago by stefantalpalaru

follow-up: ↓ 2   Changed 3 years ago by stefantalpalaru

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

in reply to: ↑ 1 ; follow-up: ↓ 3   Changed 3 years ago by dave

Replying to stefantalpalaru:

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

Hi, The first patch works well, but I encountered a problem with the second patch. It generated the base layer ok but then crashed immediately before beginning the first overview layer.

Here's the output:

...[1800000 lines removed]...
Generating Overview Tiles:
Traceback (most recent call last):
 File "./gdal2tiles.py", line 2283, in <module>
   for tz in range(gdal2tiles.tmaxz-1, gdal2tiles.tminz-1, -1):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' 

in reply to: ↑ 2 ; follow-up: ↓ 4   Changed 3 years ago by stefantalpalaru

Replying to dave:

What's the command line? Looks like you're missing the zoom levels so tmaxz and tminz default to None.

in reply to: ↑ 3   Changed 3 years ago by dave

Replying to stefantalpalaru:

Thats it! If I specify zoom levels on the command line ( --zoom 1-11 ) everything works as expected. I was not specifying the zoom levels - I wanted all levels and gdal-tiles.py normally defaults to that. It appears that this patch requires --zoom to be specified, which is fine but perhaps should be documented.

I am very impressed with with speedup obtained form this. Thanks!

follow-up: ↓ 6   Changed 3 years ago by toolboy

I'm trying to get this running on Windows 7 Ultimate. This update looks like it's for UNIX\LINIX. Is there any chance this can get ported to Windows?

Thanks.

in reply to: ↑ 5   Changed 3 years ago by stefantalpalaru

Replying to toolboy:

The multiprocessing module should run fine on Windows. What errors are you getting?

  Changed 3 years ago by toolboy

Stefan, Thanks you for the quick response. The first thing I did was rename gdal2tiles_parallelize_base_and_overview_tiles.patch to gdal2tiles.py. Then I ran the following in OSGeo4W:

J:\>gdal2tiles Nord.tif

File "C:\OSGeo4W\bin\gdal2tiles.py", line 1

--- /usr/bin/gdal2tiles.py 2011-11-24 16:20:02.340068830 +0100

SyntaxError?: invalid syntax

I'm sure I've done something stupid.

Thanks kyle

  Changed 3 years ago by stefantalpalaru

You're supposed to apply the patch like this (in a directory with the original gdal2tiles.py): patch -p0 -i gdal2tiles_parallelize_base_and_overview_tiles.patch

Here's the patched script, in case you don't have access to the 'patch' program:  http://pastebin.com/JrRLcyjA

  Changed 3 years ago by toolboy

Ah, I was doing something stupid. That would explain that. Thanks.

I have FWTools 2.4.7, OSGeo4W with GDAL 1.7.3, and Python 2.7.2. The non-patched gdal2tiles was working fine with this setup.

When I run the patched version through the OSGeo shell I get this:

J:\>gdal2tiles.py -zoom 1-11 Nord.tif Traceback (most recent call last):

File "C:\OSGeo4W\bin\gdal2tiles.py", line 44, in <module>

import gdal

File "C:\OSGeo4W\apps\Python27\lib\site-packages\gdal.py", line 2, in <module>

from osgeo.gdal import deprecation_warn

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 21,

in <module>

_gdal = swig_import_helper()

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 17,

in swig_import_helper

_mod = imp.load_module('_gdal', fp, pathname, description)

ImportError?: DLL load failed: %1 is not a valid Win32 application.

It seems to not like my GDAL version. Is there a version you can recommend?

Thanks again, Kyle

  Changed 3 years ago by stefantalpalaru

I'm using the latest stable release: gdal-1.9.0

  Changed 22 months ago by myp

Hi, is there currently any reason why this patch was not applied to the current gdal version ? It seems to run pretty well. It speeds up the processing of my files significantly.

  Changed 15 months ago by michalis

Tried using this one on Windows 7.

Getting a Windows Error 32: cannot access the temp file because it is used by another process.

Made sure there are no leftover temp files, rerun, getting the same error. Any ideas?

  Changed 11 days ago by hamish

  • keywords gdal2tiles added
Note: See TracTickets for help on using tickets.