Ticket #4379 (new enhancement)

Opened 3 years ago

Last modified 9 days ago

[PATCH] gdal2tiles.py: parallelize base tile generation

Reported by: stefantalpalaru Owned by: warmerdam
Priority: normal Milestone:
Component: Utilities Version: 1.8.1
Severity: normal Keywords: gdal2tiles
Cc:

Description

Attached is a patch to make gdal2tiles.py use all the available cores when generating the base tiles.

Attachments

gdal2tiles_parallelize_base_tiles.patch Download (5.9 KB) - added by stefantalpalaru 3 years ago.
gdal2tiles_parallelize_base_and_overview_tiles.patch Download (12.5 KB) - added by stefantalpalaru 3 years ago.
gdal2tiles_parallelize_base_and_overview_tiles-1.11.1.patch Download (15.9 KB) - added by bnorman 3 weeks ago.
updated patch for GDAL 1.11.1, removes requirement to specify zoom level

Change History

Changed 3 years ago by stefantalpalaru

Changed 3 years ago by stefantalpalaru

follow-up: ↓ 2   Changed 3 years ago by stefantalpalaru

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

in reply to: ↑ 1 ; follow-up: ↓ 3   Changed 3 years ago by dave

Replying to stefantalpalaru:

After seeing how slow the 'lanczos' resampling method is, I did the same for the overview tile generation. This second patch replaces the first.

Hi, The first patch works well, but I encountered a problem with the second patch. It generated the base layer ok but then crashed immediately before beginning the first overview layer.

Here's the output:

...[1800000 lines removed]...
Generating Overview Tiles:
Traceback (most recent call last):
 File "./gdal2tiles.py", line 2283, in <module>
   for tz in range(gdal2tiles.tmaxz-1, gdal2tiles.tminz-1, -1):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' 

in reply to: ↑ 2 ; follow-up: ↓ 4   Changed 3 years ago by stefantalpalaru

Replying to dave:

What's the command line? Looks like you're missing the zoom levels so tmaxz and tminz default to None.

in reply to: ↑ 3   Changed 3 years ago by dave

Replying to stefantalpalaru:

Thats it! If I specify zoom levels on the command line ( --zoom 1-11 ) everything works as expected. I was not specifying the zoom levels - I wanted all levels and gdal-tiles.py normally defaults to that. It appears that this patch requires --zoom to be specified, which is fine but perhaps should be documented.

I am very impressed with with speedup obtained form this. Thanks!

follow-up: ↓ 6   Changed 3 years ago by toolboy

I'm trying to get this running on Windows 7 Ultimate. This update looks like it's for UNIX\LINIX. Is there any chance this can get ported to Windows?

Thanks.

in reply to: ↑ 5   Changed 3 years ago by stefantalpalaru

Replying to toolboy:

The multiprocessing module should run fine on Windows. What errors are you getting?

  Changed 3 years ago by toolboy

Stefan, Thanks you for the quick response. The first thing I did was rename gdal2tiles_parallelize_base_and_overview_tiles.patch to gdal2tiles.py. Then I ran the following in OSGeo4W:

J:\>gdal2tiles Nord.tif

File "C:\OSGeo4W\bin\gdal2tiles.py", line 1

--- /usr/bin/gdal2tiles.py 2011-11-24 16:20:02.340068830 +0100

SyntaxError?: invalid syntax

I'm sure I've done something stupid.

Thanks kyle

  Changed 3 years ago by stefantalpalaru

You're supposed to apply the patch like this (in a directory with the original gdal2tiles.py): patch -p0 -i gdal2tiles_parallelize_base_and_overview_tiles.patch

Here's the patched script, in case you don't have access to the 'patch' program:  http://pastebin.com/JrRLcyjA

  Changed 3 years ago by toolboy

Ah, I was doing something stupid. That would explain that. Thanks.

I have FWTools 2.4.7, OSGeo4W with GDAL 1.7.3, and Python 2.7.2. The non-patched gdal2tiles was working fine with this setup.

When I run the patched version through the OSGeo shell I get this:

J:\>gdal2tiles.py -zoom 1-11 Nord.tif Traceback (most recent call last):

File "C:\OSGeo4W\bin\gdal2tiles.py", line 44, in <module>

import gdal

File "C:\OSGeo4W\apps\Python27\lib\site-packages\gdal.py", line 2, in <module>

from osgeo.gdal import deprecation_warn

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 21,

in <module>

_gdal = swig_import_helper()

File "C:\OSGeo4W\apps\Python27\lib\site-packages\osgeo\init.py", line 17,

in swig_import_helper

_mod = imp.load_module('_gdal', fp, pathname, description)

ImportError?: DLL load failed: %1 is not a valid Win32 application.

It seems to not like my GDAL version. Is there a version you can recommend?

Thanks again, Kyle

  Changed 3 years ago by stefantalpalaru

I'm using the latest stable release: gdal-1.9.0

  Changed 2 years ago by myp

Hi, is there currently any reason why this patch was not applied to the current gdal version ? It seems to run pretty well. It speeds up the processing of my files significantly.

  Changed 22 months ago by michalis

Tried using this one on Windows 7.

Getting a Windows Error 32: cannot access the temp file because it is used by another process.

Made sure there are no leftover temp files, rerun, getting the same error. Any ideas?

  Changed 8 months ago by hamish

  • keywords gdal2tiles added

  Changed 4 months ago by jratike80

I made a quick test on Windows 7. A modified script does not feel like production ready without some further work.

c:\temp\gdal2tiles_2.py --s_srs epsg:3067 c:\temp\poista\tile\uk3l.png
Process Process-1:
Traceback (most recent call last):
  File "C:\Program Files\QGIS Dufour\apps\Python27\lib\multiprocessing\process.p
y", line 258, in _bootstrap
    self.run()
  File "C:\Program Files\QGIS Dufour\apps\Python27\lib\multiprocessing\process.p
y", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "C:\temp\gdal2tiles_2.py", line 2243, in worker_metadata
    gdal2tiles.open_input()
  File "C:\temp\gdal2tiles_2.py", line 878, in open_input
    os.unlink(tempfilename)
WindowsError: [Error 32] Process can't file because it is used by another process 'c:\\users\\rahkonju\\appdata\\local\\temp\\tmpjsdkl7-gdal2tiles.vrt'
Generating Base Tiles:
Generating Overview Tiles:
Traceback (most recent call last):
  File "C:\temp\gdal2tiles_2.py", line 2283, in <module>
    for tz in range(gdal2tiles.tmaxz-1, gdal2tiles.tminz-1, -1):
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

  Changed 2 months ago by rouault

  • summary changed from gdal2tiles.py: parallelize base tile generation to [PATCH] gdal2tiles.py: parallelize base tile generation

Changed 3 weeks ago by bnorman

updated patch for GDAL 1.11.1, removes requirement to specify zoom level

  Changed 3 weeks ago by bnorman

I have added an updated version of the patch that works against GDAL 1.11.1 . It also removes the requirement for manually setting the zoom levels on the command line (so the behavior is the same as the unmodified script).

It would be great if this was mainlined.

  Changed 9 days ago by knutole

@bnorman, I'm not getting valid tiles with this script.

With unmodified gdal2tiles.py, the common output is: Generating Base Tiles: 0...10...20...30...40...50...60...70...80...90...100 - done. Generating Overview Tiles: 0...10...20...30...40...50...60...70...80...90...100 - done.

With the parallel version you committed 10 days ago, the output is this:

Generating Base Tiles: 0...10...20...Generating Overview Tiles:

...and the tiles are not created. Run with simple flags: gdal2tiles.py -w none -p mercator --no-kml [...]

Any idea why? Thanks!

  Changed 9 days ago by knutole

http://systemapic.com/wp-content/uploads/badtile.png I'm getting tiles like these.

  Changed 9 days ago by knutole

Running with flag --processes=1, everything works well... Hit me back on @kosjoli on twitter if you need more debug. Thanks.

Note: See TracTickets for help on using tickets.