Ability to extract tiles from an existing cache to another

[aboudreault]

One of our needs is to generate tiles for a specific extent/region and make a packages or those tiles. This is what I would need that utility to do:

• Generated the tiles list of the region given (taking in account the zoom levels given, ogr intersects options etc.)
• If a tile already exists in the cache of the tileset given, it will be simply copied in the extract cache specified
• If a tile doesn't exists in the cache of the tileset give, render it then copy it in the extract cache specified
• If a tile already exists in the extract cache specified, do nothing.

My first idea was to create a new utilise called mapcache_extract. I then realized that mapcache_seed handle 80% of the needs and that it would be ugly to duplicate and maintain twice that code.

Here's what I propose:

• Add a command line option in mapcache_seed: -x, --extract <cache_name>

By specifying that command line option, mapcache_seed will render all needed tiles (if they don't already exists) and extract those tiles in the specified cache. The cache specified must have been added in the xml config. By implementing this this way, all cache formats will be supported directly.

Example of config and use:

<cache name="beauce-disk" type="disk">
      <template>/tmp/{tileset}/{grid}/{z}/{x}/{inv_y},{ext}</template>
      <symlink_blank/>
   </cache>

-----------------------------

mapcache_seed --config /opt/osm/mapcache/osm.xml --tileset osm -z 0,10 -e -71.34,45.9,-70.25,46.61 -x beauce-disk

Implementation details

---

The attached patch is a simple proof of concept that test the idea. It's not the final patch that I would commit in svn. In that patch, I hack the tile->tileset->cache to do the appropriate operations. The patch works well for disk cache (DEPTH_MODE only) but do not work with mbtiles. This is due to some initial operations that should be done in the post_configuration function but aren't since my cache isn't associated to any tileset.

In the final patch, If the extract option is specified, I would create a temporary tileset (clone of the specified tileset) with a different cache set (the extract cache) and add it cleanly in mapcache_configuration. So, things would be cleaner (no more hack with tile->tileset->cache) and all cache formats will work as expected.

Before finalizing the patch, tbonfort, what do you think of that feature and the approach. Do you think it's worth to add it in mapcache and agree with to fact to include that in mapcache_seed? Anything else to suggest?

[aboudreault]

Simple tile extract option example

[tbonfort]

I think this might be added as a "mode" option to mapcache_seed (currently we have "seed" and "delete"), named transfer.

In cmd_thread, the "transfer" task would be sent down to the seed_threads, and in that function there could be some kind of hack/mechanism to transfer the tile data from the source to the destination cache.

Note that cmd_thread and cmd_recurse will have to be adapted in this case to loop over all the tiles instead just all the metatiles, otherwise you'd only have one tile per metatile transfered to the destination cache.

[aboudreault]

I've implemented the transfer mode in mapcache_seed. Here's my mapcache.xml:

<cache name="transfercachedisk" type="disk">

<template>/tmp/{tileset}/{grid}/{z}/{x}/{inv_y}.{ext}</template> <symlink_blank/>

</cache>

<cache name="transfercache" type="mbtiles">

<dbname_template>/tmp/{tileset}-{grid}.mbtiles</dbname_template>

</cache>

<tileset name="osm">

<metadata>

<title>osm mapserver served map of midi-pyrénées</title> <abstract>see ​http://mapserver-utils.googlecode.com</abstract>

</metadata> <source>osm</source> <cache>transfercachedisk</cache> <format>PNG</format> <grid>WGS84</grid> <grid>g</grid> <metatile>5 5</metatile> <expires>10000</expires> <auto_expire>86400</auto_expire> <metabuffer>10</metabuffer>

</tileset>

<tileset name="transferosm">

<cache>transfercache</cache> <source>osm</source> <format>PNG</format> <grid>WGS84</grid> <grid>g</grid> <metatile>5 5</metatile> <expires>10000</expires> <auto_expire>86400</auto_expire> <metabuffer>10</metabuffer>

</tileset>

the command used

mapcache_seed -n 8 -c /opt/mapcache/mapcache.xml -t osm -x transferosm -m transfer -e -71.6636,48.5297,-71.6279,48.5704 -z 0,5

It works well for disk_cache to disk_cache, sqlite to disk_cache, but unfortunately, even if my cache and tileset are defined in the xml config, I'm getting errors if I try to transfer to a sqlite3 cache. Here's the output:

seeding tile 20 25 5sqlite backend failed on has_tile: near ":x": syntax error

sqlite backend failed on has_tile: near ":x": syntax errorsqlite backend failed on has_tile: near ":x": syntax errorsqlite backend failed on has_tile: near ":x": syntax error

sqlite backend failed on has_tile: near ":x": syntax error sqlite backend failed on has_tile: near ":x": syntax error /WGS84/{z}/20/10.png png

I see the initial mbtiles file properly created with tables... not sure why it fails to write tiles. I'm going to attach the patch.

[tbonfort]

there was a typo in the sqlite statements, fixed in r12821, can you try again ?

[aboudreault]

great. works well now. Let me know if the patch looks good to commit.

[tbonfort]

ps: the tests that look for tile presence, intersection and modifcation time should probably be refactored into an examine_tile() function so as not to duplicate that code in cmd_thread() and cmd_recurse()

cmd examine_tile(mapcache_context *ctx, mapcache_tile *tile) {
/* returns MAPCACHE_SEED, MAPCACHE_SKIP, MAPCACHE_DELETE or MAPCACHE_TRANSFER depending on the presence
    or absence of the tile and the mode in wich the seeder is run
*/
}

[tbonfort]

arg, trac seems to have swallowed my comment :(

Alan,

you should refactor your patch to follow the architecture the seeder uses:

• in cmd_thread (and then cmd_recurse), the code determines for a given tile what action should be undertaken on the tile
• in seed_thread, multiple number of threads consume the tiles that have been queued by cmd_thread.

In your case:

• in cmd_thread you should check for the presence of the tile in the destination cache, and if not push the tile with a MAPCACHE_TRANSFER command.
• in seed_thread, you get the tile from the source tileset, and set it in the destination one:

  if(cmd==MAPCACHE_TRANSFER) {
     mapcache_tileset_tile_get(seed,ctx,tile);
     tile->tileset = transfer_tilest;
     mapcache_tileset_tile_set(seed,ctx,tile);
  }
  

there are at least two more issues to account for:

• the tileset metatile properties are used in cmd_thread to only loop over one tile per metatile. this should be changed to loop over all tiles in this case, but the original metatile info should be kept so correct images are requested from the wms source should some tiles be missing in the original cache.
• there should be a check to make sure the destination source tileset both reference the same grid.

[aboudreault]

Ok... I see what you mean. There is a lock on all metatiles during their generation, so I can push the first tile with MAPCACHE_SEED and the others with MAPCACHE_TRANSFER, with the seeder. I'm going to modify this. I assume it would be ok to use mapcache_tileset_metatile_get to get all x/y tile values of a metatile in cmd_thread/recurse?

[tbonfort]

I'd do do it this way:

• push with MAPCACHE_TRANSFER (i.e. only once per metatile)
• in seed_thread(), do a mapcache_tileset_metatile_get, followed by looping through the returned tiles with mapcache_tileset_tile_get()

doing it this way you won't have multiple threads locking on a wms source request

[aboudreault]

in mapcache_seed, if I have a tileset.. and that the first tile of a metatile exists and all other tiles have been manually deleted for some reason.. they wont be regenerated. I think this could be an issue for some users...

[tbonfort]

This will be an issue only if the tile is already present in the destination cache *and* the rest of the destination cache has been manually tampered with. I don't think this is a sufficient use-case to loop through every tile in cmd_thread, as that case will be non negligeable (especially if you're using ogr geometry filtering)

[aboudreault]

well.. I was more talking about a general issue (not related to transfer mode)

[tbonfort]

true, it would also affect seeding. If you consider this sufficiently important you can add an option so that cmd_thread loops through all tiles instead of all metatiles. I personally don't think this is a high priority, and definitely would not want this to be the default.

[aboudreault]

yeah, an option would be useful I think. Not critical for now.

[aboudreault]

new patch

[aboudreault]

I've attached the new patch. I created an examine_tile function to return the appropriate action. So, now, both cmd_recurse and cmd_thread supports delete mode. In transfer mode, the thread will generate the needed tiles and transfer them.

btw, I'm going to remove the tile_get() call at line 501. It is not really useful.

[tbonfort]

Alan,

• int tranfer_mode_enabled is not used, and should be removed
• the help message should be a bit more explicit, e.g "tileset where tiles should be transfered to"
• yes, you should remove the line 501 call.

seems good to me once that is done

[tbonfort]

another optimization after line 501 is to remove the call to transfer_tileset->..->tile_exists, and unconditionally call tile_set in that case (tile_exists can be costly, tile_set will silently overwrite an existing tile if it already exists, and this situation is only likely to arise if the cache has been manually tampered with)

[aboudreault]

Comitted in r12826.