This page is part of the MapGuide Future section, where ideas are proposed and refined before being turned into RFCs (or discarded). Visit the Future page to view more!

Overview

This page is a living proposal for improving the scalability of the Tile Service.
Mapguide supports tiled maps via both WMS and it's own Tile Service using GETTILETIMAGE.

Recently support was added in Openlayers & Fusion to access the Tile Service via an API.
​OpenLayers. Layer. MapGuide
​#995 (add support for MapGuide OS layer type) - OpenLayers - Trac

API Ideas

SETTILEIMAGE would allow the tile cache to seeded without needing to have remote access to the server file system.

GETTILEPARAMS would return a list of scale ranges, tile sizes and bounds for a given tile cache.

GETTILECACHE would return a 'map' of the tile cache, listing which tiles exist and which ones haven't yet been created

HAVETILEIMAGE would enable a server to poll another server(s) in the cluster,
returning status codes of 200 ok or 404 Not Found respectively. This allows the server to
check the tile cache without triggering a tile render on the polled server.

These would enable more extensive API based management of a cluster of Mapguide servers, via accurate seeding
and management of the tile caches.

Cacheability by HTTP

MapGuideRfc11 added support for a Stateless Http GETTILEIMAGE request, however, these tiles are served
without any cache headers which means they can only be proxied using a custom 'aggressive' proxy service.

Add proper cache headers & file date to GETTILEIMAGE response.

One of the the issues to be resolved is how long to set the cache headers to cache the tiles before checking back
to the Mapguide Server. This could done with a serverconfig.ini default and then an overriding parameter in the
ResourceHeader.

API additions to MgTileService

To support HTTP cacheability we need to be able to do the following via the MgTileService:

a) Get timestamp information about a generated tile (to be able to apply expiry dates)

b) "Peek" at the timestamp for a tile that may or may not be generated (to be able to compare against dates from if-modified-since headers)

An API addition to MgTileService like the one below should be able to support the above scenarios.

class MG_MAPGUIDE_API MgTileService : MgService
{
PUBLISHED_API:
   /// Returns the timestamp of when the tile for the specified map/group/row/col was generated. Returns NULL if no such tile exists
   ///
   MgDateTime* GetTileCreationDate(MgMap* map, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow);

   /// Returns the timestamp of when the tile for the specified map/group/row/col/scale was generated. Returns NULL if no such tile exists
   ///
   MgDateTime* GetTileCreationDate(MgResourceIdentifier* mapDefinition, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex);

   /// Returns the specified base map tile for the given map.  If a cached tile
   /// image exists it will return it, otherwise the tile is rendered and added
   /// to the cache.
   MgTile* GetTile(MgMap* map, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow);

   /// Returns the specified base map tile for the given map.  If a cached tile
   /// image exists it will return it, otherwise the tile is rendered and added
   /// to the cache.
   ///
   MgTile* GetTile(MgResourceIdentifier* mapDefinition, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex);
};

MgTile is defined like so.

class MG_MAPGUIDE_API MgTile : public MgSerializable
{
PUBLISHED_API:
    /// Returns the tile image
    ///
    MgByteReader* GetImage();

    /// Returns the date this tile was created
    ///
    MgDateTime* GetCreationDate();
};

MapAgent modifications

These API changes do not really affect the mapagent interface. From a calling client's perspective, they should still be able to send v1.2.0 GETTILEIMAGE requests with the same parameters and take advantage of HTTP caching behind the scenes. Behind the scenes the GETTILEIMAGE operation handler simply needs to check for the existence of a "If-Modified-Since" request header to take on a new code path.

A rough pseudocode overview of this process would be

if (ifModifiedSince header exists) {
    extract date from header
    call GetTileCreationDate
    if (tileCreationDate != NULL) {
        if (tileCreationDate is newer than header date) {
            call new GetTile
            write image from MgTile into result
            write date from MgTile into last modified response header
            write expires date a long period from that date (6 months? 1 year?)
        } else {
            set status code of 304. Write nothing into the result. Outer CGI/Apache/ISAPI handler is expected to handle this and write out the appropriate responses (see below)
        }        
    } else {
        call new GetTile
        write image from MgTile into result
        write date from MgTile into last modified response header
        write expires date a long period from that date (6 months? 1 year?)
    }
} else {
    call new GetTile
    write image from MgTile into result
    write date from MgTile into last modified response header
    write expires date a long period from that date (6 months? 1 year?)    
}

In our applicable CGI/Apache/ISAPI handlers, their additional responsibilities are to:

• Pack any HTTP request headers into the request metadata of the MgHttpRequest before executing it. In the case of this API, look for if-modified-since
• Handle the 304 internal status and write out any applicable response headers from the MgHttpResult that operations supporting HTTP cacheability (ie. The GETTILEIMAGE) should provide.

MgHttpRequestMetadata and MgHttpHeader classes are not currently used in any of the existing CGI/Apache/ISAPI handlers. We should use them for this purpose.

Re-usable tile sets

Currently the concept of a tile set is tightly-bound to the Map Definition that severely limits its re-usability between other maps.

The second problem is that because a tile set is bound to a Map Definition, we have the all too common problem of shotgun tile cache invalidation on even the most minute changes in the Map Definition and/or upstream dependent resources.

The TileSetDefinition resource

Ideally the tile set should be modelled not as a part of a Map Definition, but as a separate resource.

A tentative schema for this resource would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" attributeFormDefault="unqualified">
  <xs:element name="TileSetDefinition" minOccurs="0">
    <xs:annotation>
      <xs:documentation>Defines a tile cache</xs:documentation>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element name="TileStoreParameters" type="TileStoreParametersType">
           <xs:annotation>
            <xs:documentation>Defines the parameters to access the tile cache</xs:documentation>
          </xs:annotation>
        </xs:element>
        <xs:element name="Extents" type="Box2DType">
          <xs:annotation>
            <xs:documentation>A bounding box around the area of the tile cache</xs:documentation>
          </xs:annotation>
        </xs:element>
        <xs:element name="CoordinateSystem" type="xs:string">
          <xs:annotation>
            <xs:documentation>The coordinate system as WKT used by the TileSetDefinition</xs:documentation>
          </xs:annotation>
        </xs:element>
        <xs:element name="FiniteDisplayScale" type="xs:double" maxOccurs="unbounded">
          <xs:annotation>
            <xs:documentation>The display scales that the base map layers will have tiles available. Applies to the HTML viewer.</xs:documentation>
          </xs:annotation>
        </xs:element>
        <xs:element name="BaseMapLayerGroup" type="BaseMapLayerGroupCommonType" minOccurs="1" maxOccurs="unbounded">
          <xs:annotation>
            <xs:documentation>A group of layers that is used to compose a tiled layer in the HTML viewer</xs:documentation>
          </xs:annotation>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:complexType name="TileStoreParametersType">
    <xs:annotation>
      <xs:documentation>TileStoreParameters defines the parameters of this tile cache.</xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="TileWidth" type="xs:integer">
        <xs:annotation>
          <xs:documentation>The width of tile images in this tile cache</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="TileHeight" type="xs:integer">
        <xs:annotation>
          <xs:documentation>The height of tile images in this tile cache</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="ImageFormat" type="xs:string">
        <xs:annotation>
          <xs:documentation>The image format of tile images in this tile cache</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="TileProvider" type="xs:string">
        <xs:annotation>
          <xs:documentation>The tile image provider</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="Parameter" type="NameValuePairType" minOccurs="0" maxOccurs="unbounded">
        <xs:annotation>
          <xs:documentation>Collection of name value pairs for connecting to the tile image provider</xs:documentation>
        </xs:annotation>
      </xs:element>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="NameValuePairType">
    <xs:annotation>
      <xs:documentation>A type describing name and value pairs</xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="Name" type="xs:string">
        <xs:annotation>
          <xs:documentation>Text for the name of parameter</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="Value" type="xs:string">
        <xs:annotation>
          <xs:documentation>Text for value of parameter</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="ExtendedData1" type="ExtendedDataType" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="Box2DType">
    <xs:annotation>
      <xs:documentation>Box2D encapsulates the the coordinates of a box in 2-D space</xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="MinX" type="xs:double">
        <xs:annotation>
          <xs:documentation>Minimum x-coordinate</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="MaxX" type="xs:double">
        <xs:annotation>
          <xs:documentation>Maximum x-coordinate</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="MinY" type="xs:double">
        <xs:annotation>
          <xs:documentation>Minimum y-coordinate</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="MaxY" type="xs:double">
        <xs:annotation>
          <xs:documentation>Maximum y-coordinate</xs:documentation>
        </xs:annotation>
      </xs:element>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="BaseMapLayerType">
    <xs:annotation>
      <xs:documentation>BaseMapLayerType encapsulates the properties of a BaseMapLayer.</xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="Name" type="xs:string">
        <xs:annotation>
          <xs:documentation>Name of the MapLayer</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="ResourceId" type="xs:string">
        <xs:annotation>
          <xs:documentation>ResourceId of the MapLayer</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="Selectable" type="xs:boolean">
        <xs:annotation>
          <xs:documentation>Whether or not the Layer can be selected</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="ShowInLegend" type="xs:boolean">
        <xs:annotation>
          <xs:documentation>Whether or not the Layer should be shown in the legend</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="LegendLabel" type="xs:string">
        <xs:annotation>
          <xs:documentation>Label to be shown for the Layer in the legend</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="ExpandInLegend" type="xs:boolean">
        <xs:annotation>
          <xs:documentation>Whether or not the Layer should be expanded in the legend.</xs:documentation>
        </xs:annotation>
      </xs:element>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="MapLayerGroupCommonType">
    <xs:annotation>
      <xs:documentation>MapLayerGroupCommonType is a common subclass of MapLayerGroupCommonType and BaseMapLayerGroupCommonType</xs:documentation>
    </xs:annotation>
    <xs:sequence>
      <xs:element name="Name" type="xs:string">
        <xs:annotation>
          <xs:documentation>The name of this LayerGroup</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="Visible" type="xs:boolean">
        <xs:annotation>
          <xs:documentation>Whether this group's visiblity should be visible or not when it first comes into range</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="ShowInLegend" type="xs:boolean">
        <xs:annotation>
          <xs:documentation>Whether or not the LayerGroup should be shown in the legend</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="ExpandInLegend" type="xs:boolean">
        <xs:annotation>
          <xs:documentation>Whether or not the LayerGroup should be initially expanded in the legend</xs:documentation>
        </xs:annotation>
      </xs:element>
      <xs:element name="LegendLabel" type="xs:string">
        <xs:annotation>
          <xs:documentation>Label to be shown for the LayerGroup in the legend</xs:documentation>
        </xs:annotation>
      </xs:element>
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="BaseMapLayerGroupCommonType">
    <xs:annotation>
      <xs:documentation>BaseMapLayerGroupCommonType encapsulates the properties of a BaseMapLayerGroup. It extends MapLayerGroupCommonType by holding the layers in the group.  The base map layer groups defines what layers are used to render a tile set in the HTML viewer.</xs:documentation>
    </xs:annotation>
    <xs:complexContent>
      <xs:extension base="MapLayerGroupCommonType">
        <xs:sequence>
          <xs:element name="BaseMapLayer" type="BaseMapLayerType" minOccurs="0" maxOccurs="unbounded">
            <xs:annotation>
              <xs:documentation>The layers that are part of this group. The order of the layers represents the draw order, layers first is the list are drawn over top of layers later in the list.</xs:documentation>
            </xs:annotation>
          </xs:element>
        </xs:sequence>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>
</xs:schema>

From a high-level view, the TileSetDefinition is essentially the BaseMapDefinition portion of the MapDefinition schema ripped out and put into its own resource, but with the following unique characteristics:

• Ability to have its own custom tile image sizes
• Ability to have its own tile image format

As a forward looking measure a TileSetDefinition supports the concept of a Tile Provider (think FDO for tile sets). For the short term to see this feature realized there will only be one Tile Provider (the existing MapGuide-designated directory on the file system), but the schema provides a name/value pair mechanism for specifying connection parameters, that allows for MapGuide to support tile storage/retrieval from sources other than the MapGuide-designated file system (eg. MBTiles)

APIs to support tile sets

The MgTileService will get new APIs to work with TileSetDefinitions:

class MG_MAPGUIDE_API MgTileService : public MgService
{
PUBLISHED_API:
    /// Clears the entire tile cache for the given map.  Tiles for all base
    /// map layer groups and finite scales will be removed.
    ///
    virtual void ClearCache(MgResourceIdentifier* tileSetDefinition);

    /// Marks a given tile cache as locked. A locked tile cache will prevent resource
    /// content changes in up-stream dependent resources like Layer Definitions and 
    /// Feature Sources
    ///
    virtual void LockCache(MgResourceIdentiifer* tileSetDefinition);

    /// Marks a given tile cache as un-locked. An un-locked tile cache can be invalidated
    /// by any resource content changes in up-stream dependent resources like Layer Definitions
    /// and Feature Sources.
    ///
    virtual void UnlockCache(MgResourceIdentiifer* tileSetDefinition);
};

Or if we want to combine this with the HTTP 304 cacheability support

class MG_MAPGUIDE_API MgTileService : public MgService
{
PUBLISHED_API:
    ///
    ///
    virtual void ClearCache(MgResourceIdentifier* tileSetDefinition);

    /// Marks a given tile cache as locked. A locked tile cache will prevent resource
    /// content changes in up-stream dependent resources like Layer Definitions and 
    /// Feature Sources
    ///
    virtual void LockCache(MgResourceIdentiifer* tileSetDefinition);

    /// Marks a given tile cache as un-locked. An un-locked tile cache can be invalidated
    /// by any resource content changes in up-stream dependent resources like Layer Definitions
    /// and Feature Sources.
    ///
    virtual void UnlockCache(MgResourceIdentiifer* tileSetDefinition);

    /// Returns the timestamp of when the tile for the specified map/group/row/col/scale was generated. Returns NULL if no such tile exists
    ///
    MgDateTime* GetTileCreationDate(MgResourceIdentifier* resource, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex);

    /// Returns the specified base map tile for the given map.  If a cached tile
    /// image exists it will return it, otherwise the tile is rendered and added
    /// to the cache.
    ///
    MgTile* GetTile(MgResourceIdentifier* resource, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex);
};

Notice that no new GetTile overload is introduced (assuming this feature is separate from the HTTP 304 cacheability support). This is because tile access requires the same set of parameters, except that instead of a Map Definition, we can specify a TileSetDefinition instead. On the implementation side, the GetTile methods need to support TileSetDefinition as a valid resource type and be able to resolve the physical tile storage location to fetch and store tiles in from the TileSetDefinition. For a first cut implementation, it is safe to just hard-code this logic to resolve to the same physical tile cache directory that the existing MapGuide tile cache code uses.

Tile Set Locking

To prevent accidental invalidation due to editing upstream dependent resources, the MgTileService will support the concept of locking and unlocking TileSetDefinition resources.

When a TileSetDefinition is saved for the first time as a new resource it is in a default unlocked state. An unlocked Tile Set is susceptible to all the usual tile cache invalidation mechanisms:

• The TileSetDefinition is overwritten
• An upstream Layer Definition or Feature Source has changed

When a TileSetDefinition is locked with the MgTileService::LockCache() API, the above scenarios will be prevented via throwing the appropriate MgException (class type TBD) on saving of resource content on a TileSetDefinition or any of its upstream resource types. The exception thrown should carry the resource id of the downstream TileSetDefinition that would be affected by this operation so it can be relayed to the user or client application that they can't save this resource because it would invalidate a locked tile set.

The act of locking a TileSetDefinition is basically saying that this tile cache and all upstream resources are "read-only" and its tile cache can not be invalidated unless explicitly unlocked.

The act of having a TileSetDefinition unlocked is saying that this tile cache could be invalidated at any time.

The actual implementation details are still TBD. The tentative idea would be to leverage the existing server Resource Service APIs (ie. DBXML) to store the tile set lock information, relieving us of synchronization/multi-threaded concerns to read/write lock state (Further discussion required on whether this is a good idea)

Map Definition Schema changes

To retain the additive qualities of previous schema revisions. The Map Definition schema will simply support a new optional TileSetSource element which is simply a reference to a TileSetDefinition

       <xs:element name="BaseMapDefinition" minOccurs="0">
        ...
       </xs:element>
       <xs:element name="TileSetSource" minOccurs="0">
        <xs:annotation>
          <xs:documentation>A reference to the tile set source to use</xs:documentation>
        </xs:annotation>
        <xs:complexType>
          <xs:sequence>
            <xs:element name="ResourceId" type="xs:string">
              <xs:annotation>
                <xs:documentation>ResourceId of the TileSetDefinition</xs:documentation>
              </xs:annotation>
            </xs:element>
          </xs:sequence>
        </xs:complexType>
      </xs:element>

Only up to 1 TileSetSource can be defined for any Map Definition as having multiple TileSetSources presents complex situations like how to address disparity of scale ranges between different TileSetSources.

Because TileSetDefinitions are bound to a specific coordinate system, the CoordinateSystem element of the Map Definition will no longer be the single point of truth when determining if dynamic layers in a map require re-projection. Instead if a Map Definition references a TileSetDefinition, its CoordinateSystem element takes precedence over the one defined in the Map Definition. This requirement can be relaxed/lifted if/when we support raster re-projection of existing rendered tiles (Discussion required: Is this feasible?)

In terms of parsing into a MgMap at runtime, the TileSetSource and BaseMapDefinition are mutually exclusive. In the event that both are defined (because the XML schema allows for it), the BaseMapDefinition element "wins".

MgMap changes

Because Map Definitions are parsed into MgMap objects at runtime, we need to update MgMap and its associated classes (MgLayer/MgLayerGroup) to also take into account support for TileSetDefinitions

The GetMapSRS() method will stay the same, but its behaviour will change. It will now return the CoordinateSystem of the referenced TileSetDefinition if one exists.

Depending on which one is defined (or which one "wins" under our rules of element precendence), the finite scale list will be initialized from either the BaseMapDefinition element or the referenced TileSetDefinition

The MgLayerGroupType class will introduce a new constant

class MgLayerGroupType
{
PUBLISHED_API:
    /// Specifies that the layer is a base map layer from a TileSetDefinition resource
    ///
    static const INT32 BaseMapFromTileSet = 3;
}

The MgLayerGroup will have a new API to return the TileSetDefinition resource id where this group comes from

class MG_PLATFORMBASE_API MgLayerGroup : public MgNamedSerializable
{
PUBLISHED_API:
    /// Returns the resource id of the Tile Set Definition that this group originates from.
    /// Returns NULL if GetLayerGroupType() is not MgLayerGroupType::BaseMapFromTileSet 
    ///
    MgResourceIdentifier* GetTileSetDefinition();
}

Client application logic changes to support TileSetDefinition resources

The beautiful thing about this new feature is that from the client application perspective, the calling interface remains the same. The only difference is that the Map Definition can now also be a TileSetDefintion, and client applications should know which resource type to use with the new API in MgLayerGroup

WMS

WMS Cache headers - see above.

Publish MapDefinitions via WMS Currently only layers are exposed via WMS, maps could be as well.

The WMS service doesn't currently use the tile cache, which means every WMS request is rendered.

Linking the WMS service up to utilize the tile cache would dramatically improve the performance
and capacity of WMS with Mapguide.

TMS

TMS