[WMS Client] Combine multiple layers from same server into a single request

[dmorissette]

With the current implementation of the WMS client support, multiple layers 
coming from the same WMS server each generate an individual request to the 
remote server.  This makes rendering very slow for maps with multiple WMS 
layers.

One option is that someone building a mapfile with WMS connections can place 
multiple layer names in the 'LAYERS' WMS parameter, but then they lose the 
ability to turn layers on and off individually via the user interface.

The ultimate solution will be to make the required changes in MapServer's 
msDrawMap() to detect consecutive layers coming from the server in the mapfile 
and to combine them into a single request.

[dmorissette]

See also bug 117

[dmorissette]

This should be a P1 for version 4.1

[dmorissette]

Finally getting to this one... unfortunately it's a bit more tricky than I
initially thought to change the way the WMS layers are processed.

[dmorissette]

Finally!  I have committed the changes in the 4.1 CVS source that combine
multiple WMS layers into a single request when possible.  Seems to work fine in
my first round of tests.  I'll test some more, and also add an option to force
the layers to be requested individually and then I'll mark this fixed.

[dmorissette]

OK.this is FIXED.  I have tested some more, and added a
wms_force_separate_request metadata that can be used to force a given layer to
be requested in its own separate GetMap request. I also updated the
wms-client-howto.xml.

Jeff, I was unable to test the wms-client-howto.xml with the online "MDP XML
Transformer"... are you able to test the latest version of the document at your
end and confirm that I didn't break anything?

[jmckenna@…]

ok i'll test the latest doc version

[bartvde@…]

Is it possible to get a new Windows binary of php_mapscript? I would be willing 
to test this functionality as I am very interested in it (especially the 
possible gain in performance).

On a totally different level, is the current approach in the Mapserver WMS Cient 
asynchronous when images come from more than 1 remote WMS server?

[dmorissette]

When requests are sent to multiple WMS servers, they are all sent at the same
time in parallel and MapServer reads data from each http connection as it
becomes available.  However we still have to wait for the longest download to
complete before MapServer can move on to draw the WMS layers map and return it
to the client.

I'll let Assefa comment on the availability of a win32 binary.

[jmckenna@…]

the latest wms-client-howto.xml doc is valid.

[dmorissette]

Thanks for checking Jeff.

[assefa]

New binaries avilable at
http://www.maptools.org/php_mapscript/index.phtml?page=downloads.html

[bartvde@…]

Works swell!

If I use 11 layers coming from 1 external WMS the time gain is about 2 to 2,5 
seconds (total Chameleon startup time decreases from 6 to less than 4 seconds).

If I use 3 layers coming from 1 external WMS the time gain is around 1 second 
(total Chameleon startup time decreases from 4 to around 3 seconds).

So judging on Daniel's last response, most of the time gain is probably not in 
the HTTP area (as already all requests were made parallel), but in less drawing 
effort for Mapscript?

[dmorissette]

Actually, the time to draw the combined map on the client side is minimal in the
tests that I've made compared to the processing time on the remote WMS server.
In many cases, the gain is that instead of hitting the same server with 3, 5 or
more concurrent requests, you hit it with a single request which is much easier
on the WMS server and allows it to respond much faster.  

Let's take an extreme case: the well-known "GMap context" which contains more
than half a dozen layers pointing to our www2.dmsolutions.ca server.  If you hit
our server with 7 concurrent requests you get something like this:

-------------------------
[Tue Apr  6 21:45:24 2004].729178 msHTTPExecuteRequests() timing summary per
layer (connect_time + time_to_first_packet + download_time = total_time in seconds)
[Tue Apr  6 21:45:24 2004].729318 Layer 0: 0.000 + 6.960 + 0.132 = 7.092s
[Tue Apr  6 21:45:24 2004].729889 Layer 1: 0.000 + 6.975 + 0.109 = 7.084s
[Tue Apr  6 21:45:24 2004].730133 Layer 3: 0.000 + 6.715 + 0.160 = 6.875s
[Tue Apr  6 21:45:24 2004].730337 Layer 4: 0.000 + 7.041 + 0.056 = 7.097s
[Tue Apr  6 21:45:24 2004].730538 Layer 5: 0.000 + 7.008 + 0.080 = 7.088s
[Tue Apr  6 21:45:24 2004].730750 Layer 6: 0.000 + 6.820 + 0.002 = 6.822s
[Tue Apr  6 21:45:24 2004].730962 Layer 9: 0.005 + 7.119 + 0.007 = 7.131s
[Tue Apr  6 21:45:24 2004].731214 msDrawRasterLayerLow(l0:bathymetry): entering.
[Tue Apr  6 21:45:24 2004].764075 msDrawMap(): Layer 0 (l0:bathymetry), 0.033s
[Tue Apr  6 21:45:24 2004].764204 msDrawRasterLayerLow(l1:land_fn): entering.
[Tue Apr  6 21:45:24 2004].791593 msDrawMap(): Layer 1 (l1:land_fn), 0.027s
[Tue Apr  6 21:45:24 2004].791728 msDrawRasterLayerLow(l3:drain_fn): entering.
[Tue Apr  6 21:45:24 2004].818644 msDrawMap(): Layer 3 (l3:drain_fn), 0.027s
[Tue Apr  6 21:45:24 2004].818783 msDrawRasterLayerLow(l4:drainage): entering.
[Tue Apr  6 21:45:24 2004].846193 msDrawMap(): Layer 4 (l4:drainage), 0.027s
[Tue Apr  6 21:45:24 2004].846369 msDrawRasterLayerLow(l5:prov_bound): entering.
[Tue Apr  6 21:45:24 2004].873657 msDrawMap(): Layer 5 (l5:prov_bound), 0.027s
[Tue Apr  6 21:45:24 2004].873799 msDrawRasterLayerLow(l6:fedlimit): entering.
[Tue Apr  6 21:45:24 2004].900657 msDrawMap(): Layer 6 (l6:fedlimit), 0.027s
[Tue Apr  6 21:45:24 2004].900797 msDrawRasterLayerLow(l9:popplace): entering.
[Tue Apr  6 21:45:24 2004].928165 msDrawMap(): Layer 9 (l9:popplace), 0.027s
[Tue Apr  6 21:45:24 2004].928288 msDrawMap(): Drawing Label Cache, 0.000s
[Tue Apr  6 21:45:24 2004].928328 msDrawMap() total time: 7.648s
-------------------------

OTOH if you combine all layers into a single request you get this:

-------------------------
[Tue Apr  6 21:43:39 2004].679218 msHTTPExecuteRequests() timing summary per
layer (connect_time + time_to_first_packet + download_time = total_time in seconds)
[Tue Apr  6 21:43:39 2004].679350 Layer 0: 0.005 + 1.442 + 0.019 = 1.466s
[Tue Apr  6 21:43:39 2004].679876 msDrawRasterLayerLow(l0:bathymetry): entering.
[Tue Apr  6 21:43:39 2004].714816 msDrawMap(): Layer 0 (l0:bathymetry), 0.035s
[Tue Apr  6 21:43:39 2004].714937 msDrawMap(): Layer 1 (l1:land_fn), 0.000s
[Tue Apr  6 21:43:39 2004].714964 msDrawMap(): Layer 3 (l3:drain_fn), 0.000s
[Tue Apr  6 21:43:39 2004].714989 msDrawMap(): Layer 4 (l4:drainage), 0.000s
[Tue Apr  6 21:43:39 2004].715013 msDrawMap(): Layer 5 (l5:prov_bound), 0.000s
[Tue Apr  6 21:43:39 2004].715038 msDrawMap(): Layer 6 (l6:fedlimit), 0.000s
[Tue Apr  6 21:43:39 2004].715064 msDrawMap(): Layer 9 (l9:popplace), 0.000s
[Tue Apr  6 21:43:39 2004].715090 msDrawMap(): Drawing Label Cache, 0.000s
[Tue Apr  6 21:43:39 2004].715116 msDrawMap() total time: 1.815s
-------------------------

In this (extreme) case we go from 7.6 seconds to 1.8 seconds to draw the map...
of course you will not always get that king of performance gain.  You can
produce your own timing reports if you compile mapserver using --enable-debug,
and then set DEBUG TRUE in your mapfile at the map level and in every layer of
interest.

[dmorissette]

Note that both examples above are really producing the same map. In the second
example all the download and processing time is accounted for on "layer 0"
because all other layers in the map were merged into the same request as layer
0... that's why you see 0s times for all other layers... but all the layers were
really requested from the remote WMS.