Opened 21 years ago

Closed 17 years ago

Last modified 17 years ago

#302 closed enhancement (fixed)

Use semi-dynamic allocation for layer, classes, styles, etc.

Reported by: dmorissette Owned by: dmorissette
Priority: high Milestone: 5.0 release
Component: MapServer C Library Version: 4.1
Severity: minor Keywords:
Cc: pspencer@…, bear@…, tamas

Description (last modified by dmorissette)

Attila Csipa wrote:
> 
> On Thursday 27 March 2003 18:21, Daniel Morissette wrote:
> > > #define MS_MAXLAYERS 1000 /* maximum number of layers in a map file */
> > Is this really an intentional change?  This can have a quite significant
> > impact on memory usage.
> 
> While we are at it, I'd like to ask why isn't this solved ina semi-dynamic way
> ? We all know that the issue of insufficient layer/class space keeps popping
> up in the lists, but I'm not sure why it has to be done this way.
> 
> If there has been a discussion about this I'm sorry for bringing it up again.
> 
> If I'm correct the problem is here:
> 
> int initMap(mapObj *map)
> {
>   map->numlayers = 0;
>   if((map->layers = (layerObj *)malloc(sizeof(layerObj)*MS_MAXLAYERS)) ==
> NULL) {
>     msSetError(MS_MEMERR, NULL, "initMap()");
>     return(-1);
>   }
> ...
> 
> So if MAXLAYERS is high, we're mallocing huge amounts of data. My question is,
> why is this better then mallocing a fixed amount (which would be, say, the
> current MS_MAXLAYERS value), and then when/if we fill up this space just do a
> realloc on a new layer init (this could be done by duplicating allocated
> space and not simply increasing so overhead of both processing and memory
> would not be significant). This way there would be only a memory limit on the
> number of layers and classes, while the loss of speed would be insignificant
> (advanced users could still change the initial number of layers, in which
> case there would be practically no slowdown).
> 
> Regards,
> Attila

Change History (15)

comment:1 by dmorissette, 21 years ago

Cc: bear@… added
Severity: normalenhancement
BTW, there was a thread on mapserver-dev talking about a method to
increase the array of points in a lineObj dynamically, the same
technique could also be applied here as well:
http://mapserver.gis.umn.edu/wilma/mapserver-dev/0301/msg00024.html


comment:2 by dmorissette, 21 years ago

Component: MapServer CGIMapServer C Library
See also bug 303 about discussions on a method to optimize msAddLine() and that 
could be applied here as well.

comment:3 by dmorissette, 21 years ago

Version: 4.04.1
Not in 4.0, maybe 4.1

comment:4 by dmorissette, 20 years ago

Cc: assefa@… added

comment:5 by dmorissette, 20 years ago

Cc: spencer@… added

comment:6 by sdlime, 20 years ago

I wonder if this can be done once across the board. What sucks is that we 
either have to reallocate one object at a time or allocate say 5 new objects 
each time and track the number of allocated vs actual "things". What about 
switching to NULL terminated arrays and not even carrying things like the 
number of classes, layers or styles as part of the C structs and switching to 
while loops? Could we even do something like that with arrays of structs...

comment:7 by dmorissette, 17 years ago

Description: modified (diff)
Milestone: 5.0 release
Owner: changed from sdlime to dmorissette

MS-RFC-17 has just been adopted and will solve this issue for MS_MAXSYMBOLS, MS_MAXLAYERS, MS_MAXCLASSES and MS_MAXSTYLES:

http://mapserver.gis.umn.edu/development/rfc/ms-rfc-17

comment:8 by dmorissette, 17 years ago

Status: newassigned

comment:9 by dmorissette, 17 years ago

Committed a first round of changes in r6337 for dynamic allocation of symbols as per RFC-17:

Replaced checks on MS_MAXSYMBOLS with calls to msGrowSymbolSet() everywhere allocation of symbols is required. Also replaced macro MS_MAXSYMBOLS with MS_SYMBOL_ALLOCSIZE.

comment:10 by dmorissette, 17 years ago

Cc: bear@… added; bear@… removed

I have committed r6356 for dynamic allocation of layers as per RFC-17.

MS_MAXLAYERS replaced with MS_LAYER_ALLOCSIZE and all checks for MS_MAXLAYERS replaced by calls to msGrowMapLayers().

We also had to create a msGrowMapservLayers() to get rid of the static Layers[MS_MAXLAYERS] in the mapservObj struct (used by maptemplate.c and mapserv.c only).

The httpRequestObj array in mapdraw.c is now allocated to a size of map->maxlayers, but there would be room for optimizing this and allocating it only when really needed using a similar approach as the grow methods used for other objects. I have created ticket #2163 for this separate enhancement.

The mappluginlayer.c stuff has also been updated to dynamically allocate the vtItems[] in the VTFactoryObj inside insertNewVTFItem(). I have created ticket #2164 for Tamas to review my changes and eventually add a call to free that struct in msCleanup().

comment:11 by dmorissette, 17 years ago

Cc: bear@… tamas added; bear@… removed

comment:12 by dmorissette, 17 years ago

I have committed r6360 for dynamic allocation of classes as per RFC-17.

This changeset also fixes a few issues that I found related to dynamic allocation of layers and symbols.

Finally, there are some potential leaks of classes in mapogcsld.c and mapogcfilter.c that aregoing to be addressed in ticket #2168.

comment:13 by dmorissette, 17 years ago

Resolution: fixed
Status: assignedclosed

I have committed r6368 for dynamic allocation of styles as per RFC-17.

This completes the implementation of RFC-17. Closing ticket.

comment:14 by sdlime, 17 years ago

Dan: Can you check the current revision to make sure I didn't muck up my changes to mapfile.c with yours? It looks like all is there but I want another set of eyes to check and be sure...

Steve

comment:15 by dmorissette, 17 years ago

I had a quick look at the latest mapfile.c and everything seems fine.

Note: See TracTickets for help on using tickets.