Improve WFS memory usage

[gregboone]

Hi,

I'm looking into a problem in an application using the WFS provider, whereby if it is trying to process a large number of features it uses a large amount of memory - even though the application itself only ever needs one top-level feature to be in memory at a time - resulting in a limit to the size of data that can be processed without the application crashing. As far as I can determine, this isn't a leak. This is happing with builds done against 3.2 and 3.6 releases of FDO.

I had a look at the source code: when the IFeatureReader interface is used to iterate over the available features it only allows access the the current feature, but it appears to be maintaining a collection of every feature seen. At least, this was the impression I got from FdoXmlFeatureReaderImpl::ReadNext - with all the top-level features ending up in m_featureCollection.

So, the questions I have are: (a) Am I correct in thinking that it is holding on to every feature processed by the reader? (b) If so, is there any way of stopping it from doing this?

Thanks, Robin Newton

[gregboone]

I guess I should have also asked: (c) Is FdoXmlFeatureReaderImpl hanging on to all these features likely to be be a major factor in memory use? To which my current answer would have to be "no".

It looks like the real reason my application was crashing was due to FdoXmlUtilXrcs::Xrcs2Unicode overflowing the stack by trying to allocate up to 0.5 meg on it, which I believe should be fixable by changing the limit on stack size.

Thanks, Robin

[gregboone]

Robin Newton wrote:

I guess I should have also asked: (c) Is FdoXmlFeatureReaderImpl hanging on to all these features likely to be be a major factor in memory use? To which my current answer would have to be "no".

That was a bit hasty. It turned out my replication example was falling foul of the stack issue before it had a chance to exhaust heap space. With the stack issue dealt with a I get a memory exhaustion problem that can be fixed by tweaking the code to FdoXmlFeatureReaderImpl::ReadNext so that it releases features which aren't needed any more.

I'm not quite sure what I'm going to do with this, but if I wanted to submit a patch for consideration for the next release of FDO, what's the etiquette? Is this something I should ask about on the 'internals' list?

Thanks, Robin