Maestro shp file load cause OutOfMemory Exception

[captainst]

When trying to load a shp file @ 300MB (using SHP Load Procedure), Maestro shows up an "OutOfMemoryException". The computer has 3GB of physical memory, and 5GB of virtual memory. Is there a work-around on this? Many Thanks!

[jng]

1. Does that OutOfMemoryException come with a stack trace?

2. Is it possible to use a different file format like SDF or SQLite? You can convert SHP files to SDF or SQLite with FDO Toolbox (​http://fdotoolbox.googlecode.com)

[jng]

3. Have you considered using aliases? You don't have to actually "load" a SHP file this way. Instead you copy the SHP file into a directory that has been set up as a MapGuide alias, and create a SHP feature source from that aliased path.

[captainst]

Greately Thanks to you jng!

I have attached the OutOfMemory log message in the .txt format below:

I am quite new to the MapGuide. I'll try out your suggestions 2, and 3.

The log messages: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

at System.IO.MemoryStream.set_Capacity(Int32 value) at System.IO.MemoryStream.EnsureCapacity(Int32 value) at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at OSGeo.MapGuide.MaestroAPI.Utility.CopyStream(Stream source, Stream target, StreamCopyProgressDelegate callback, Int64 updateFrequence) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI\Utility.cs:line 245 at OSGeo.MapGuide.MaestroAPI.RequestBuilder.AppendFormContent(String name, String filename, String boundary, Stream responseStream, Stream dataStream, StreamCopyProgressDelegate callback) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI.Http\RequestBuilder.cs:line 401 at OSGeo.MapGuide.MaestroAPI.RequestBuilder.SetResourceData(String id, String dataname, ResourceDataType datatype, Stream outStream, Stream content, StreamCopyProgressDelegate callback) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI.Http\RequestBuilder.cs:line 458 at OSGeo.MapGuide.MaestroAPI.HttpServerConnection.SetResourceData(String resourceid, String dataname, ResourceDataType datatype, Stream stream, StreamCopyProgressDelegate callback) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI.Http\HttpServerConnection.cs:line 431 at OSGeo.MapGuide.MaestroAPI.PlatformConnectionBase.SetResourceData(String resourceid, String dataname, ResourceDataType datatype, Stream stream) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI\PlatformConnectionBase.cs:line 1197 at OSGeo.MapGuide.MaestroAPI.Commands.ExecuteLoadProcedure.ExecuteShpLoadProcedure(LengthyOperationProgressCallBack cb, IShpLoadProcedure shpl, Boolean& firstExecution) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI\Commands\ExecuteLoadProcedure.cs:line 306 at OSGeo.MapGuide.MaestroAPI.Commands.ExecuteLoadProcedure.Execute(ILoadProcedure proc, LengthyOperationProgressCallBack callback) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI\Commands\ExecuteLoadProcedure.cs:line 176 at OSGeo.MapGuide.MaestroAPI.PlatformConnectionBase.ExecuteLoadProcedure(ILoadProcedure loadProc, LengthyOperationProgressCallBack callback, Boolean ignoreUnsupported) in d:\Workspace\maestro-trunk\OSGeo.MapGuide.MaestroAPI\PlatformConnectionBase.cs:line 1842 at Maestro.Editors.LoadProcedure.LoadProcedureEditorCtrl.ExecuteLoadProcedure(BackgroundWorker worker, DoWorkEventArgs e, Object[] args) in d:\Workspace\maestro-trunk\Maestro.Editors\LoadProcedure\LoadProcedureEditorCtrl.cs:line 147 at Maestro.Shared.UI.ProgressDialog.BackgroundWorker_DoWork(Object sender, DoWorkEventArgs e) in d:\Workspace\maestro-trunk\Maestro.Shared.UI\ProgressDialog.cs:line 89 at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e) at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

[captainst]

the SDF file converted from the 300MB SHP file using FDO2FDO still provoke the System.OutOfMemoryException, with almost the same error log messages.

[jng]

I think I know what the problem is. We're trying to create a temp. 300mb memory stream for the purposes of uploading, but there isn't a 300mb contiguous region to allocate hence the OutOfMemoryException.

I'll take a look to see if we can avoid unnecessarily creating such temp memory streams where possible.

[captainst]

Again, thank you very much for the quick support.

I was watching the task manager while clicking the "Load Resource" button. The system requested memory start @ 383M, which arrives @ 657M when the exception raises.

I am using the Italy shp file "roads.shp" downloaded from ​http://download.geofabrik.de/osm/europe/ The file size is roughly 316MB.

I searched a bit on google before creating this ticket while did not find anybody else raising this question. I was wondering if I have any wrong settings/setups.

[jng]

The problem I believe, is that Maestro is taking a very memory inefficient code path that it should not have to take.

Thanks for this link, it will give me plenty to work with.

[jng]

I've had some thoughts about whether this "bug" is actually worth fixing, and I've come to the conclusion that it isn't.

The simple reason is that uploading something as large as OSM SHP files that range in several hundreds of MB in size is not what I believe is the standard use case for Load Procedures.

Aliases is a feature of MapGuide that is suited for this purpose and is something you should look at. Aliases can be created by using the MapGuide Site Administrator web application, or by editing serverconfig.ini in your MapGuide Server installation.

Once you have created an alias, copy your OSM shape files into this aliased directory and create a SHP feature source that references this aliased path

[captainst]

I agree with this solution.

I have tried several available options provided by Maestro, including alias, Unmanaged file, and datasource connection. It is very benificial to have Mapguide interface with the external sources rather than importing all the data inside Mapguide repositories.

A very simple reason is that when you update the external datasource, the work already done remains untouched, especially for layer styling.