1.2.0 Beta 2, GDAL and Server Crashes

[amorsell]

MapGuide 1.2.0 beta 2. A GDAL based layer referencing a folder of 850 ECW image tiles. This actually reasonably well and is fast enough when not running the server in interactive mode. But, it frequently crashes the server leaving it in a completely unresponsive state. I can simulate this by running two browser sessions with the same map and zooming and panning within the same area. Sometimes individual images don't render, and sometimes a complete lockup occurs. This is not a consistent problem with an one image. The console yields the following:

FIRST CRASH---------

ECWDataset: RasterIO(0,1476,1566,4762 -> 566x797) - doing interleaved read. ERROR 1: NCScbmReadViewLineBIL failed. ECWDataset: RasterIO(9043,2764,957,3175 -> 565x798) - doing interleaved read. ECWDataset: RasterIO(0,2764,1295,3175 -> 565x798) - doing interleaved read. (5940|5912) An unclassified exception occurred. Exception occurred in method MgServerRenderingService.RenderDynamicOverlay at li ne 283 in file c:\build_bond_area\mapguide_open_source_v1.2\build_13.7\mgdev\ser ver\src\services\rendering\ServerRenderingService.cpp

(5940|6036) An unclassified exception occurred. Exception occurred in method MgApplicationResourceContentManager.EnumerateParent MapDefinitions at line 214 in file c:\build_bond_area\mapguide_open_source_v1.2\ build_13.7\mgdev\server\src\services\resource\ApplicationResourceContentManager. cpp

With this message then repeated over and over until restart

SECOND CRASH------------

ECWDataset: RasterIO(0,0,8996,1557 -> 670x452) - doing interleaved read. ECWDataset: RasterIO(0,750,8996,9250 -> 1021x689) - doing interleaved read. (5544|6064) An unclassified exception occurred. Exception occurred in method MgApplicationResourceContentManager.EnumerateParent MapDefinitions at line 214 in file c:\build_bond_area\mapguide_open_source_v1.2\ build_13.7\mgdev\server\src\services\resource\ApplicationResourceContentManager. cpp

(5544|3832) An unclassified exception occurred. Exception occurred in method MgApplicationResourceContentManager.SetResourceTags

at line 369 in file c:\build_bond_area\mapguide_open_source_v1.2\build_13.7\mgd

ev\server\src\services\resource\ApplicationResourceContentManager.cpp

(5544|6064) An unclassified exception occurred. Exception occurred in method MgApplicationResourceContentManager.EnumerateParent MapDefinitions at line 214 in file c:\build_bond_area\mapguide_open_source_v1.2\ build_13.7\mgdev\server\src\services\resource\ApplicationResourceContentManager. cpp

With this message then repeated over and over until restart

The error.log file yields the following:

<2007-06-02T18:41:09> Anonymous

Error: Failed to stylize layer: Austin_2006_aerials_ECW

An unclassified exception occurred.

StackTrace:

• MgStylizationUtil.StylizeLayers line 923 file c:\build_bond_area\mapguide_open_source_v1.2\build_13.7\mgdev\server\src\services\mapping\StylizationUtil.cpp Failed to stylize layer: Austin_2006_aerials_ECW

An unclassified exception occurred.

I can provide the development team full access to this server via remote desktop if desired.

[amorsell]

In addition, Windows Event Viewer has the following entry:

Event Type: Error Event Source: Application Error Event Category: (100) Event ID: 1000 Date: 6/2/2007 Time: 6:13:55 PM User: N/A Computer: SAGE Description: Faulting application mgserver.exe, version 1.2.0.1307, faulting module ntdll.dll, version 5.2.3790.3959, fault address 0x000290b9.

For more information, see Help and Support Center at ​http://go.microsoft.com/fwlink/events.asp. Data: 0000: 41 70 70 6c 69 63 61 74 Applicat 0008: 69 6f 6e 20 46 61 69 6c ion Fail 0010: 75 72 65 20 20 6d 67 73 ure mgs 0018: 65 72 76 65 72 2e 65 78 erver.ex 0020: 65 20 31 2e 32 2e 30 2e e 1.2.0. 0028: 31 33 30 37 20 69 6e 20 1307 in 0030: 6e 74 64 6c 6c 2e 64 6c ntdll.dl 0038: 6c 20 35 2e 32 2e 33 37 l 5.2.37 0040: 39 30 2e 33 39 35 39 20 90.3959 0048: 61 74 20 6f 66 66 73 65 at offse 0050: 74 20 30 30 30 32 39 30 t 000290 0058: 62 39 b9

[amorsell]

Here is the result of another test as I continue to hammer on this application

---

ECWDataset: RasterIO(0,0,10000,10000 -> 427x288) - doing interleaved read. (5020|3420) An unclassified exception occurred. Exception occurred in method MgFdoConnectionManager.RemoveExpiredConnections at line 504 in file c:\build_bond_area\mapguide_open_source_v1.2\build_13.7\mgdev\s erver\src\common\manager\FdoConnectionManager.cpp

(5020|3912) An unclassified exception occurred. Exception occurred in method MgOpRenderDynamicOverlay.Execute at line 93 in file

c:\build_bond_area\mapguide_open_source_v1.2\build_13.7\mgdev\server\src\servic

es\rendering\OpRenderDynamicOverlay.cpp

GDAL: GDALOpen(D:\GIS_Data\City_of_Austin\2006Aerials\ECW/145075.ecw) called. GDAL: Now trying ECW driver. ECW: OpenECW(D:\GIS_Data\City_of_Austin\2006Aerials\ECW/145075.ecw) ECW: Open(D:\GIS_Data\City_of_Austin\2006Aerials\ECW/145075.ecw) (5020|992) An unclassified exception occurred. Exception occurred in method MgOpRenderDynamicOverlay.Execute at line 93 in file

c:\build_bond_area\mapguide_open_source_v1.2\build_13.7\mgdev\server\src\servic

es\rendering\OpRenderDynamicOverlay.cpp

(5020|4220) An unclassified exception occurred. Exception occurred in method MgApplicationResourceContentManager.EnumerateParent MapDefinitions at line 214 in file c:\build_bond_area\mapguide_open_source_v1.2\ build_13.7\mgdev\server\src\services\resource\ApplicationResourceContentManager. cpp

(5020|4220) An unclassified exception occurred. Exception occurred in method MgApplicationResourceContentManager.EnumerateParent MapDefinitions at line 214 in file c:\build_bond_area\mapguide_open_source_v1.2\ build_13.7\mgdev\server\src\services\resource\ApplicationResourceContentManager. cpp

(5020|5016) Application terminated abnormally. Unknown exception. (5020|5016) Server stopped.

From event viewer:

Event Type: Error Event Source: Application Error Event Category: (100) Event ID: 1000 Date: 6/4/2007 Time: 11:12:48 AM User: N/A Computer: SAGE Description: Faulting application mgserver.exe, version 1.2.0.1307, faulting module unknown, version 0.0.0.0, fault address 0xec5d89df.

For more information, see Help and Support Center at ​http://go.microsoft.com/fwlink/events.asp. Data: 0000: 41 70 70 6c 69 63 61 74 Applicat 0008: 69 6f 6e 20 46 61 69 6c ion Fail 0010: 75 72 65 20 20 6d 67 73 ure mgs 0018: 65 72 76 65 72 2e 65 78 erver.ex 0020: 65 20 31 2e 32 2e 30 2e e 1.2.0. 0028: 31 33 30 37 20 69 6e 20 1307 in 0030: 75 6e 6b 6e 6f 77 6e 20 unknown 0038: 30 2e 30 2e 30 2e 30 20 0.0.0.0 0040: 61 74 20 6f 66 66 73 65 at offse 0048: 74 20 65 63 35 64 38 39 t ec5d89 0050: 64 66 df

[amorsell]

Added the OSGeo.Gdal.3.2 provider to the list of DataConnectionPoolExcludedProviders parameter in serverconfig.ini and it seems more stable after initial testing. I won't know for sure until larger scale testing.

[amorsell]

Still experiencing this problem with RC2. Excluding the provider from pooling does not appear to help as much as I had thought. I can still freeze the MGOS service by simply running two sessions that are accessing the same images. Bob Bray supplied this information on 6/4: "Sounds exactly like the symptoms I would expect when a multi-threaded application (e.g. MapGuide Server) accesses a non thread-safe FDO Provider"

[amorsell]

Correction: adding OSGeo.Gdal.3.2 provider to DataConnectionPoolExcludedProviders had no effect since this is not the agnostic provider name needed by serverconfig.ini and was being ignored. Adding OSGeo.Gdal to the list will cause the GDAL provider to become very unstable so it SHOULD NOT be added to DataConnectionExcludedProviders.

Newer versions of dll's from Jason Birch and Haris Kurtagic which includes image tile indexing appears to have greatly stabilized this. Only longer term testing in multiple environments will tell.

[warmerdam]

Andy,

If this is multi-threading related, then ​http://trac.osgeo.org/fdo/ticket/100 is the normative ticket. I am working on a patch based on Haris' mutex work for 3.2.x.

[jbirch]

Closing, based on improvements in 1.2 Release.