Opened 20 months ago

Last modified 20 months ago

#2751 new defect

Error selecting more than 1000 features from SQLite FeatureSource

Reported by: gBecker Owned by: jng
Priority: high Milestone:
Component: Map Agent Version: 3.0.0
Severity: major Keywords:
Cc: External ID:

Description

When trying to select more than 1000 Features from a SQLite FeatureSource? an error occured and is logged in Error.log:

 StackTrace:
  - MgMappingUtil.StylizeLayers() line 893 file c:\working\build_area\mapguide\3.0.0\x64\mgdev\server\src\services\mapping\MappingUtil.cpp	
<2016-10-25T12:14:08> 	4876	Ajax Viewer	::1	Administrator
 Error: An exception occurred in FDO component.
        Error occurred in Feature Source (Library://FFM/CSOGIS/Daten/Sqlite/basisgis.FeatureSource): Expression tree is too large (maximum depth 1000) (Cause: , Root Cause: Expression tree is too large (maximum depth 1000))
 StackTrace:
  - MgFeatureServiceHandler.ProcessOperation() line 83 file c:\working\build_area\mapguide\3.0.0\x64\mgdev\server\src\services\feature\FeatureServiceHandler.cpp
  - MgOpSelectFeatures.Execute() line 142 file c:\working\build_area\mapguide\3.0.0\x64\mgdev\server\src\services\feature\OpSelectFeatures.cpp
  - MgServerFeatureService.SelectFeatures(Resource=Library://FFM/CSOGIS/Daten/Sqlite/basisgis.FeatureSource	ClassName=Default:KS31101_Gebaeude	
Options={{ClassProps={{KGF_WERT}{Geometrie}}}{Operator=1}{Filter=(PRIMARYINDEX=1928) OR (PRIMARYINDEX=2024) OR (PRIMARYINDEX=2190) OR (PRIMARYINDEX=2442) OR (PRIMARYINDEX=3281) OR 
...
(PRIMARYINDEX=19556) OR (PRIMARYINDEX=19568) OR (PRIMARYINDEX=19717) OR (PRIMARYINDEX=19918) OR (PRIMARYINDEX=19929) OR (PRIMARYINDE<2016-10-25T12:14:09> 	13036	Ajax Viewer	::1	Administrator
 Warning: An exception occurred in FDO component.
        String does not represent a valid filter. 
 StackTrace:
  - MgStylizationUtil.ExceptionTrap() line 751 file GeometryAdapter.cpp	
<2016-10-25T12:14:09> 	13036	Ajax Viewer	::1	Administrator
 Warning: An exception occurred in FDO component.
        String does not represent a valid filter.

This error leads to an terminated MapGuideAppPool?. You have to restart the application pool.

Application Error in Windows application.Log:

Name der fehlerhaften Anwendung: w3wp.exe, Version: 10.0.14393.0, Zeitstempel: 0x57899b8a
Name des fehlerhaften Moduls: isapi_MapAgent.dll, Version: 3.0.0.8701, Zeitstempel: 0x559fd022
Ausnahmecode: 0xc0000409
Fehleroffset: 0x0000000000009060
ID des fehlerhaften Prozesses: 0x3214
Startzeit der fehlerhaften Anwendung: 0x01d22ea886d41d6f
Pfad der fehlerhaften Anwendung: c:\windows\system32\inetsrv\w3wp.exe
Pfad des fehlerhaften Moduls: \\?\C:\Program Files\OSGeo\MapGuide3.0\Web\www\mapagent\isapi_MapAgent.dll
Berichtskennung: 345f7d37-abe7-4ef2-bb64-1e745f1d4e02

Attachments (3)

HtmlController.cpp.patch (11.4 KB) - added by jng 20 months ago.
MgMapGuideCommon.zip (394.4 KB) - added by jng 20 months ago.
Patched MgMapGuideCommon?.dll for MGOS 3.1 64-bit
MgMapGuideCommon_3.0.zip (385.2 KB) - added by jng 20 months ago.
Patched MgMapGuideCommon?.dll for MGOS 3.0 64-bit

Download all attachments as: .zip

Change History (12)

comment:1 Changed 20 months ago by jng

Is this a feature query from a MgFeatureService?.SelectFeatures? API call or from a map tool selection?

Is your filter literally 1000s of PRIMARYKEY=value conditions OR'd together, or is that from a MgSelection??

I ask because I know MgSelection? has the ability to sub-divide filters into manageable parts, but I'm not sure that SelectFeatures? has the same ability.

comment:2 Changed 20 months ago by gBecker

The feature query is from the default selection tool (rectangle selection) of the AjaxViewer?. Apparently 1000 "PRIMARYKEY=value" conditions are concatenated by "OR". The error message in the logfile is truncated at around 200 conditions.

comment:3 Changed 20 months ago by jng

Okay, I see where this is happening (building the attributes of the QUERYMAPFEATURES response).

We need to sub-divide the selection set filter to a reasonable number and incrementally write out the attributes from all the sub-divided feature readers.

comment:4 Changed 20 months ago by jng

Owner: set to jng

Changed 20 months ago by jng

Attachment: HtmlController.cpp.patch added

comment:5 Changed 20 months ago by jng

Firstly, does this happen with MGOS 3.1 as well?

Secondly, if the above answer is yes, I'm going to build and attach a patched MgMapGuideCommon?.dll for MGOS 3.1 64-bit that incorporates the attached source code changes. Could you check if this dll fixes the issue in 3.1? If it does, I'll backport this fix for 3.0 and 2.6 as well.

Changed 20 months ago by jng

Attachment: MgMapGuideCommon.zip added

Patched MgMapGuideCommon?.dll for MGOS 3.1 64-bit

comment:6 Changed 20 months ago by gBecker

In MGOS 3.1 the selection issue doesn't happen (without the patched MgMapGuideCommon?.dll). I can select up to approxamately 15.000 features. When selecting more features at once, then there seems to be another limit, but luckily without terminating the IIS AppliationPool?. The selection layer isn't updated anymore.

But, you can select even more features while pressing the Ctrl-Button and selecting a smaller amount of features. When selecting approxemately 25.000 features another odd behaviour occurs. The selection layer only shows the last added features while the number of selected features in the statusbar increaeses.

These behaviours aren't restricted to SQLite FeatureSources?. This also belongs to Shapes oder SDF FeatureSources?. This is happening in MGOS 3.0 and 3.1.

I believe that this has nothing to do with the initial issue. Maybe this is more a problem of the SQLite FDO Provider (4.0) in MGOS 3.0 and was fixed in MGOS 3.1?

comment:7 Changed 20 months ago by jng

Hmmm, I see no changes in the SQLite FDO provider that could indicate a fix at the FDO provider level. I'll make a patched MgMapGuideCommon?.dll for 3.0 and attach that here.

As for the strange selection behaviour when you hold the CTRL key, I backported the selection appending fix (r9036). I just haven't made a 3.0.1 release yet that incorporates this fix.

Changed 20 months ago by jng

Attachment: MgMapGuideCommon_3.0.zip added

Patched MgMapGuideCommon?.dll for MGOS 3.0 64-bit

comment:8 Changed 20 months ago by jng

Can you try this MgMapGuideCommon?.dll for your 3.0 install and let me know how it goes?

comment:9 Changed 20 months ago by gBecker

Unfortunately, there is no difference. Selecting the 1,000th feature makes the application pool stop working.

Note: See TracTickets for help on using tickets.