Thread safety issues with .map file when using Java Mapscript in a multithreaded environment

[miller_joseph@…]

When using Java Mapscript in a Servlet in Tomcat using Struts as my MVC:

The symptom I see is that I can generate the default image but after the
fourth or fifth time that I pass a new envelope to change the extent of
the image I get the following error:

The relevant part of the stack trace:
java.lang.UnknownError: Failed to draw layer named 'world'.
        at edu.umn.gis.mapscript.mapscriptJNI.mapObj_draw(Native Method)
        at edu.umn.gis.mapscript.mapObj.draw(mapObj.java:397)

Occasionally I'll get a flex scanner error or tomcat will crash with an
Exception_Access_Violation memmove error similar to the one referred to
here:
http://forum.java.sun.com/thread.jspa?forumID=52&messageID=1124599&threa
dID=286832

All of this happens whether I instatiate new instances of the mapObj and
set everything to null or if I use the same mapObj throughout and only
after a few passes, suggesting to me that there is a memory issue
somewhere.  I can pass on some specific code, but all I am really doing
is creating new envelopes to store extents and then assigning them to
the mapObj or extracting the corner coordinates and setting the extent
with them.

I was finally able to get it to work by using a suggestion to strategically 
place "synchronized" statements to lock down resources for threads.  
Specifically I locked down the mapObj during map creation and when running a 
draw operation and based on what Mario and Umberto said, I locked down the 
String variable that was storing the path to my mapfile.
 
This last one took me a long time to figure out and I would like to suggest 
that the next version of the Swig Mapscript include a mapObj constructor that 
takes in a file object (rather than a string referring to the path) or 
something like this so that the file itself can be locked down by the thread 
while parsing occurs (it looks like parsing occurs when the mapObj is 
instantiated and when it draws to an image??).

[sgillies@…]

Adding Jerry

[jerry.pisk@…]

I was able to narrow this quite a bit. There's places where the parser is not 
synchronized (bug #1206 and possibly more). I was also able to narrow down the 
drawing issue, the problem happens when shape geometry is accessed in the 
clipping function. I do not yet know whether the clipping function is the 
problem (it seems that it allocates less memory then needed) or whether it 
crashes because it's accessing memory that was freed before it got to it (i.e. 
some other function freed or misallocated the shape geometry memory block). I 
will do some more research on this next week.

There are also issues with postgis (and other database connections using 
mappool) and the "connection pool" in the 4.4.1 release, postgres connections 
are shared between threads and no effort is made to ensure that eachc 
onnection can only be used by a single thread at any time. The fix in the HEAD 
branch does not address this as it simply creates a TLS for connection objects 
that are still used multiple times per thread. The mappool implementation is 
really more like a TLS, not a connection pool, the name is really misleading.

[jerry.pisk@…]

Ad for the suggestion for mapObj constructor - if anything there should be one 
taking Stream object, not file object. But I don't understand why you're 
locking down the string object holding the path to the map file, you don't 
have to do that. The current HEAD revision of mapserver and mapscript will run 
without any problems if you don't draw the map, I got a multithreaded Java 
code that loaded a map file, enumerated layers and turned some of them on and 
queried some layers by points, to run without crashing and without any 
locking. But you will probably never be able to share your map objects between 
threads, each thread has to have its own set of objects.

[sgillies@…]

I'm setting the target of this one to 4.6 and cc'ing Frank to get clarification.
whether recent changes in the clipping code are improving the situation that
Jerry mentions in the first paragraph of comment 2.

I am also very uncomfortable with the lack of resolution and agreement between
Frank and Jerry on the impact of 4.4's connection "pooling" for Java mapscript.

Jerry, can you please come up with some tests that demonstrate the problem?

Frank, will you consider some documentation that explains the difference 
between your implementation and the narrower "pool" that Jerry insists is
necessary?  
 
Thanks. The MapServer project really needs this to be resolved!

[fwarmerdam]

I don't think I have made any relavent changes to the clipping code.  I just
short circuited one do-nothing case.  I don't see any obvious issues in the
clipping code so I think Jerry will will need to track this down further, or
provide a reproducable test case we can pursue practically.

The patch related to the raster expression parsing was commited in 4.5, but
not 4.4.2.  I could back port it if needed, but it is a bit scary.  

It isn't at all clear to me that pooling issues come into play for this bug 
report, but then Joseph was very non-specific about what MapServer services
are in use. 

I agree with the conclusion that at this time applications should not attempt
to share mapObj instances between threads.  Synhronizing all rendering 
operations should make things safer, but this should not be necessary if we 
fix all internal sync. bugs.  

[miller_joseph@…]

Java Mapscript mapObj bean

[miller_joseph@…]

In response to the question about what services were in use when I ran into 
these problems, I was testing client functionality with a wide variety of data 
sources including standalone shapefiles, PostGIS vector data, and WMS services.  
I isolated each source of data in its own mapfile and saw the most problems with 
with my PostGIS data.  Does this lend weight to the connection pooling theory?  

Also FYI, I locked down the mapfile path as a way to make sure that only one 
thread at a time was parsing the mapfile.

[fwarmerdam]

Joseph, 

Yes, that you had the most problems with PostGIS does suggest a pooling
issue, though other issues could be in play as well.  Are you using 4.4.1? 
Would you be willing to try 4.5 to see if the locking changes made there 
in the pooling area help? 

[sgillies@…]

reassigning.

[unicoletti]

Postgis, Oracle and WxS features are considered safe to use in 5.0. Closing.