Mapserver not working in fcgi mode and external spawning manager

[gehasia]

Hi, i did some test with mapserver as fcgi + spawn-fcgi + nginx.

The problem is that in the case mapserver generates an error (invalid mapfile syntax or invalid mapfile path for example) the external fcgi process kill himself ! It should just send the error and keep running...

Here is my test case and result :

Nginx + spawn-fcgi + mapserver :

Calling valid mapfile like this :

 http://localhost/cgi-bin/mapserv.fcgi?map=/var/mapfile.map&mod=map

= OK - Process Remains

Calling invalid mapfile like this :

 http://localhost/cgi-bin/mapserv.fcgi?map=/var/mpfile.map&mod=map

(in this case mpfile.map does not exists) = NO OK = Throw an error but then terminate the fcgi process !! So no more fcgi process running and all others queries, valid or invalid, won't work !

Apache + fcgid (or fastcgi) + mapserver

= OK for both case, but this is the apache fcgi manager which relaunch the fcgi process as i can see. This is the installation documented everywhere

Apache + spawn-fcgi + mapserver (fcgi is called as an external process, communication done via tcp or socket)

Same results than with Nginx in first case. Valid url = OK, invalid mapfile or any other error (error in mapfile syntax ) = fcgi process get killed

So i did some test with another well known fast-cgi aka PHP ...

Nginx + spawn-fcgi + php5-cgi

Valid or invalid php file (ie : throw errors) keep the process running, so normal behavior

Apache + spawn-fcgi + php5-cgi

Valid or invalid php file (ie : throw errors) keep the process running, so normal behavior

The php5 cgi has a normal comportment, it throws error and keep running. The mapserver throws error and kill himself.

I did some debug with fcgi-debug tool and here is the result if we send an invalid command to the mapserver cgi :

vnode004:/usr/local/nginx/conf# spawn-fcgi -s /var/run/mapserver.socket -n -u nobody -- /usr/local/bin/fcgi-debug /usr/local/bin/mapserv
new connection (0)                                                                                                                      
begin request from webserver (0, 1): role: FCGI_RESPONDER, flags: none                                                                  param from webserver (0, 1): 'QUERY_STRING' = ''                                                                                        
param from webserver (0, 1): 'REQUEST_METHOD' = 'GET'                                                                                   
param from webserver (0, 1): 'CONTENT_TYPE' = ''                                                                                        
param from webserver (0, 1): 'CONTENT_LENGTH' = ''                                                                                      
param from webserver (0, 1): 'SCRIPT_NAME' = '/cgi-bin/mapserv'                                                                         
param from webserver (0, 1): 'REQUEST_URI' = '/cgi-bin/mapserv'                                                                         
param from webserver (0, 1): 'DOCUMENT_URI' = '/cgi-bin/mapserv'                                                                        
param from webserver (0, 1): 'DOCUMENT_ROOT' = '/usr/local/nginx/html'                                                                  
param from webserver (0, 1): 'SERVER_PROTOCOL' = 'HTTP/1.1'                                                                             
param from webserver (0, 1): 'GATEWAY_INTERFACE' = 'CGI/1.1'                                                                            
param from webserver (0, 1): 'SERVER_SOFTWARE' = 'nginx/0.7.61'                                                                         
param from webserver (0, 1): 'REMOTE_ADDR' = '90.45.64.64'                                                                              
param from webserver (0, 1): 'REMOTE_PORT' = '32938'                                                                                    
param from webserver (0, 1): 'SERVER_ADDR' = '192.168.100.4'                                                                            
param from webserver (0, 1): 'SERVER_PORT' = '8000'                                                                                     
param from webserver (0, 1): 'SERVER_NAME' = 'mapserver.connecting-nature.com'                                                          
param from webserver (0, 1): 'REDIRECT_STATUS' = '200'                                                                                  
param from webserver (0, 1): 'SCRIPT_FILENAME' = '/usr/local/nginx/html/cgi-bin/mapserv'                                                
param from webserver (0, 1): 'HTTP_HOST' = 'mapserver.connecting-nature.com:8000'                                                       
param from webserver (0, 1): 'HTTP_USER_AGENT' = 'Mozilla/5.0 (X11; U; Linux i686 (x86_64); fr; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1'
param from webserver (0, 1): 'HTTP_ACCEPT' = 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'                          
param from webserver (0, 1): 'HTTP_ACCEPT_LANGUAGE' = 'fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3'                                             
param from webserver (0, 1): 'HTTP_ACCEPT_ENCODING' = 'gzip,deflate'                                                                    
param from webserver (0, 1): 'HTTP_ACCEPT_CHARSET' = 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'                                                   
param from webserver (0, 1): 'HTTP_KEEP_ALIVE' = '300'                                                                                  
param from webserver (0, 1): 'HTTP_CONNECTION' = 'keep-alive'                                                                           
params end from webserver (0, 1)                                                                                                        
stdin closed from webserver (0, 1)                                                                                                      
stdout from application (0, 1): Content-type: text/html\n                                                                               
stdout from application (0, 1): \n                                                                                                      
stdout from application (0, 1): No query information to decode. QUERY_STRING is set, but empty.\n                                       stdout closed from application (0, 1)                                                                                                   
end request from application (0, 1): applicationStatus: 0, protocolStatus: FCGI_REQUEST_COMPLETE                                        connection closed (0)                                                                                                                   
** Message: process 5975 exited with status 256                                                                                         
** Message: exit fcgi-debug 

As you can see, there is a

Message: process 5975 exited with status 256

but the fcgi did send the html error message correctly (and after that, died)

in a php-cgi session, you have the message but the process does not exit after (it just continues to listen).

I don't think this is a normal behavior, as with my different test case i can conclude it's not the nginx (because it does the same with apache) or spawn-fcgi (because it runs well php5-cgi -with nginx and apache-) fault. Why mapserver kill himself when it throws an error ?? Is it not the mapserver fcgi role to keep himself running and spawn fork or keep them clean ?

I can add more test case or debug info if needed ! (and i hope i do not make a mistake on who, in the fcgi world, has to keep up running and spawning process)

This affect every version i tried (Debian binary, stable source, trunk source)

[gehasia]

I join a log of Apache + mod_fcgid + mapserver

[Mon Aug 17 23:11:59 2009] [warn] mod_fcgid: cleanup zombie process 7488
[Mon Aug 17 23:12:05 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7491) exit(server exited), terminated by calling exit(), return code: 0
[Mon Aug 17 23:12:05 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7490) exit(server exited), terminated by calling exit(), return code: 0
[Mon Aug 17 23:12:11 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7493) exit(server exited), terminated by calling exit(), return code: 0
[Mon Aug 17 23:12:11 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7492) exit(server exited), terminated by calling exit(), return code: 0
[Mon Aug 17 23:12:21 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7494) exit(normal exit), terminated by calling exit(), return code: 0
[Mon Aug 17 23:12:21 2009] [warn] mod_fcgid: cleanup zombie process 7494
[Mon Aug 17 23:15:27 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7496) exit(normal exit), terminated by calling exit(), return code: 0
[Mon Aug 17 23:15:27 2009] [warn] mod_fcgid: cleanup zombie process 7496
[Mon Aug 17 23:28:04 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7506) exit(server exited), terminated by calling exit(), return code: 0
[Mon Aug 17 23:28:08 2009] [notice] mod_fcgid: process /usr/lib/cgi-bin/mapserv.fcgi(7507) exit(server exited), terminated by calling exit(), return code: 0
[Mon Aug 17 23:28:26 2009] [error] [client 90.45.64.64] script not found or unable to stat: /usr/lib/cgi-bin/mapserv.cgi

As you can see when we hit an errorfull url (because the cgi can't load the mapfile for example) mapserver is calling exit(0); The process die (apache logs it as "exit(normal exit), return code : 0" ) and then mod_fcgid has to cleanup the zombie process (!) and restart a new one. It's lot of crappy work for mod_fcgid i think, maybe lots of those exit(0) has to be in pre-processor definition like this ?

#ifndef USE_FASTCGI
exit(0)
#endif

I'm not an C nor FCGI developer (or expert), but i think the process has to exit in standard CGI mode, but not in FastCGI (as process is running as a background daemon), but maybe i'm wrong.

[pmauduit]

I was able to reproduce the bug, and got this stack trace into gdb :

(gdb) break exit
Breakpoint 1 at 0xb5eb3304
(gdb) bt
#0  0xb777e424 in __kernel_vsyscall ()
#1  0xb6937281 in accept () from /lib/i686/cmov/libpthread.so.0
#2  0xb60ec254 in OS_Accept () from /usr/lib/libfcgi.so.0
#3  0xb60ea1bc in FCGX_Accept_r () from /usr/lib/libfcgi.so.0
#4  0xb60ea2e3 in FCGX_Accept () from /usr/lib/libfcgi.so.0
#5  0xb60ebf20 in FCGI_Accept () from /usr/lib/libfcgi.so.0
#6  0x08088ac4 in main (argc=1, argv=0xbfd39004) at mapserv.c:1202

Is it then possible that the current issue is beyond the scope of mapserver ? I'm not a fastcgi / C expert either, but it seems that the exit() call is triggered into fastcgi library code, and not into mapserver.

[pmauduit]

Sorry, I spoke a bit too fast in my previous comment ; there was another process alive after the first call to exit() I traced.

I figured out that the function loadParams from cgiutil.c did a lot of another calls to exit(), although this is the role of the function mapserv.c:main() to call exit() when necessary (mapserv.c, around line 1219), if I understood correctly.

Please find attached a patch, which seems to solve the current issue. Anyway, since I'm pretty new to mapserver codebase, I don't know if I did it in a good way. Comments are welcome.

Cheers,

[pmauduit]

patch diffutil.c

[pmauduit]

Please find attached a patch, which seems to solve the current issue. Anyway, since I'm pretty new to mapserver codebase, I don't know if I did it in a good way. Comments are welcome.

More complicated than that, there are exit() calls from mapserv.c:writeError() to track down as well. Sorry for polluting the trac, i'll try to dig on the problem and come back when i'll be sure to have something relevant to give.

[warmerdam]

Skimming the patch and the code, it seems like the patch is pretty safe and will fix some problems, but that there are still cases it does not address. Would you like me to take care of applying it Steve?

[warmerdam]

I will add that it was already known that mapserv exits in some cases, and it was felt that letting this happen and letting apache start a new fastcgi was quite acceptable. I'm not all that keen on bending over backward to support other fastcgi technology but since we have a patch and it looks pretty safe we might as well apply it.

There will still be fatal errors that cause the fastcgi to terminate - so the original person really ought to find a way of restarting as needed.

[sdlime]

Should we wait pmauduit does a bit more leg work (as volunteered)? Or would be better to handle with subsequent patches...

Steve

[warmerdam]

Steve,

That sounds reasonable to me.

[pmauduit]

Hi again,

I sent a patch on the dev mailing-list yesterday, which actually was catastrophic (double-free, sigabrt ...) ; so I reworked my local modifications and did some better test cases (with a non-existent mapfile given in parameter, with a mapfile existing but with junk in the parameters, with a correct mapfile), I was able to get the process still working after each requests. And I actually tried a GetMap? with a correct mapfile, and achieved to get a resulting image.

So please find attached an update of my current work on this issue ; anyway I did not find any better solutions to track possible memory leaks than launching a "watch -n1 ps aux | grep mapserv", and see after some tests if the memory used was increasing (I've tried to launch valgrind on top of my spawn-fcgi process, but without success). I'm pretty sure that you have better solutions than that. And of course, my remarks concerning my own work made on the mailing-list did not change. I would be glad to improve it but I may need your support, comments, critics ... ;-)

Cheers,

-- Pierre

[pmauduit]

mapserv fastcgi current patch

[sdlime]

Frank, what do you think? -Steve

[warmerdam]

Steve,

I don't like the duplication of error formatting logic between writeError() and the new writeErrorWithoutExit().

I dislike the proliferation of USE_FASTCGI ifdefs.

I don't think the patch is comprehensive in catching errors, though I haven't reviewed this version "in context" well enough to be sure about that.

I'm not really keen enough on it to apply this patch. I just don't believe in the use of a fastcgi manager that doesn't know how to restart things. But if you like you can apply it.

[dmorissette]

While I agree with Frank that spawn-fcgi is flawed if it cannot restart aborted FCGI processes, especially in extreme situations such as processes running out of memory, I also hope that we can be more friendly of non-CGI environments and provide some ways to gracefully recover from non-fatal situations such as mapfile access/parsing errors.

I am also -1 on the last proposed patch... we need to find a cleaner solution, maybe through the use of configurable error handlers that could be used to trap error situations in non-CGI mode and would default to exit() in CGI mode? I don't have a solution to propose yet, just wishful thinking.

[tbonfort]

attached is a patch with some heavy refactoring of mapserv.c and a few dependencies

- remove usage of global variables - split main() into smaller subfunctions for legibility.

Instead of calls to exit(), child functions now return an error code, and it's up to main() to decide to print the error and eventually continue the fastcgi loop or not. The changes are widespread because exit() was being called from a number of small functions, namely getNumeric, writeError...

[tbonfort]

The refactoring in r12907 suppressed the calls to exit() (except for the ones in msSmallMalloc but that seems reasonable)