itemnquery limitation

[tmelhuish@…]

Daniel,

This is from the Postgis bug log. You request that we submit it to Mapserver 
Bugzilla.
++++++++++++++++++++++++++++++++++++++++++++++++
Hard to tell what's wrong without being able to run this in the 
debugger myself.  The problem seems to appear here:

 > #19 0x08057006 in msReturnPage (msObj=0x80c2008,
 >     html=0x1 <Address 0x1 out of bounds>, mode=0, papszBuffer=0x0)
 >     at maptemplate.c:2259
 > #20 0x080577e7 in msReturnQuery (msObj=0x80c2008,
 >     pszMimeType=0x80a83d1 "text/html", papszBuffer=0x0) at
 > maptemplate.c:2549


msReturnPage() is passed 'html=0x1' ... this should be the HTML 
template, so it should be a pointer to a valid string buffer.

+++++++++++++++++++++++++++++++++++++++

Here is the mapserv command line:

http://192.168.119.25/cgi-bin/mapserv?
map=/var/www/html/mapserver/wc/WCms.map&MINX=2001272.875&MINY=643833.625&MAXX=2
156198.25&MAXY=771771.5625&mode=itemnquery&zoomsize=2&program=%2Fcgi-bin%
2Fmapserv&map_web_header=WCms_headerSch.html&map_web_footer=WCms_footerSch.html
&HeaderHtml=WCms_headerSch.html&FooterHtml=WCms_footerSch.html&map_web_imagepat
h=/var/www/html/tmp/&map_web_imageurl=/tmp/&qlayer=nodes&qitem=nodeid&qstring=%
22nodeid+in+%28413.0%2C430.0%2C431.0%2C432.0%2C433.0%2C434.0%2C436.1%2C450.0%
2C451.0%2C452.0%2C453.0%2C454.0%2C455.0%2C456.0%2C457.1%2C457.2%2C457.3%
2C457.4%2C458.1%2C458.2%2C434.7%2C434.6%2C434.5%2C459.0%2C434.4%2C434.3%
2C434.2%2C434.1%2C528.0%29%22&layer=gp_nodes&layer3=gp_nodes"

I installed Postgres 7.3.4, postgis 0.7.5 and the nightly CVS build of 
Mapserver. I'm running RH8.0 & Apache 2.0.40. David Blasby was not able to 
reproduce the problem and was running Apache 1.3. Somebody else was running 
Apache 2.0 and had the same problem as me.

I will send you the test.map, test.html, and the sql file to build the 
postgresql gis table directly to Daniel Morissette [morissette@dmsolutions.ca]

Hopefully you can reproduce the problem

Thanks,
Tom Melhuish

[tmelhuish@…]

Zip file containing the infor to recreate the problem in mapserv

[dmorissette]

I'll try to have a look ASAP but it won't be before a few days for sure.

[dmorissette]

Tom Melhuish wrote:

> Daniel,
>
> I check the bug list last night and didn't see any changes. Have you had a
> chance to look at this? My customer is trying to stick with a vanilla RedHat
> install of 8.0 or 9.0 which uses apache 2.0. We are going into production
> later this month and hoped to have this resolved.
>

The other day I tried loading your DB in my older PostGIS installation but it
wouldn't load, quite likely because my PostGIS was too old.  I have upgraded to
PostgreSQL 7.3.4 and PostGIS 0.7.5 tonight, then created db postgis_delme,
loaded the postgis.sql and your nodescomb_2003.sql in it.  Then I made a few
minor changes to the mapfile and when I try to run the test URL that you sent I
get the error below.  I know very little about PostgreSQL/PostGIS so I think
I'll need some help to be able to reproduce your crash.

Here is the error I get:

-----------------
msPOSTGISLayerWhichShapes(): Query error. Error executing POSTGIS SQL statement
(in FETCH ALL): DECLARE mycursor BINARY CURSOR FOR SELECT
nodeid::text,asbinary(force_collection(force_2d(the_geom)),'NDR'),OID::text from
nodescomb_2003 WHERE (nodeid in
(413.0,430.0,431.0,432.0,433.0,434.0,436.1,450.0,451.0,452.0,453.0,454.0,455.0,456.0,457.1,457.2,457.3,457.4,458.1,458.2,434.7,434.6,434.5,459.0,434.4,434.3,434.2,434.1,528.0))
and (the_geom && setSRID( 'BOX3D(2001272.875 643833.625,2156198.25
771771.5625)'::BOX3D,find_srid('','nodescomb_2003','the_geom') )) -server closed
the connection unexpectedly This probably means the server terminated abnormally
before or while processing the request. Error with POSTGIS data variable. You
specified ''.
Standard ways of specifiying are :
(1) 'geometry_column from geometry_table'
(2) 'geometry_column from (<sub query>) as foo using unique <column name> using
SRID=<srid#>'

Make sure you put in the 'using unique <column name>' and 'using SRID=#' clauses in.

For more help, please see http://postgis.refractions.net/documentation.php

Mappostgis.c - version of June 12/2003.
----------------

Any idea?

[tmelhuish@…]

Fix to Postgis query error

[dmorissette]

OK, that new file fixed my DB.  I am running Apache 2.x on RH 9 with mapserver
4.0.  When I run the test URL that you sent in a web browser then it goes into
an infinite loop apparently somewhere in the postgresql code.  When I run the
same request at the command line using "./mapserv QUERY_STRING=...." then it
passes and produces a map image in the tmp dir with the selected polygons
highlighted in grey as expected.  I tried running this query through Valgrind
and it doesn't report any memory error.

There is definitely something different happening when running under Apache 2.x
... I'll continue to investigate.

[dmorissette]

FYI here is the backtrace that I get when I connect to the mapserv process that
is in an infinite loop under Apache 2.x.  Any ideas Dave, Paul?

(gdb) bt
#0  0xffffe002 in ?? ()
#1  0x4206d09e in new_do_write () from /lib/tls/libc.so.6
#2  0x4206e28f in _IO_new_file_xsputn () from /lib/tls/libc.so.6
#3  0x42062262 in fputs () from /lib/tls/libc.so.6
#4  0x403d020b in PQsetNoticeProcessor () from /usr/lib/libpq.so.3
#5  0x403d1b72 in PQexec () from /usr/lib/libpq.so.3
#6  0x403d1067 in PQconsumeInput () from /usr/lib/libpq.so.3
#7  0x403d1964 in PQgetResult () from /usr/lib/libpq.so.3
#8  0x403d1a78 in PQexec () from /usr/lib/libpq.so.3
#9  0x08075c91 in msPOSTGISLayerGetShape (layer=0x80de980, shape=0x80c4fa8,
    record=3532753) at mappostgis.c:1151
#10 0x0805780e in msReturnQuery (msObj=0x80c4db0,
    pszMimeType=0x80a8cb1 "text/html", papszBuffer=0x0) at maptemplate.c:2500
#11 0x08052145 in msReturnTemplateQuery (msObj=0x80c4db0,
    pszMimeType=0x80a8cb1 "text/html", papszBuffer=0x0) at maptemplate.c:212
#12 0x0804fd65 in main (argc=372, argv=0xbffff124) at mapserv.c:1351
#13 0x42015574 in __libc_start_main () from /lib/tls/libc.so.6

[dblasby@…]

I have no idea what is causing this.  Other users had this problem and noticed
it works fine on the command line or under the apache 1x series, and only had
problems under apache 2x.  I tested under valgrind too, and didnt find any problems.

If this is a problem in libpq (the postgresql client library), then upgrading to
the latest possible libpq might help.  I havent seen any disscussion wrt libpq
and apache on the postgresql mailing list.

Also, you might want to actually create a new database ("CREATE DATABASE
<name>") and install postgis in it instead of using the postgis_undef.sql script.

I've been unable to reproduce the bug here.

[dmorissette]

OK, I compiled libpq with debug information and found out that the problems
happens when libpq's default notice handler writes to stderr.  I modified
mappostgis.c to set its own (empty) notice handler and then the same freeze
started happening in MapServer's msDebug() which also writes to stderr.

Conclusion: Apache doesn't like CGIs writing to stderr!  I don't understand
exactly why and still need to research that, but at least we have a workaround
in 2 steps:

1- Make sure MapServer is not configured with --enable-debug (so that msDebug()
is disabled)

2- Apply the following patch to mappostgis.c to set an empty postgres notice
handler:


diff -r1.27 mappostgis.c
124c124,126
< //void postresql_NOTICE_HANDLER(void *arg, const char *message);
---
> void postresql_NOTICE_HANDLER(void *arg, const char *message)
> {
> }
214c216
< //    PQsetNoticeProcessor(layerinfo->conn, postresql_NOTICE_HANDLER ,(void *)
layerinfo);
---
>       PQsetNoticeProcessor(layerinfo->conn, postresql_NOTICE_HANDLER ,(void *)
layerinfo);


Long term solution:

1- I will try to figure why Apache 2.0 doesn't like output to stderr and
implement whatever we need in msDebug() to make apache 2.x happy

2- David: can you please implement a notice handler in mappostgis.c that would
call msDebug() so that we have a single mechanism in MapServer for logging to
stderr.

[dmorissette]

This stderr issue is still an open Apache 2.x bug:
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=22030

Apache 2.x has been out for over a year I think.  It's surprising that nobody
had reported this problem before and that there is still no fix available.  Duh!

[dblasby@…]

Thats funny that the notice handler is the problem.  There used to be a notice
handler in mappostgis.c, but I took it out...  I'll put another one back in.

Arent CGI's supposed to write errors to stderr?

[dblasby@…]

mappostgis.c -- set notice handler to send message to msDebug().
COMMITED

[dmorissette]

I committed a few changes to MapServer 4.0.1 and 4.1 (CVS dev) to workaround the
Apache stderr bug, see bug 458 for all the details.

I also modified Dave's postgresql notice handler to send output only when
layer->debug is set.  Also committed it in both the 4.0 and 4.1 branches.

Tom, with those changes everything works fine for the test case that you sent
with both version 4.0.1 (to be released in the next day or so, this was the last
fix holding it) and 4.1 (CVS dev).  I'll mark the bug fixed, please confirm that
things work fine for you once you have a chance to upgrade.

[tmelhuish@…]

Testing CVS version

[dmorissette]

Did you rerun configure before compiling?  Did you try setting DEBUG FALSE in
your layer?

You should not get any output to stderr when you set DEBUG FALSE in your layer
with the new version.

[dmorissette]

FYI DEBUG is FALSE by default unless you explicitly set it to true in the layer
object in the mapfile.

[dblasby@…]

>> "Closing pre-existing portal "mycursor". 

You shouldnt be getting this message in the CVS version.  Could you verify that
you're not using an older version of mapserv?

You should see this code at line #1254::mappostgis.c

	query_result = PQexec(layerinfo->conn, "ROLLBACK");

[dmorissette]

Dave is referring to the mappostgis.c in the main development trunk (4.1).  It
is quite likely that Tom is using the stable 4.0 branch version (cvs branch-4-0)
which doesn't contain this ROLLBACK line.  

However both the main trunk and the branch-4-0 versions contain the notice
handler fix so we should not get this postgis notice with either version if the
layer doesn't have DEBUG TRUE.

[tmelhuish@…]

Yes, I see the line "query_result = PQexec(layerinfo->conn, "ROLLBACK");" in 
the mappostgis.c file. 

I'm having a problem re-compiling mapscript now. I see that I'm getting an 
error "mapscript_wrap.o  Error1" when I try to do a make on perl mapscript. I 
have the mapscript_wrap.c file in the mapscript/perl directory already that 
the README file refers to. I see this error in the archives was related to 
version 3.6.3 was do to a change in compiling with CC to C++.

I can try to download another CVS version in the morning to see if the problem 
is fixed.

tom   

[dmorissette]

It is probably safer to stick to the stable 4.0 branch unless you have good
reasons to use the 4.1 development version.  There are important development and
changes being made in 4.1 and it can be broken at times.

You can get the latest 4.0 source using 'cvs update -r branch-4-0'.

[tmelhuish@…]

I copied "mappostgis.c" from the 4.1 CVS version over to the 4.0 released 
version and recompiled it and it now works fine!!

Thanks again for all your help.
Tom Melhuish
WebInsights