GDAL error message screws up HTTP headers

[dmorissette]

Frank,

We've encountered the problem described below a few times in the past where it
seems that GDAL writes an "ERROR X: ....." message to stderr (or stdout?) when a
raster in a tiled dataset is not found. This results in a server error if/when
the error message is produced before the HTTP headers are returned by MapServer.

I can't remember if a solution has been found to this, but I received the
following message today confirming that the problem was still present in 4.2.
Does this ring any bell to you?

Daniel

-------- Original Message --------
Subject: Re: Tiled rasters cause map to show source of HTMLtemplate
Date: Tue, 21 Sep 2004 23:16:51 +0200
From: Marek Topol <mara@nospam.....>
To: <morissette@dmsolutions.ca>

> > I've just started working to display my images (TIF topo maps) using a =
> > tiled index. When I try to load the map with one of the tiles in my view
=
> > extents, I see the following in place of the webpage, as plain text:
> >
> > Content-type: text/html

> We've seen this problem before but it's very hard to reproduce and we
> were never able to reproduce and track it down.  Is there any chance you
> could submit a bug or send us your whole mapfile + data to reproduce
> this?  We would really like to fix this one.

Hi,

I'm using Mapserver 4.2.0 with IIS and I had similar problem. When I sniffer
complete HTTP headers I found reason. Error message was written to HTTP
header.

Mapserver then ignore error and continues to generate content with
Content-Type on beginning, but it's not already in header bacause of
error message ended with double CRLF.

Header:

00000030                    48 54 54 50 2F 31 2E 31 20 32       HTTP/1.1.2
00000040  30 30 20 4F 4B 0D 0A 53 65 72 76 65 72 3A 20 4D 00.OK..Server:.M
00000050  69 63 72 6F 73 6F 66 74 2D 49 49 53 2F 35 2E 30 icrosoft-IIS/5.0
00000060  0D 0A 44 61 74 65 3A 20 54 75 65 2C 20 32 31 20 ..Date:.Tue,.21.
00000070  53 65 70 20 32 30 30 34 20 32 31 3A 30 30 3A 33 Sep.2004.21:00:3
00000080  37 20 47 4D 54 0D 0A 43 6F 6E 6E 65 63 74 69 6F 7.GMT..Connectio
00000090  6E 3A 20 63 6C 6F 73 65 0D 0A 43 6F 6E 74 65 6E n:.close..Conten
000000A0  74 2D 54 79 70 65 3A 20 61 70 70 6C 69 63 61 74 t-Type:.applicat
000000B0  69 6F 6E 2F 6F 63 74 65 74 2D 73 74 72 65 61 6D ion/octet-stream
000000C0  0D 0A 45 52 52 4F 52 20 34 3A 20 60 70 72 37 31 ..ERROR.4:.`pr71
000000D0  32 2E 74 69 66 27 20 64 6F 65 73 20 6E 6F 74 20 2.tif'.does.not.
000000E0  65 78 69 73 74 20 69 6E 20 74 68 65 20 66 69 6C exist.in.the.fil
000000F0  65 20 73 79 73 74 65 6D 2C 0D 0A 61 6E 64 20 69 e.system,..and.i
00000100  73 20 6E 6F 74 20 72 65 63 6F 67 6E 69 73 65 64 s.not.recognised
00000110  20 61 73 20 61 20 73 75 70 70 6F 72 74 65 64 20 .as.a.supported.
00000120  64 61 74 61 73 65 74 20 6E 61 6D 65 2E 0D 0A 0D dataset.name....
00000130  0A                                              .

I hope it will help.

Marek Topol

[mara@…]

Template

[mara@…]

MAP file

[fwarmerdam]

Daniel / Marek,

The error message is certainly familiar.  What isn't so obvious to me why
this would happen sometimes and not others.  Generally speaking error messages
from GDAL will go to stderr.  On Apache (linux anyways) this goes to the errors
log.  Is it possible that stderr effectively goes to the same place as stdout
on IIS?  If so, does this mean we need to be especially careful to not send
anything unplanned to stderr within MapServer?  

It wouldn't be that hard to substitute a GDAL (CPL) error handler within
MapServer that would redirect all GDAL errors through msSetError() or some
other mechanism if desired.  We can also ensure that open errors in mapraster.c
for tile indexes only go to msDebug() is IGNORE_MISSING_DATA is defined. 

But, I would like some assurance that this would resolve the problem.  If
so, we should also add a note somewhere to a "rules for MapServer developers".

I have no access to IIS to know if it has an unusual behaviour.  Can anyone
with experience on this comment?

[fwarmerdam]

Daniel, 

Where would you like to go with this?  I will need some pretty specific
information to reproduce and decide what should be done. 

As noted before, my assumption has been that writing to stderr is harmless
and at worst fills up the error_log.  If this is not the case I need to know,
and we might want to make it more broadly known amoung MapServer developers. 

[mara@…]

Hi Frank,

I hope I could help with reproduce this error. Your message means, that 
attached files doesn't help with problem. I will try to simplify map/data 
files and than attach package with command lines, which has this HTTP header 
problem. Does this files help you?

Marek Topol

[fwarmerdam]

Marek,

I am in no position to test with IIS which I presume is key to your
issue.  So, if the issue really is that IIS sends stderr back in the
document stream then we might as well go ahead and ensure that GDAL doesn't
write to stderr.  I can accomplish this by "pushing" an error handler that
supresses error output or perhaps sends it via msDebug() instead. 

But ... again ... I am not interested in trying to get IIS running at my end.
There are things I'll do for free (fix bugs in GDAL), and there are things
someone has to pay me to do (install and setup IIS). :-)

However, if you wanted, I could implement the stderr (from GDAL) redirection
change in 4.3 CVS if you are in a position to test it with IIS. 

to do.  

[mara@…]

No problem to test 4.3 binary (I'm not sure I could compile from CVS) with 
configuration, which produced reported error. This problem isn't critical for 
me. Now I already know, how to read this type of error message :)

Marek

[fwarmerdam]

I have looked at msGDALInitialize() in mapgdal.c, and even in 4.2.x it
does a "CPLPushErrorHandler( CPLQuietErrorHandler );" call.  This should
supress all direct GDAL error output. 

I tried running shp2img on an existing raster map, but with the filename
changed so it would fail.  I did not get any output error messages to stdout
or stderr from GDAL (or mapserver for that matter - due to the IGNORE_MISSING_DATA
stuff).  

Could you provide a more detailed explanation of a situation that demonstrates
the problem?  Can you reproduce the GDLA error message with a .map and shp2img? 

[dmorissette]

Frank, I have seen this (or a strangely similar problem) under Apache before.
However I have no way to reproduce it at the moment and Marek couldn't provide
enough information either.

So I'd suggest we close this as WONTFIX. We'll reopen if we ever have a testcase
to reproduce.

[fwarmerdam]

Agreed.

[mara@…]

Next week I'll have enough time to work on this problem and I hope I'll send 
you testcase to reproduce this problem.

[fwarmerdam]

OK, we can reopen it then.  Keep in mind I don't have IIS.   It would
be best if you could demonstrate the issue from the commandline.