MapScript not closing font file handles and/or "include"d mapfiles

[pete@…]

When using PHP Mapscript to generate map tiles (using tile.php from the Ka-Map
project), file handles to TTF font files aren't being closed after the script is
finished.  The output of lsof shows:

/xxx/VeraBd.ttf
httpd     31401   apache  mem       REG      253,0      65932   18645269
/xxx/Vera.ttf
httpd     31439   apache  mem       REG      253,0      63684   18645272
/xxx/VeraIt.ttf
httpd     31439   apache  mem       REG      253,0      65932   18645269
/xxx/Vera.ttf

In addition, the same thing seems to happen when using the "INCLUDE" directive
inside a mapfile.  The included mapfiles aren't being closed:

In main mapfile:

INCLUDE "/xxx/subfile1.map"
INCLUDE "/xxx/subfile2.map"

Output of lsof:

httpd     31978   apache   60r      REG      253,0      85032    9389109
/xxx/subfile1.map
httpd     31978   apache   61r      REG      253,0       3580    9389098
/xxx/subfile2.map
httpd     31978   apache   62r      REG      253,0      85032    9389109
/xxx/subfile1.map
httpd     31978   apache   63r      REG      253,0       3580    9389098
/xxx/subfile2.map

As a result, continued requests to the script eventually exceed the system's
maximum number of open file handles and the requests begin to fail.

This doesn't occur when using the mapserv executable on the same mapfiles.

[dmorissette]

I can speak for the INCLUDEed file handles, but a few months ago I have done
some work to trace leaked file handles and I never noticed any leaks of TTF
files. I wonder if this might happen only with a specific version of Freetype.

Can you please tell us which OS and version of Freetype you are using?

[pete@…]

I'm using Red Hat Enterprise Linux 4.  Relevant libraries:

freetype: 2.1.9 (stock Red Hat)
gd: 2.0.28 (stock Red Hat)

[pete@…]

More information if needed:

Inside the mapfile, the fontset declaration looks like this:

  FONTSET "../etc/fonts/fonts.txt"

Contents of fonts.txt:

sans                    Vera.ttf
sans-bold               VeraBd.ttf
sans-bold-oblique       VeraBI.ttf
sans-oblique            VeraIt.ttf
monospace-bold          VeraMoBd.ttf
monospace-bold-oblique  VeraMoBI.ttf
monospace-oblique       VeraMoIt.ttf
monospace               VeraMono.ttf
serif                   VeraSe.ttf
serif-bold              VeraSeBd.ttf

All fonts are in the same directory as fonts.txt

[sdlime]

Changing components. The font file handle is news to me. I suspected I screwed 
up the include so this is good confirmation and I can fix quickly.

[sdlime]

Pete: I committed a fix for the include problem to the main branch. Any chance 
you can verify? If positive then I'll backport to the 4.10 branch.

MapServer isn't responsible for managing the TTF files. That falls to GD I 
believe, or perhaps Freetype. It would be interesting to test a newer version 
of GD and Freetype and see if the problem goes away. GD 2.0.34RC2 is out at 
www.libgd.org...

Steve

[jmckenna@…]

this include problem does not occur in windows builds.

[sdlime]

Jeff: To clarify, do you mean before it didn't or now it doesn't?

Steve

[jmckenna@…]

sorry.  This problem does not exist with the 4.10.0 release on Windows.

[pete@…]

I rebuilt Mapserver with only the change to maplexer.l added, and still get the
same problem with the included mapfiles.  Is there some other change to the
source I need to include?

[sdlime]

No, that should do it. You should rebuild and reinstall PHP mapscript too.

Steve

[sdlime]

Pete: Try something for me. My last mod may not be affecting the right file 
handle. In maplexer.l try changing:

fclose(msyyin);

to:

fclose(YY_CURRENT_BUFFER->yy_input_file);

and rebuild things. Let's see if that works. If not I'll have to dig through my 
flex book at home.

Steve

[pete@…]

Tried the change from #11 - still the same results and lots of open files.  I
rebuilt and installed mapserver and mapscript-php and restarted Apache before
testing the software again.

[sdlime]

Maybe this is PHP specific. I'm not sure what else to try. I'll check the 
O'Reilly Flex/Bison book for ideas.

Steve

[sdlime]

Pete: Have you experienced anything different than last reported? The 
documentation is of little use here. As near as I can tell included files are 
being closed in CVS HEAD version.

Steve

[dmorissette]

Another report of the same included file leak on the users list:

Subject: Re: [UMN_MAPSERVER-USERS] Too many open files on mapserv 4.10.1
Date: Tue, 22 May 2007 11:55:59 -0300
From: Rodrigo Del C. Andrade <rodrigo.andrade@...>
Reply-To: Rodrigo Del C. Andrade <rodrigo.andrade@...>
To: MAPSERVER-USERS@LISTS.UMN.EDU

   I am way too swamped right now to fill a bug report, but check in 
the INCLUDES on your mapfile.
   I have this problem on 4.10.1 and the culprit was the includes I had 
on my main mapfile being left open.

   Cheers,

[hobu]

I suspect #1994 is related to this as well.

[sdlime]

Replying to hobu:

I suspect #1994 is related to this as well.

I believe the include issue is fixed in the 5.0 main branch. I could never recreate the problem so it's hard for me to test. I recall one of the users who posted the problem confirming that fix. If someone else can verify that would help.

The TTF issue is a GD issue (MapServer doesn't open/close .ttf files, just the font set). I would suggest the user upgrade to gd 2.0.33+ and try that. I know there were a number of TTF patches applied recently.

Steve

[jmckenna]

#2045 is also an include-related issue. I'll have to re-test with the 5.0 branch.

[assefa]

pete,

can you confirm if this issue is fixed for you with recent Mapserver source ?

[sdlime]

Marking as fixed for now. Can create more specific tickets in the future if necessary.

Steve

[pdeffendol]

I've tested with 5.0 beta 5, and I do not seem to be getting open INCLUDEd map files, but the font file problem still exists.

[sdlime]

Let's file a seperate bug on the font file...

Steve

[sdlime]

I looked at the code (see msLoadFontSet in maplabel.c) and the file handle for the font file is clearly being closed or at least I can't see how it could be being left open...

Steve