outputformat filename as subtitition variable

[msmitherdc]

Add the OUTFORMAT filename as a runtime substitution variable

[sdlime]

Mike: I'll get this into the 3rd beta... Steve

[sdlime]

Mike: Is this a must have for 6.0 or could it wait until 6.2? I'm leery of anything that writes files and this might impact that. I don't know if Frank uses the filename when output is created or if it's just used as part of the http response.

Steve

[msmitherdc]

Steve,

Could you just create the patch for this and I'll incorporate it by hand until 6.2? It would also allow me to test and report any issues.

Mike

[sdlime]

Yup, will do!

Steve

[sdlime]

Hi Mike: Added a patch against trunk that adds substitution support to output formats that define a FILENAME format option. A validation string must be defined in the web object's validation (preferred) or metadata block. If the supplied value doesn't match or there is no validation expression then the substitution isn't done. I've not added support for a default value yet. Give this a test and let me know how it goes.

Steve

[sdlime]

Updated to allow setting of default_* values in appropriate metadata or validation (preferred) blocks.

[sdlime]

Committed to trunk in r11965. Closing...

Steve

[msmitherdc]

Steve,

Do you have an example validation pattern? I've tried

"outputformat_filename_validation_pattern" "."

in the web metadata section along with

OUTPUTFORMAT

NAME "Shapefile" DRIVER "OGR/ESRI Shapefile" FORMATOPTION "STORAGE=memory" FORMATOPTION "FORM=zip" FORMATOPTION "FILENAME=%outfile%"

END

and then specified outfile=xxx.zip in the url but i suspect I have the validation incorrect

Mike

[sdlime]

Try:

VALIDATION
  outfile 'some pattern'
  default_outfile 'some default'
END 

or

METADATA
  outfile_validation_pattern 'some pattern'
  default_outfile 'some default'
END 

Seems to me you'd most often fix the extension, that is:

FORMATOPTION "FILENAME=%outfile%.zip" 

Steve

[msmitherdc]

Thanks, exports are working great!

And i did change it so the extension is fixed.

Mike

[sdlime]

Reopening for documentation... Steve

[havatv]

Where is the FILENAME formatoption used? Only for OGR?

[msmitherdc]

Yes, in the OUTPUTFORMAT block for OGR output.

[msmitherdc]

Steve,

I'm getting a 500 error (core dump) when using the output filename but only on windows. I have no problems on linux. Not sure where the error is but as soon as I remove the outfile substitution from the URL request, no error. Add it back and I get core dump.

[sdlime]

And if the format contains no runtime sub then there's not issue? Unfortunately I have no ability to debug under windows. Can you generate any sort of stack trace as a hint?

Steve

[msmitherdc]

Correct about summary. However, i had jmckenna test on windows and he had no error (with shapefiles and i was using oracle). I will try to track this down further. I don't have good windows debugging tools either.

M

[msmitherdc]

I've been able to track down the circumstances that causes the error. If i have two different substitution strings (the output filename at least one other) in the same file. I added a layer with a FILTER subst string and got the error. Commented the FILTER line out, no error.

This will trigger the error on windows.

LAYER
  NAME test
  DATA "test.shp"
  TYPE point
  STATUS on
  PROJECTION
    "init=epsg:4326"
  END
  METADATA
  	"gml_types"   "auto"
        "gml_include_items" "all"
  END
  VALIDATION
    "test_filter"  "."
    "default_test_filter" "1=1"
  END
   FILTER "%test_filter%"
   CLASS
    SYMBOL 0
    SIZE 9
    COLOR 200 0 0
   END    
END

[havatv]

Documentation: OUTPUTFORMAT FORMATOPTION FILENAME was added to the runsub document in r12646. Done for both trunk and 6.0 documentation (a note says that it "was" added in version 6.2).

[jmckenna]

Regarding Mike's Windows crash reported in comment:15 I did some more testing, with today's trunk, compiled with Windows/MSVC2008, and here is my results:

• I have a small test package to use (attaching)
  • in /map there is a requests.txt file with the request to use
  • this uses a single spain provinces shapefile, and a filter is requested for a single province
  • relevant parts of the mapfile are:

       OUTPUTFORMAT
         NAME "SHAPEZIP"
         DRIVER "OGR/ESRI Shapefile"
         FORMATOPTION "STORAGE=memory"
         FORMATOPTION "FORM=zip"
         FORMATOPTION "FILENAME=%outfile%.zip"
       END
   
       ...

       WEB
         VALIDATION
         "outfile" "."
         "default_outfile" "result"
         "test_filter" "."
         "default_test_filter" "Toledo"
       END  
       
       ...

       LAYER
         NAME "spain_provinces"
         ...
         FILTER ("[ADMIN_NAME]" ="%test_filter%")
         ...
       END

 - what I've found:

   - for my first test, I comment out the FILTER line and just send an output filename, and in a web browser this returns a 500 Internal Server Error:

     {{{
       http://127.0.0.1/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=x
     }}}

   - fair enough, so my second test instead of "&output=x" I try "&output=x.zip", and I get a valid zipfile "x.zip.zip" containing the the full spain provinces shapefile correctly:

     {{{
       http://127.0.0.1/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=x.zip
     }}}

   - next, I remove the comment for the FILTER, and then in a web browser try the request "&outfile=x.zip&test_filter=Toledo", and the correct zipfile (named "x.zip.zip") is returned with a filtered shapefile only of the one spain province:

     {{{
       http://127.0.0.1/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=x.zip&test_filter=Toledo
     }}}

During my tests I think I was stuck because I was always trying "&outfile=x&test_filter=Toledo" and was getting an Internal Server Error.

So I am not sure if my test package helps/is needed, but, it seems I can in fact now make a FILTER work with variable substitution on Windows (with today's trunk)...however with some confusing filename results.

-jeff

[jmckenna]

test package: ​http://labs.gatewaygeomatics.com/dl/ogr-outputformat.zip (531KB)

[assefa]

I was able to reproduce a crash using Jeff's package and mapserver-svn. The crash, happens for me when the validation block has more than one default_xxx value (regardless of the other settings) The crash does not happen when in debug mode so difficult to trace it. The fact that It happens when there are 2 or more default_xxx, tend to mean that there might be memory corruption in/around applyOutputFormatDefaultSubstitutions. I could not pin point the exact problem. I also think that this might not be just a windows issue. I think we should start by running valdgrind or another tool on Linux with the test files.

[msmitherdc]

I can get the crash with no or one default value and a filter on a layer if I have a substitution variable block for the output filename but only in windows. I have not been able to reproduce in linux.

[assefa]

Hi Mike. This is waht I have (using the test case):

WEB
...
VALIDATION
    "outfile" "."
    "default_outfile" "result"
    "test_filter" "."
#    "default_test_filter" "Toledo"  #uncommenting this will always give a crash
  END  

Request looks like:

http://127.0.0.1/cgi-bin/mapserv.exe?MAP=c:/msapps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=zz&test_filter=Toledo

The crash happens regardless of the FILTER ("[ADMIN_NAME]" ="%test_filter%") setting on the layer (commented out or not).

Are we using the same test files?

[msmitherdc]

I have my validation in different blocks. I have my output filename in a WEB...VALIDATION block and have have LAYER level VALIDATION blocks inside the LAYER...END blocks.

Don't know if that makes a difference.

[assefa]

I committed r12781 that seems to fixed the issues I was having. Is it possible to give it a try?

[msmitherdc]

Assefa,

I'm still getting the core dump (error-500) whenever the outfile validation matches a passed pass site. Other validations work (layer filter validations) but we always get the core dump with the output filename validation block (and only on windows).

[assefa]

Mike,

Can you send me the map file and url you are using causing the crash, based on the test kit attached to this bug? It will be easier to debug.

[msmitherdc]

Example Test Mapfile

[msmitherdc]

URL Request

​http://server/cgi-bin/mapserv.exe?map=c:/cm2_files/mapfiles/cm2_test2.map&service=wfs&version=1.1.0&request=getFeature&typename=test&outputformat=Shapefile&outfile=test&SRSNAME=EPSG:4326

[assefa]

test map file

[assefa]

I took the same setting that were in your map file and used a similar url and I get the expected result (zip file). I have attached the map file.

The fix committed yesterday was the only one that I could find that was using corrupted memory causing a crash (BTW It was not particular to windows).

Jeff, when you get a couple of minutes, can you please rerun your tests with new code from svn?

http://127.0.0.1/cgi-bin/mapserv.exe?map=c:/msapps/ogr-outputformat/map/ogr-outputformat-mike-settings.map&service=wfs&version=1.1.0&request=getFeature&typename=spain_provinces&outputformat=SHAPEZIP&outfile=test&SRSNAME=EPSG:23030

[jmckenna]

I re-ran the tests from the package in comment:22 (which has 2 variables, one in the FILTER statement and the other in the OUTPUTFORMAT) with trunk and I have no problems. Good work Assefa!

[jmckenna]

Mike: can you try the test package (​http://labs.gatewaygeomatics.com/dl/ogr-outputformat.zip) locally? (there is a requests.txt file in there as well) If you can make that test package break, can you repackage that with a modified mapfile and requests.txt so I/we can test it locally?

(after your turkey-day of course)

[jmckenna]

I also notice a problem with the attached 'cm2_test2.map' (notice the double underscore):

  VALIDATION
    "test__filter"  "."
    "default_test_filter" "1=1"
  END

The following change causes the filter to work properly in my testing:

  VALIDATION
    "test_filter"  "."
    "default_test_filter" "1=1"
  END

[assefa]

oh turkey, love that bird :)

Jeff, you mentioned "a problem" with the double underscore: problem meaning it does not work or is it crashing?

Mike, note that I am willing to put more time into this if I get a reproducible case.

[msmitherdc]

I'll try to retest in the next couple of hours

[jmckenna]

With that double underscore, the outfile was generated but the zip was empty. (so this isn't related to mike's problem)

[msmitherdc]

Several things going on here.

1). Removing the parens around the filter statement triggers a 500 error

2). I was doing my filters differently. Rather than just passing the right side of the predicate (eg FILTER ("[admin_name] = "%filter%") ), I am passing the entire filter condition ( FILTER ("%filter%) ). In the test case, the subst variable is test_filter=Toledo, what I need to do is pass test_filter="admin_name='Toledo'". This works very well for Oracle layers (i haven't been doing this with shapefiles).

[msmitherdc]

1). load data to Oracle Spatial

2) use attached mapfile

3). map draw works: ​http://localhost/cgi-bin/mapserv.exe?MAP=c:/cm2_files/mapfiles/ogr-outputformat.map&mode=map&layer=spain_provinces&test_filter=ADMIN_NAME='Len'

4).wfs query works: ​http://localhost/cgi-bin/mapserv.exe?MAP=c:/cm2_files/mapfiles/ogr-outputformat.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=spain_provinces&test_filter=ADMIN_NAME='Len'

5). Fails with outfile=x: cgi-bin/mapserv.exe?MAP=c:/cm2_files/mapfiles/ogr-outputformat.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=spain_provinces&test_filter=ADMIN_NAME='Len'&outfile=x

[msmitherdc]

test mapfile using oracle

[jmckenna]

Tested with 'Oracle Database 10g Enterprise Edition Release 10.2.0.1.0', here are my steps:

• used attached 'ogr-outputformat-oracle.map'
• today's mapserver trunk
• loaded esp.shp through ogr2ogr to Oracle instance
• tested mapfile with shp2img without substitutions, no problem
• enabled substitutions
• used request above (5), but added missing OUTPUTFORMAT parameter to the URL:

    localhost/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat-oracle.map
    &SERVICE=WFS&VERSION=1.1.0
    &REQUEST=GetFeature
    &TYPENAME=spain_provinces
    &OUTPUTFORMAT=SHAPEZIP
    &test_filter=ADMIN_NAME='Len'
    &outfile=x 
  

• oracle/mapserver returns a valid 'x.zip' file containing just the single 'Len' spain province

[msmitherdc]

I'll try again with todays trunk. AT r12781, it errors on me.

[jmckenna]

test mapfile using postgres and 2 layers

[jmckenna]

There were many problems trying to duplicate this with shapefiles, but it is easily duplicated through Postgres. Here are the steps:

• use the same test package with esp.shp data from comment:22
• load the esp.shp file into your local postgres database. Here is my ogr2ogr command:

    ogr2ogr -f PostgreSQL PG:"host=localhost port=5433 user=postgres password=postgres dbname=spain" esp.shp -nlt MULTIPOLYGON
  

• my versions are PostgreSQL Version: 9.0.4 and PostGIS Version: 1.5.2
• use the attached mapfile ('ogr-outputformat-pg-2layers.map')
• you will note that this mapfile has 2 layers, and 2 filters through variable substitution
• Mike discovered that 1 filter is fine, but 2 crashes mapserv (only with the 'outfile=x' parameter included)
• this request crashes mapserv (this is the same request that crashes an equivalent Oracle mapfile as well): ​http://localhost/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat-pg-2layers.map&mode=map&layer=spain_provinces&test_filter=admin_name=%27Toledo%27&outfile=x
• if you remove '&outfile=x' from the above request, for oracle and postgres, there is no issues generating the map image

[assefa]

committed r12822. This did fix for me the problem described in the last comment.

[msmitherdc]

Confirmed. The fix is working for me in the tests and in my production mapfiles. Thanks Assefa and Jeff!

[jmckenna]

agreed, testcase works locally with Oracle and Postgres! :)

[assefa]

Perfect. Thx all for tracking this.

[aboudreault]

I think this ticket can be closed. Thanks Assefa.