Opened 9 years ago

Closed 8 years ago

#3751 closed defect (fixed)

outputformat filename as subtitition variable

Reported by: msmitherdc Owned by: sdlime
Priority: normal Milestone: 6.2 release
Component: MapServer C Library Version: svn-trunk (development)
Severity: normal Keywords: outputformat, filename, runtime
Cc: havatv, jmckenna, assefa

Description

Add the OUTFORMAT filename as a runtime substitution variable

Attachments (5)

3751.patch (4.7 KB) - added by sdlime 8 years ago.
Updated to allow setting of default_* values in appropriate metadata or validation (preferred) blocks.
cm2_test2.map (4.4 KB) - added by msmitherdc 8 years ago.
Example Test Mapfile
ogr-outputformat-mike-settings.map (2.4 KB) - added by assefa 8 years ago.
test map file
ogr-outputformat-oracle.map (2.5 KB) - added by msmitherdc 8 years ago.
test mapfile using oracle
ogr-outputformat-pg-2layers.map (3.5 KB) - added by jmckenna 8 years ago.
test mapfile using postgres and 2 layers

Download all attachments as: .zip

Change History (53)

comment:1 Changed 9 years ago by sdlime

Mike: I'll get this into the 3rd beta... Steve

comment:2 Changed 9 years ago by sdlime

Status: newassigned

comment:3 Changed 9 years ago by sdlime

Mike: Is this a must have for 6.0 or could it wait until 6.2? I'm leery of anything that writes files and this might impact that. I don't know if Frank uses the filename when output is created or if it's just used as part of the http response.

Steve

comment:4 Changed 9 years ago by msmitherdc

Steve,

Could you just create the patch for this and I'll incorporate it by hand until 6.2? It would also allow me to test and report any issues.

Mike

comment:5 Changed 9 years ago by sdlime

Yup, will do!

Steve

comment:6 Changed 8 years ago by sdlime

Milestone: 6.0 release6.2 release

comment:7 Changed 8 years ago by sdlime

Hi Mike: Added a patch against trunk that adds substitution support to output formats that define a FILENAME format option. A validation string must be defined in the web object's validation (preferred) or metadata block. If the supplied value doesn't match or there is no validation expression then the substitution isn't done. I've not added support for a default value yet. Give this a test and let me know how it goes.

Steve

Changed 8 years ago by sdlime

Attachment: 3751.patch added

Updated to allow setting of default_* values in appropriate metadata or validation (preferred) blocks.

comment:8 Changed 8 years ago by sdlime

Resolution: fixed
Status: assignedclosed

Committed to trunk in r11965. Closing...

Steve

comment:9 Changed 8 years ago by msmitherdc

Steve,

Do you have an example validation pattern? I've tried

"outputformat_filename_validation_pattern" "."

in the web metadata section along with

OUTPUTFORMAT

NAME "Shapefile" DRIVER "OGR/ESRI Shapefile" FORMATOPTION "STORAGE=memory" FORMATOPTION "FORM=zip" FORMATOPTION "FILENAME=%outfile%"

END

and then specified outfile=xxx.zip in the url but i suspect I have the validation incorrect

Mike

comment:10 Changed 8 years ago by sdlime

Try:

VALIDATION
  outfile 'some pattern'
  default_outfile 'some default'
END 

or

METADATA
  outfile_validation_pattern 'some pattern'
  default_outfile 'some default'
END 

Seems to me you'd most often fix the extension, that is:

FORMATOPTION "FILENAME=%outfile%.zip" 

Steve

comment:11 Changed 8 years ago by msmitherdc

Thanks, exports are working great!

And i did change it so the extension is fixed.

Mike

comment:12 Changed 8 years ago by sdlime

Component: MapServer C LibraryMapServer Documentation
Resolution: fixed
Status: closedreopened

Reopening for documentation... Steve

comment:13 Changed 8 years ago by havatv

Cc: havatv added

Where is the FILENAME formatoption used? Only for OGR?

comment:14 Changed 8 years ago by msmitherdc

Yes, in the OUTPUTFORMAT block for OGR output.

comment:15 Changed 8 years ago by msmitherdc

Component: MapServer DocumentationMapServer C Library

Steve,

I'm getting a 500 error (core dump) when using the output filename but only on windows. I have no problems on linux. Not sure where the error is but as soon as I remove the outfile substitution from the URL request, no error. Add it back and I get core dump.

comment:16 Changed 8 years ago by msmitherdc

Type: enhancementdefect

comment:17 Changed 8 years ago by sdlime

And if the format contains no runtime sub then there's not issue? Unfortunately I have no ability to debug under windows. Can you generate any sort of stack trace as a hint?

Steve

comment:18 Changed 8 years ago by msmitherdc

Cc: jmckenna added

Correct about summary. However, i had jmckenna test on windows and he had no error (with shapefiles and i was using oracle). I will try to track this down further. I don't have good windows debugging tools either.

M

comment:19 Changed 8 years ago by msmitherdc

I've been able to track down the circumstances that causes the error. If i have two different substitution strings (the output filename at least one other) in the same file. I added a layer with a FILTER subst string and got the error. Commented the FILTER line out, no error.

This will trigger the error on windows.

LAYER
  NAME test
  DATA "test.shp"
  TYPE point
  STATUS on
  PROJECTION
    "init=epsg:4326"
  END
  METADATA
  	"gml_types"   "auto"
        "gml_include_items" "all"
  END
  VALIDATION
    "test_filter"  "."
    "default_test_filter" "1=1"
  END
   FILTER "%test_filter%"
   CLASS
    SYMBOL 0
    SIZE 9
    COLOR 200 0 0
   END    
END

comment:20 Changed 8 years ago by havatv

Documentation: OUTPUTFORMAT FORMATOPTION FILENAME was added to the runsub document in r12646. Done for both trunk and 6.0 documentation (a note says that it "was" added in version 6.2).

comment:21 Changed 8 years ago by jmckenna

Regarding Mike's Windows crash reported in comment:15 I did some more testing, with today's trunk, compiled with Windows/MSVC2008, and here is my results:

  • I have a small test package to use (attaching)
    • in /map there is a requests.txt file with the request to use
    • this uses a single spain provinces shapefile, and a filter is requested for a single province
    • relevant parts of the mapfile are:
       OUTPUTFORMAT
         NAME "SHAPEZIP"
         DRIVER "OGR/ESRI Shapefile"
         FORMATOPTION "STORAGE=memory"
         FORMATOPTION "FORM=zip"
         FORMATOPTION "FILENAME=%outfile%.zip"
       END
   
       ...

       WEB
         VALIDATION
         "outfile" "."
         "default_outfile" "result"
         "test_filter" "."
         "default_test_filter" "Toledo"
       END  
       
       ...

       LAYER
         NAME "spain_provinces"
         ...
         FILTER ("[ADMIN_NAME]" ="%test_filter%")
         ...
       END

 - what I've found:

   - for my first test, I comment out the FILTER line and just send an output filename, and in a web browser this returns a 500 Internal Server Error:

     {{{
       http://127.0.0.1/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=x
     }}}

   - fair enough, so my second test instead of "&output=x" I try "&output=x.zip", and I get a valid zipfile "x.zip.zip" containing the the full spain provinces shapefile correctly:

     {{{
       http://127.0.0.1/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=x.zip
     }}}

   - next, I remove the comment for the FILTER, and then in a web browser try the request "&outfile=x.zip&test_filter=Toledo", and the correct zipfile (named "x.zip.zip") is returned with a filtered shapefile only of the one spain province:

     {{{
       http://127.0.0.1/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=x.zip&test_filter=Toledo
     }}}

During my tests I think I was stuck because I was always trying "&outfile=x&test_filter=Toledo" and was getting an Internal Server Error.

So I am not sure if my test package helps/is needed, but, it seems I can in fact now make a FILTER work with variable substitution on Windows (with today's trunk)...however with some confusing filename results.

-jeff




comment:23 Changed 8 years ago by assefa

Cc: assefa added

comment:24 Changed 8 years ago by assefa

I was able to reproduce a crash using Jeff's package and mapserver-svn. The crash, happens for me when the validation block has more than one default_xxx value (regardless of the other settings) The crash does not happen when in debug mode so difficult to trace it. The fact that It happens when there are 2 or more default_xxx, tend to mean that there might be memory corruption in/around applyOutputFormatDefaultSubstitutions. I could not pin point the exact problem. I also think that this might not be just a windows issue. I think we should start by running valdgrind or another tool on Linux with the test files.

comment:25 Changed 8 years ago by msmitherdc

I can get the crash with no or one default value and a filter on a layer if I have a substitution variable block for the output filename but only in windows. I have not been able to reproduce in linux.

comment:26 Changed 8 years ago by assefa

Hi Mike. This is waht I have (using the test case):

WEB
...
VALIDATION
    "outfile" "."
    "default_outfile" "result"
    "test_filter" "."
#    "default_test_filter" "Toledo"  #uncommenting this will always give a crash
  END  

Request looks like:

http://127.0.0.1/cgi-bin/mapserv.exe?MAP=c:/msapps/ogr-outputformat/map/ogr-outputformat.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=spain_provinces&OUTPUTFORMAT=SHAPEZIP&outfile=zz&test_filter=Toledo

The crash happens regardless of the FILTER ("[ADMIN_NAME]" ="%test_filter%") setting on the layer (commented out or not).

Are we using the same test files?

comment:27 Changed 8 years ago by msmitherdc

I have my validation in different blocks. I have my output filename in a WEB...VALIDATION block and have have LAYER level VALIDATION blocks inside the LAYER...END blocks.

Don't know if that makes a difference.

comment:28 Changed 8 years ago by assefa

I committed r12781 that seems to fixed the issues I was having. Is it possible to give it a try?

comment:29 Changed 8 years ago by msmitherdc

Assefa,

I'm still getting the core dump (error-500) whenever the outfile validation matches a passed pass site. Other validations work (layer filter validations) but we always get the core dump with the output filename validation block (and only on windows).

comment:30 Changed 8 years ago by assefa

Mike,

Can you send me the map file and url you are using causing the crash, based on the test kit attached to this bug? It will be easier to debug.

Changed 8 years ago by msmitherdc

Attachment: cm2_test2.map added

Example Test Mapfile

Changed 8 years ago by assefa

test map file

comment:32 Changed 8 years ago by assefa

I took the same setting that were in your map file and used a similar url and I get the expected result (zip file). I have attached the map file.

The fix committed yesterday was the only one that I could find that was using corrupted memory causing a crash (BTW It was not particular to windows).

Jeff, when you get a couple of minutes, can you please rerun your tests with new code from svn?

http://127.0.0.1/cgi-bin/mapserv.exe?map=c:/msapps/ogr-outputformat/map/ogr-outputformat-mike-settings.map&service=wfs&version=1.1.0&request=getFeature&typename=spain_provinces&outputformat=SHAPEZIP&outfile=test&SRSNAME=EPSG:23030

comment:33 Changed 8 years ago by jmckenna

I re-ran the tests from the package in comment:22 (which has 2 variables, one in the FILTER statement and the other in the OUTPUTFORMAT) with trunk and I have no problems. Good work Assefa!

comment:34 Changed 8 years ago by jmckenna

Mike: can you try the test package (http://labs.gatewaygeomatics.com/dl/ogr-outputformat.zip) locally? (there is a requests.txt file in there as well) If you can make that test package break, can you repackage that with a modified mapfile and requests.txt so I/we can test it locally?

(after your turkey-day of course)

comment:35 Changed 8 years ago by jmckenna

I also notice a problem with the attached 'cm2_test2.map' (notice the double underscore):

  VALIDATION
    "test__filter"  "."
    "default_test_filter" "1=1"
  END

The following change causes the filter to work properly in my testing:

  VALIDATION
    "test_filter"  "."
    "default_test_filter" "1=1"
  END

comment:36 Changed 8 years ago by assefa

oh turkey, love that bird :)

Jeff, you mentioned "a problem" with the double underscore: problem meaning it does not work or is it crashing?

Mike, note that I am willing to put more time into this if I get a reproducible case.

comment:37 Changed 8 years ago by msmitherdc

I'll try to retest in the next couple of hours

comment:38 Changed 8 years ago by jmckenna

With that double underscore, the outfile was generated but the zip was empty. (so this isn't related to mike's problem)

comment:39 Changed 8 years ago by msmitherdc

Several things going on here.

1). Removing the parens around the filter statement triggers a 500 error

2). I was doing my filters differently. Rather than just passing the right side of the predicate (eg FILTER ("[admin_name] = "%filter%") ), I am passing the entire filter condition ( FILTER ("%filter%) ). In the test case, the subst variable is test_filter=Toledo, what I need to do is pass test_filter="admin_name='Toledo'". This works very well for Oracle layers (i haven't been doing this with shapefiles).

comment:40 Changed 8 years ago by msmitherdc

1). load data to Oracle Spatial

2) use attached mapfile

3). map draw works: http://localhost/cgi-bin/mapserv.exe?MAP=c:/cm2_files/mapfiles/ogr-outputformat.map&mode=map&layer=spain_provinces&test_filter=ADMIN_NAME='Len'

4).wfs query works: http://localhost/cgi-bin/mapserv.exe?MAP=c:/cm2_files/mapfiles/ogr-outputformat.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=spain_provinces&test_filter=ADMIN_NAME='Len'

5). Fails with outfile=x: cgi-bin/mapserv.exe?MAP=c:/cm2_files/mapfiles/ogr-outputformat.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=spain_provinces&test_filter=ADMIN_NAME='Len'&outfile=x

Changed 8 years ago by msmitherdc

Attachment: ogr-outputformat-oracle.map added

test mapfile using oracle

comment:41 Changed 8 years ago by jmckenna

Tested with 'Oracle Database 10g Enterprise Edition Release 10.2.0.1.0', here are my steps:

  • used attached 'ogr-outputformat-oracle.map'
  • today's mapserver trunk
  • loaded esp.shp through ogr2ogr to Oracle instance
  • tested mapfile with shp2img without substitutions, no problem
  • enabled substitutions
  • used request above (5), but added missing OUTPUTFORMAT parameter to the URL:
      localhost/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat-oracle.map
      &SERVICE=WFS&VERSION=1.1.0
      &REQUEST=GetFeature
      &TYPENAME=spain_provinces
      &OUTPUTFORMAT=SHAPEZIP
      &test_filter=ADMIN_NAME='Len'
      &outfile=x 
    
  • oracle/mapserver returns a valid 'x.zip' file containing just the single 'Len' spain province

comment:42 Changed 8 years ago by msmitherdc

I'll try again with todays trunk. AT r12781, it errors on me.

Changed 8 years ago by jmckenna

test mapfile using postgres and 2 layers

comment:43 Changed 8 years ago by jmckenna

There were many problems trying to duplicate this with shapefiles, but it is easily duplicated through Postgres. Here are the steps:

  • use the same test package with esp.shp data from comment:22
  • load the esp.shp file into your local postgres database. Here is my ogr2ogr command:
      ogr2ogr -f PostgreSQL PG:"host=localhost port=5433 user=postgres password=postgres dbname=spain" esp.shp -nlt MULTIPOLYGON
    
  • my versions are PostgreSQL Version: 9.0.4 and PostGIS Version: 1.5.2
  • use the attached mapfile ('ogr-outputformat-pg-2layers.map')
  • you will note that this mapfile has 2 layers, and 2 filters through variable substitution
  • Mike discovered that 1 filter is fine, but 2 crashes mapserv (only with the 'outfile=x' parameter included)
  • this request crashes mapserv (this is the same request that crashes an equivalent Oracle mapfile as well): http://localhost/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ogr-outputformat/map/ogr-outputformat-pg-2layers.map&mode=map&layer=spain_provinces&test_filter=admin_name=%27Toledo%27&outfile=x
  • if you remove '&outfile=x' from the above request, for oracle and postgres, there is no issues generating the map image

comment:44 Changed 8 years ago by assefa

committed r12822. This did fix for me the problem described in the last comment.

comment:45 Changed 8 years ago by msmitherdc

Confirmed. The fix is working for me in the tests and in my production mapfiles. Thanks Assefa and Jeff!

comment:46 Changed 8 years ago by jmckenna

agreed, testcase works locally with Oracle and Postgres! :)

comment:47 Changed 8 years ago by assefa

Perfect. Thx all for tracking this.

comment:48 Changed 8 years ago by aboudreault

Resolution: fixed
Status: reopenedclosed

I think this ticket can be closed. Thanks Assefa.

Note: See TracTickets for help on using tickets.