ossim-batch-test
ossim-batch-test is a command line test application that can execute programs, test the results against expected results, and log the output with differences between expected results and run times. ossim-batch-test requires a test configuration file for input and generates a log file on output. Multiple test can be in a single configuration file. A single test, all test or multiple test can be ran with a single ossim-batch-test command. All times are logged for each command. So you can easily test for changes in output and changes in output times. This application is written in c++ so it's portable on windows, linux, MACs. Typically to switch from one operating system to another the file paths and diff commands have to be query / replaced in config file. On linux, MAC typically "diff" is used. On windows "fc" works well.
Example of ossim-orthoigen test on a NITF image with RPC model:
For this test three things:
1) The height about the center of the test image. Because this image has a RPC sensor model which is affected by elevation, the first test is the height at the center of the image. If this changes the output bounds will also change.
2) The initial output of ossim-info -d -i -p of the source test image. This alone will test many changes in the code.
3) The orthoigen output. Test reprojecting image to an output file. Testing the histogram checks for pixel changes. Testing the ossim-info -d -i -p <output-image> tests for geometry changes.
From here on out commands start with $ and comments start with .
This example was generated on linux. Windows example coming soon...
Some things to note. Test image is under OSSIM_TEST environment variable:
$ env | grep OSSIM_TEST OSSIM_TEST=/data1/test $ pwd /data1/test $ ls source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/ 5V090205M0001912264B220000100072M_001508507.his 5V090205M0001912264B220000100072M_001508507.ntf 5V090205M0001912264B220000100072M_001508507.ovr Single_Organization_license.txt
Generate a configuration file template.
$ ossim-batch-test -w orthoigen-test.kwl Wrote file: orthoigen-test.kwl $
Edit as required.
Contents of this file:
$ cat orthoigen-test.kwl
// --- // File: orthoigen-test.kwl // Generated by command: ossim-batch-test -w orthoigen-test.kwl // Description: ossim-batch-test config file. // --- // --- // Control flags: // Use 1 or true, 0 or false to turn on and off sections. // Typically preprocess and expected results only turned on for first run. // Clean commands should erase anything created by this test. // Command order if flag turned on: // 1) clean, 2) pre-process, 3) expected, 4) test, 5) post-process // --- // --- // NOTES: // 1) Use forward slash, slash(c++ comment) at beginning of line for comments. // 2) For environment variables use $(YOUR_VARIABLE). They will be expanded at // run time if valid. // 3) Clean, preprocess, run, and accept test sections can be switched on or off // at run time. These override config file flags. Optional arguments are: // --clean-test // --preprocess-test // --run-test // --accept-test // Multiple tests can be entered by quoting string of space separated // test, e.g. "test1 test2". To accept all tests use "all" for test. // --- // Where you want the log files to go: log_directory: $(OSSIM_TEST)/logs/osgeo // --- // Begin test1: // --- test1.name: test1 test1.description: Test height for the center of the image for test2 and test3. // Controls/turns on/off whole test (all sections): test1.enabled: 1 // Individual control flags: test1.run_clean_commands: 0 test1.run_preprocessing_commands: 0 test1.run_expected_results_commands: 0 test1.run_test_commands: 1 test1.run_postprocessing_commands: 1 // Temp file to catch diff output. test1.temp_file: /tmp/tmp.txt // Clean up commands. test1.clean_command0: rm -rf $(OSSIM_TEST)/exp/orthoigen-test/test1 test1.clean_command1: rm -rf $(OSSIM_TEST)/out/orthoigen-test/test1 // Pre-process commands: test1.preprocess_command0: mkdir -p $(OSSIM_TEST)/exp/orthoigen-test/test1 test1.preprocess_command1: mkdir -p $(OSSIM_TEST)/out/orthoigen-test/test1 test1.preprocess_command1: mkdir -p $(OSSIM_TEST)/logs/osgeo // Commands to generate expected results: // Since test2 and test3 are dependent on elevation test the center of the image. test1.expected_results_command0: ossim-info --height -42.8508 147.2537 > $(OSSIM_TEST)/exp/orthoigen-test/test1/height.txt // The actual commands to test: test1.test_command0: ossim-info --height -42.8508 147.2537 > $(OSSIM_TEST)/out/orthoigen-test/test1/height.txt // Post process commands for diffs and stuff: test1.postprocess_command0: diff -w $(OSSIM_TEST)/exp/orthoigen-test/test1/height.txt $(OSSIM_TEST)/out/orthoigen-test/test1/height.txt // --- // End test1: // --- // --- // Begin test2: // --- test2.name: test2 test2.description: Test geometry for GeoEye NITF with RPC model. // Controls/turns on/off whole test (all sections): test2.enabled: 1 // Individual control flags: test2.run_clean_commands: 0 test2.run_preprocessing_commands: 0 test2.run_expected_results_commands: 0 test2.run_test_commands: 1 test2.run_postprocessing_commands: 1 // Temp file to catch diff output. test2.temp_file: /tmp/tmp.txt // Clean up commands. test2.clean_command0: rm -rf $(OSSIM_TEST)/exp/orthoigen-test/test2 test2.clean_command1: rm -rf $(OSSIM_TEST)/out/orthoigen-test/test2 // Pre-process commands: test2.preprocess_command0: mkdir -p $(OSSIM_TEST)/exp/orthoigen-test/test2 test2.preprocess_command1: mkdir -p $(OSSIM_TEST)/out/orthoigen-test/test2 test2.preprocess_command1: mkdir -p $(OSSIM_TEST)/logs/osgeo // Commands to generate expected results: test2.expected_results_command0: ossim-info -d -i -p $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > $(OSSIM_TEST)/exp/orthoigen-test/test2/info.txt // The actual commands to test: test2.test_command0: ossim-info -d -i -p $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > $(OSSIM_TEST)/out/orthoigen-test/test2/info.txt // Post process commands for diffs and stuff: test2.postprocess_command0: diff -w $(OSSIM_TEST)/exp/orthoigen-test/test2/info.txt $(OSSIM_TEST)/out/orthoigen-test/test2/info.txt // --- // End test2: // --- // --- // Begin test3: // --- test3.name: test3 test3.description: Test orthorectification of a GeoEye NITF with RPC model. // Controls/turns on/off whole test (all sections): test3.enabled: 1 // Individual control flags: test3.run_clean_commands: 0 test3.run_preprocessing_commands: 0 test3.run_expected_results_commands: 0 test3.run_test_commands: 1 test3.run_postprocessing_commands: 1 // Temp file to catch diff output. test3.temp_file: /tmp/tmp.txt // Clean up commands. test3.clean_command0: rm -rf $(OSSIM_TEST)/exp/orthoigen-test/test3 test3.clean_command1: rm -rf $(OSSIM_TEST)/out/orthoigen-test/test3 // Pre-process commands: test3.preprocess_command0: mkdir -p $(OSSIM_TEST)/exp/orthoigen-test/test3 test3.preprocess_command1: mkdir -p $(OSSIM_TEST)/out/orthoigen-test/test3 test3.preprocess_command1: mkdir -p $(OSSIM_TEST)/logs/osgeo // Commands to generate expected results: // Put the output in the "out" directory as the "exp" may be under svn for small things: // Ortho: test3.expected_results_command0: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf $(OSSIM_TEST)/out/orthoigen-test/test3/t1.tif // Histogram: test3.expected_results_command1: ossim-img2rr -r --create-histogram $(OSSIM_TEST)/out/orthoigen-test/test3/t1.tif // Geometry: test3.expected_results_command2: ossim-info -d -i -p $(OSSIM_TEST)/out/orthoigen-test/test3/t1.tif > $(OSSIM_TEST)/out/orthoigen-test/test3/t1-info.txt // Copy to exp directory which is under svn control: test3.expected_results_command3: cp $(OSSIM_TEST)/out/orthoigen-test/test3/t1.his $(OSSIM_TEST)/exp/orthoigen-test/test3/t1.his test3.expected_results_command4: cp $(OSSIM_TEST)/out/orthoigen-test/test3/t1-info.txt $(OSSIM_TEST)/exp/orthoigen-test/test3/t1-info.txt // The actual commands to test: // Ortho: test3.test_command0: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf $(OSSIM_TEST)/out/orthoigen-test/test3/t2.tif // Histogram: test3.test_command1: ossim-img2rr -r --create-histogram $(OSSIM_TEST)/out/orthoigen-test/test3/t2.tif // Geometry: test3.test_command2: ossim-info -d -i -p $(OSSIM_TEST)/out/orthoigen-test/test3/t2.tif > $(OSSIM_TEST)/out/orthoigen-test/test3/t2-info.txt // Post process commands for diffs and stuff: // Histogram: test3.postprocess_command0: diff -w $(OSSIM_TEST)/out/orthoigen-test/test3/t1.his $(OSSIM_TEST)/out/orthoigen-test/test3/t2.his // Geometry: test3.postprocess_command1: diff -w $(OSSIM_TEST)/out/orthoigen-test/test3/t1-info.txt $(OSSIM_TEST)/out/orthoigen-test/test3/t2-info.txt // --- // End test3: // ---
---
First time through run with the --accept-test all option.
Note this will run the preprocess commands that make directories and so on.
This will populate the expected results section.
---
$ ossim-batch-test --accept-test all orthoigen-test.kwl Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105557.txt 100% 100% Time elapsed : 12.510 MPI running with 1 processors... Creating overviews for file: /data1/test/out/orthoigen-test/test3/t1.tif creating r1... 100% creating r2... 100% creating r3... 100% creating r4... 100% creating r5... 100% creating r6... 100% creating r7... 100% Finished... Elapsed time: 0 Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105557.txt
Now test. A run with no options runs all the test in the configuration file.
$ ossim-batch-test orthoigen-test.kwl Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105716.txt test1[0]: PASSED test2[0]: PASSED 100% 100% Time elapsed : 9.574 MPI running with 1 processors... Creating overviews for file: /data1/test/out/orthoigen-test/test3/t2.tif creating r1... 100% creating r2... 100% creating r3... 100% creating r4... 100% creating r5... 100% creating r6... 100% creating r7... 100% Finished... Elapsed time: 1 test3[0]: PASSED test3[1]: PASSED Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105716.txt
Contents of log file:
// --- // ossim-batch-test log: // date format = yyyymmddhhmmss //--- start_time: 20111216105716 config_file: /data1/test/orthoigen-test.kwl ---------------------------------------------------------------------- processing_test: test1 description: Test height for the center of the image for test2 and test3. preProcessFlag: 0 expectedFlag: 0 testFlag: 1 postProcessFlag: 1 cleanFlag: 0 executing command: ossim-info --height -42.8508 147.2537 > /data1/test/out/orthoigen-test/test1/height.txt begin: 20111216105716 end: 20111216105716 test1[0]: elapsed time in seconds: 0.2918 return status: 0 executing command: diff -w /data1/test/exp/orthoigen-test/test1/height.txt /data1/test/out/orthoigen-test/test1/height.txt > /tmp/tmp.txt return status: 0 test1[0]: PASSED end_test: test1 ---------------------------------------------------------------------- ---------------------------------------------------------------------- processing_test: test2 description: Test geometry for GeoEye NITF with RPC model. preProcessFlag: 0 expectedFlag: 0 testFlag: 1 postProcessFlag: 1 cleanFlag: 0 executing command: ossim-info -d -i -p /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > /data1/test/out/orthoigen-test/test2/info.txt begin: 20111216105716 end: 20111216105717 test2[0]: elapsed time in seconds: 0.3250 return status: 0 executing command: diff -w /data1/test/exp/orthoigen-test/test2/info.txt /data1/test/out/orthoigen-test/test2/info.txt > /tmp/tmp.txt return status: 0 test2[0]: PASSED end_test: test2 ---------------------------------------------------------------------- ---------------------------------------------------------------------- processing_test: test3 description: Test orthorectification of a GeoEye NITF with RPC model. preProcessFlag: 0 expectedFlag: 0 testFlag: 1 postProcessFlag: 1 cleanFlag: 0 executing command: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf /data1/test/out/orthoigen-test/test3/t2.tif begin: 20111216105717 end: 20111216105726 test3[0]: elapsed time in seconds: 9.6004 return status: 0 executing command: ossim-img2rr -r --create-histogram /data1/test/out/orthoigen-test/test3/t2.tif begin: 20111216105726 end: 20111216105727 test3[1]: elapsed time in seconds: 0.7058 return status: 0 executing command: ossim-info -d -i -p /data1/test/out/orthoigen-test/test3/t2.tif > /data1/test/out/orthoigen-test/test3/t2-info.txt begin: 20111216105727 end: 20111216105727 test3[2]: elapsed time in seconds: 0.3115 return status: 0 executing command: diff -w /data1/test/out/orthoigen-test/test3/t1.his /data1/test/out/orthoigen-test/test3/t2.his > /tmp/tmp.txt return status: 0 test3[0]: PASSED executing command: diff -w /data1/test/out/orthoigen-test/test3/t1-info.txt /data1/test/out/orthoigen-test/test3/t2-info.txt > /tmp/tmp.txt return status: 0 test3[1]: PASSED end_test: test3 ---------------------------------------------------------------------- stop_time: 20111216105727 total elapsed time in seconds: 11.2535
Failure Example
The expected results were edited to produce a failure, i.e. tie point shift:
Run the test:
$ ossim-batch-test orthoigen-test.kwl Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216110702.txt test1[0]: PASSED test2[0]: PASSED 100% 100% Time elapsed : 10.525 MPI running with 1 processors... Creating overviews for file: /data1/test/out/orthoigen-test/test3/t2.tif creating r1... 100% creating r2... 100% creating r3... 100% creating r4... 100% creating r5... 100% creating r6... 100% creating r7... 100% Finished... Elapsed time: 1 test3[0]: PASSED test3[1]: FAILED Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216110702.txt
Contents of log file:
// --- // ossim-batch-test log: // date format = yyyymmddhhmmss //--- start_time: 20111216110702 config_file: /data1/test/orthoigen-test.kwl ---------------------------------------------------------------------- processing_test: test1 description: Test height for the center of the image for test2 and test3. preProcessFlag: 0 expectedFlag: 0 testFlag: 1 postProcessFlag: 1 cleanFlag: 0 executing command: ossim-info --height -42.8508 147.2537 > /data1/test/out/orthoigen-test/test1/height.txt begin: 20111216110702 end: 20111216110703 test1[0]: elapsed time in seconds: 0.2289 return status: 0 executing command: diff -w /data1/test/exp/orthoigen-test/test1/height.txt /data1/test/out/orthoigen-test/test1/height.txt > /tmp/tmp.txt return status: 0 test1[0]: PASSED end_test: test1 ---------------------------------------------------------------------- ---------------------------------------------------------------------- processing_test: test2 description: Test geometry for GeoEye NITF with RPC model. preProcessFlag: 0 expectedFlag: 0 testFlag: 1 postProcessFlag: 1 cleanFlag: 0 executing command: ossim-info -d -i -p /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > /data1/test/out/orthoigen-test/test2/info.txt begin: 20111216110703 end: 20111216110703 test2[0]: elapsed time in seconds: 0.2582 return status: 0 executing command: diff -w /data1/test/exp/orthoigen-test/test2/info.txt /data1/test/out/orthoigen-test/test2/info.txt > /tmp/tmp.txt return status: 0 test2[0]: PASSED end_test: test2 ---------------------------------------------------------------------- ---------------------------------------------------------------------- processing_test: test3 description: Test orthorectification of a GeoEye NITF with RPC model. preProcessFlag: 0 expectedFlag: 0 testFlag: 1 postProcessFlag: 1 cleanFlag: 0 executing command: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf /data1/test/out/orthoigen-test/test3/t2.tif begin: 20111216110703 end: 20111216110713 test3[0]: elapsed time in seconds: 10.5560 return status: 0 executing command: ossim-img2rr -r --create-histogram /data1/test/out/orthoigen-test/test3/t2.tif begin: 20111216110713 end: 20111216110714 test3[1]: elapsed time in seconds: 0.5340 return status: 0 executing command: ossim-info -d -i -p /data1/test/out/orthoigen-test/test3/t2.tif > /data1/test/out/orthoigen-test/test3/t2-info.txt begin: 20111216110714 end: 20111216110714 test3[2]: elapsed time in seconds: 0.2372 return status: 0 executing command: diff -w /data1/test/out/orthoigen-test/test3/t1.his /data1/test/out/orthoigen-test/test3/t2.his > /tmp/tmp.txt return status: 0 test3[0]: PASSED executing command: diff -w /data1/test/out/orthoigen-test/test3/t1-info.txt /data1/test/out/orthoigen-test/test3/t2-info.txt > /tmp/tmp.txt return status: 256 test3[1]: FAILED diff -w /data1/test/out/orthoigen-test/test3/t1-info.txt /data1/test/out/orthoigen-test/test3/t2-info.txt > /tmp/tmp.txt output follows: 83c83 < image0.projection.tie_point_xy: (147.1657,-42.7826) --- > image0.projection.tie_point_xy: (147.165760750085,-42.7826084701764) end_test: test3 ---------------------------------------------------------------------- stop_time: 20111216110714 total elapsed time in seconds: 11.8294
Accept the new tie point as valid:
$ ossim-batch-test --accept-test test3 orthoigen-test.kwlLogging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111011.txt test1: disabled test2: disabled 100% 100% Time elapsed : 12.526 MPI running with 1 processors... Creating overviews for file: /data1/test/out/orthoigen-test/test3/t1.tif creating r1... 100% creating r2... 100% creating r3... 100% creating r4... 100% creating r5... 100% creating r6... 100% creating r7... 100% Finished... Elapsed time: 0 Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111011.txt
Re-run test3:
$ ossim-batch-test --run-test test3 orthoigen-test.kwl Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111102.txt test1: disabled test2: disabled 100% 100% Time elapsed : 12.504 MPI running with 1 processors... Creating overviews for file: /data1/test/out/orthoigen-test/test3/t2.tif creating r1... 100% creating r2... 100% creating r3... 100% creating r4... 100% creating r5... 100% creating r6... 100% creating r7... 100% Finished... Elapsed time: 1 test3[0]: PASSED test3[1]: PASSED Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111102.txt
Last modified
12 years ago
Last modified on Dec 16, 2011, 8:22:18 AM
Note:
See TracWiki
for help on using the wiki.
