ossim-batch-test

ossim-batch-test is a command line test application that can execute programs, test the results against expected results, and log the output with differences between expected results and run times. ossim-batch-test requires a test configuration file for input and generates a log file on output. Multiple test can be in a single configuration file. A single test, all test or multiple test can be ran with a single ossim-batch-test command. All times are logged for each command. So you can easily test for changes in output and changes in output times. This application is written in c++ so it's portable on windows, linux, MACs. Typically to switch from one operating system to another the file paths and diff commands have to be query / replaced in config file. On linux, MAC typically "diff" is used. On windows "fc" works well.

Example of ossim-orthoigen test on a NITF image with RPC model:

For this test three things:
1) The height about the center of the test image. Because this image has a RPC sensor model which is affected by elevation, the first test is the height at the center of the image. If this changes the output bounds will also change.
2) The initial output of ossim-info -d -i -p of the source test image. This alone will test many changes in the code.
3) The orthoigen output. Test reprojecting image to an output file. Testing the histogram checks for pixel changes. Testing the ossim-info -d -i -p <output-image> tests for geometry changes.

From here on out commands start with $ and comments start with .

This example was generated on linux. Windows example coming soon...

Some things to note. Test image is under OSSIM_TEST environment variable:

$ env | grep OSSIM_TEST
OSSIM_TEST=/data1/test
$ pwd
/data1/test
$ ls source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/
5V090205M0001912264B220000100072M_001508507.his
5V090205M0001912264B220000100072M_001508507.ntf
5V090205M0001912264B220000100072M_001508507.ovr
Single_Organization_license.txt

Generate a configuration file template.

$ ossim-batch-test -w orthoigen-test.kwl
Wrote file: orthoigen-test.kwl
$ 

Edit as required.
Contents of this file:
$ cat orthoigen-test.kwl

// ---
// File: orthoigen-test.kwl
// Generated by command: ossim-batch-test -w orthoigen-test.kwl
// Description: ossim-batch-test config file.
// ---

// ---
// Control flags:
// Use 1 or true, 0 or false to turn on and off sections.
// Typically preprocess and expected results only turned on for first run.
// Clean commands should erase anything created by this test.
// Command order if flag turned on:
// 1) clean, 2) pre-process, 3) expected, 4) test, 5) post-process
// ---

// ---
// NOTES:
// 1) Use forward slash, slash(c++ comment) at beginning of line for comments.
// 2) For environment variables use $(YOUR_VARIABLE).  They will be expanded at
//    run time if valid.
// 3) Clean, preprocess, run, and accept test sections can be switched on or off
//    at run time.  These override config file flags.  Optional arguments are:
//    --clean-test
//    --preprocess-test
//    --run-test
//    --accept-test
//    Multiple tests can be entered by quoting string of space separated
//    test, e.g. "test1 test2". To accept all tests use "all" for test.
// ---

// Where you want the log files to go:
log_directory: $(OSSIM_TEST)/logs/osgeo

// ---
// Begin test1:
// ---
test1.name: test1
test1.description: Test height for the center of the image for test2 and test3.

// Controls/turns on/off whole test (all sections):
test1.enabled: 1

// Individual control flags:
test1.run_clean_commands: 0
test1.run_preprocessing_commands: 0
test1.run_expected_results_commands: 0
test1.run_test_commands: 1
test1.run_postprocessing_commands: 1

// Temp file to catch diff output.
test1.temp_file: /tmp/tmp.txt

// Clean up commands.
test1.clean_command0: rm -rf $(OSSIM_TEST)/exp/orthoigen-test/test1
test1.clean_command1: rm -rf $(OSSIM_TEST)/out/orthoigen-test/test1

// Pre-process commands:
test1.preprocess_command0: mkdir -p $(OSSIM_TEST)/exp/orthoigen-test/test1
test1.preprocess_command1: mkdir -p $(OSSIM_TEST)/out/orthoigen-test/test1
test1.preprocess_command1: mkdir -p $(OSSIM_TEST)/logs/osgeo

// Commands to generate expected results:

// Since test2 and test3 are dependent on elevation test the center of the image.
test1.expected_results_command0: ossim-info --height -42.8508 147.2537 > $(OSSIM_TEST)/exp/orthoigen-test/test1/height.txt

// The actual commands to test:

test1.test_command0: ossim-info --height -42.8508 147.2537 > $(OSSIM_TEST)/out/orthoigen-test/test1/height.txt

// Post process commands for diffs and stuff:

test1.postprocess_command0: diff -w $(OSSIM_TEST)/exp/orthoigen-test/test1/height.txt $(OSSIM_TEST)/out/orthoigen-test/test1/height.txt
 
// ---
// End test1:
// ---

// ---
// Begin test2:
// ---

test2.name: test2
test2.description: Test geometry for GeoEye NITF with RPC model.

// Controls/turns on/off whole test (all sections):
test2.enabled: 1

// Individual control flags:
test2.run_clean_commands: 0
test2.run_preprocessing_commands: 0
test2.run_expected_results_commands: 0
test2.run_test_commands: 1
test2.run_postprocessing_commands: 1

// Temp file to catch diff output.
test2.temp_file: /tmp/tmp.txt

// Clean up commands.
test2.clean_command0: rm -rf $(OSSIM_TEST)/exp/orthoigen-test/test2
test2.clean_command1: rm -rf $(OSSIM_TEST)/out/orthoigen-test/test2

// Pre-process commands:
test2.preprocess_command0: mkdir -p $(OSSIM_TEST)/exp/orthoigen-test/test2
test2.preprocess_command1: mkdir -p $(OSSIM_TEST)/out/orthoigen-test/test2
test2.preprocess_command1: mkdir -p $(OSSIM_TEST)/logs/osgeo

// Commands to generate expected results:
test2.expected_results_command0: ossim-info -d -i -p $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > $(OSSIM_TEST)/exp/orthoigen-test/test2/info.txt

// The actual commands to test:
test2.test_command0: ossim-info -d -i -p $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > $(OSSIM_TEST)/out/orthoigen-test/test2/info.txt

// Post process commands for diffs and stuff:
test2.postprocess_command0: diff -w $(OSSIM_TEST)/exp/orthoigen-test/test2/info.txt $(OSSIM_TEST)/out/orthoigen-test/test2/info.txt

// ---
// End test2:
// ---

// ---
// Begin test3:
// ---
test3.name: test3
test3.description: Test orthorectification of a GeoEye NITF with RPC model.

// Controls/turns on/off whole test (all sections):
test3.enabled: 1

// Individual control flags:
test3.run_clean_commands: 0
test3.run_preprocessing_commands: 0
test3.run_expected_results_commands: 0
test3.run_test_commands: 1
test3.run_postprocessing_commands: 1

// Temp file to catch diff output.
test3.temp_file: /tmp/tmp.txt

// Clean up commands.
test3.clean_command0: rm -rf $(OSSIM_TEST)/exp/orthoigen-test/test3
test3.clean_command1: rm -rf $(OSSIM_TEST)/out/orthoigen-test/test3

// Pre-process commands:
test3.preprocess_command0: mkdir -p $(OSSIM_TEST)/exp/orthoigen-test/test3
test3.preprocess_command1: mkdir -p $(OSSIM_TEST)/out/orthoigen-test/test3
test3.preprocess_command1: mkdir -p $(OSSIM_TEST)/logs/osgeo

// Commands to generate expected results:

// Put the output in the "out" directory as the "exp" may be under svn for small things:

// Ortho:
test3.expected_results_command0: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf $(OSSIM_TEST)/out/orthoigen-test/test3/t1.tif

// Histogram:
test3.expected_results_command1: ossim-img2rr -r --create-histogram $(OSSIM_TEST)/out/orthoigen-test/test3/t1.tif

// Geometry:
test3.expected_results_command2: ossim-info -d -i -p $(OSSIM_TEST)/out/orthoigen-test/test3/t1.tif > $(OSSIM_TEST)/out/orthoigen-test/test3/t1-info.txt

// Copy to exp directory which is under svn control:
test3.expected_results_command3: cp $(OSSIM_TEST)/out/orthoigen-test/test3/t1.his $(OSSIM_TEST)/exp/orthoigen-test/test3/t1.his

test3.expected_results_command4: cp $(OSSIM_TEST)/out/orthoigen-test/test3/t1-info.txt $(OSSIM_TEST)/exp/orthoigen-test/test3/t1-info.txt

// The actual commands to test:
// Ortho:
test3.test_command0: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear $(OSSIM_TEST)/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf $(OSSIM_TEST)/out/orthoigen-test/test3/t2.tif

// Histogram:
test3.test_command1: ossim-img2rr -r --create-histogram $(OSSIM_TEST)/out/orthoigen-test/test3/t2.tif

// Geometry:
test3.test_command2: ossim-info -d -i -p $(OSSIM_TEST)/out/orthoigen-test/test3/t2.tif > $(OSSIM_TEST)/out/orthoigen-test/test3/t2-info.txt

// Post process commands for diffs and stuff:

// Histogram:
test3.postprocess_command0: diff -w $(OSSIM_TEST)/out/orthoigen-test/test3/t1.his $(OSSIM_TEST)/out/orthoigen-test/test3/t2.his

// Geometry:
test3.postprocess_command1: diff -w $(OSSIM_TEST)/out/orthoigen-test/test3/t1-info.txt $(OSSIM_TEST)/out/orthoigen-test/test3/t2-info.txt

// ---
// End test3:
// ---

---
First time through run with the --accept-test all option.
Note this will run the preprocess commands that make directories and so on.
This will populate the expected results section.
---

$ ossim-batch-test --accept-test all orthoigen-test.kwl
Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105557.txt
100%
100%
Time elapsed : 12.510
MPI running with 1 processors...
Creating overviews for file: /data1/test/out/orthoigen-test/test3/t1.tif
creating r1...
100%
creating r2...
100%
creating r3...
100%
creating r4...
100%
creating r5...
100%
creating r6...
100%
creating r7...
100%
Finished...
Elapsed time: 0
Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105557.txt

Now test. A run with no options runs all the test in the configuration file.

$ ossim-batch-test orthoigen-test.kwl
Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105716.txt
test1[0]: PASSED
test2[0]: PASSED
100%
100%
Time elapsed : 9.574
MPI running with 1 processors...
Creating overviews for file: /data1/test/out/orthoigen-test/test3/t2.tif
creating r1...
100%
creating r2...
100%
creating r3...
100%
creating r4...
100%
creating r5...
100%
creating r6...
100%
creating r7...
100%
Finished...
Elapsed time: 1
test3[0]: PASSED
test3[1]: PASSED
Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216105716.txt

Contents of log file:

// ---
// ossim-batch-test log:
// date format = yyyymmddhhmmss
//---
start_time: 20111216105716
config_file: /data1/test/orthoigen-test.kwl

----------------------------------------------------------------------
processing_test: test1
description: Test height for the center of the image for test2 and test3.
preProcessFlag:  0
expectedFlag:    0
testFlag:        1
postProcessFlag: 1
cleanFlag:       0
executing command: ossim-info --height -42.8508 147.2537 > /data1/test/out/orthoigen-test/test1/height.txt
begin: 20111216105716
end: 20111216105716
test1[0]: elapsed time in seconds: 0.2918
return status: 0
executing command: diff -w /data1/test/exp/orthoigen-test/test1/height.txt /data1/test/out/orthoigen-test/test1/height.txt > /tmp/tmp.txt
return status: 0
test1[0]: PASSED
end_test: test1
----------------------------------------------------------------------

----------------------------------------------------------------------
processing_test: test2
description: Test geometry for GeoEye NITF with RPC model.
preProcessFlag:  0
expectedFlag:    0
testFlag:        1
postProcessFlag: 1
cleanFlag:       0
executing command: ossim-info -d -i -p /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > /data1/test/out/orthoigen-test/test2/info.txt
begin: 20111216105716
end: 20111216105717
test2[0]: elapsed time in seconds: 0.3250
return status: 0
executing command: diff -w /data1/test/exp/orthoigen-test/test2/info.txt /data1/test/out/orthoigen-test/test2/info.txt > /tmp/tmp.txt
return status: 0
test2[0]: PASSED
end_test: test2
----------------------------------------------------------------------

----------------------------------------------------------------------
processing_test: test3
description: Test orthorectification of a GeoEye NITF with RPC model.
preProcessFlag:  0
expectedFlag:    0
testFlag:        1
postProcessFlag: 1
cleanFlag:       0
executing command: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf /data1/test/out/orthoigen-test/test3/t2.tif
begin: 20111216105717
end: 20111216105726
test3[0]: elapsed time in seconds: 9.6004
return status: 0
executing command: ossim-img2rr -r --create-histogram /data1/test/out/orthoigen-test/test3/t2.tif
begin: 20111216105726
end: 20111216105727
test3[1]: elapsed time in seconds: 0.7058
return status: 0
executing command: ossim-info -d -i -p /data1/test/out/orthoigen-test/test3/t2.tif > /data1/test/out/orthoigen-test/test3/t2-info.txt
begin: 20111216105727
end: 20111216105727
test3[2]: elapsed time in seconds: 0.3115
return status: 0
executing command: diff -w /data1/test/out/orthoigen-test/test3/t1.his /data1/test/out/orthoigen-test/test3/t2.his > /tmp/tmp.txt
return status: 0
test3[0]: PASSED
executing command: diff -w /data1/test/out/orthoigen-test/test3/t1-info.txt /data1/test/out/orthoigen-test/test3/t2-info.txt > /tmp/tmp.txt
return status: 0
test3[1]: PASSED
end_test: test3
----------------------------------------------------------------------

stop_time: 20111216105727
total elapsed time in seconds: 11.2535

Failure Example

The expected results were edited to produce a failure, i.e. tie point shift:

Run the test:

$ ossim-batch-test orthoigen-test.kwl
Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216110702.txt
test1[0]: PASSED
test2[0]: PASSED
100%
100%
Time elapsed : 10.525
MPI running with 1 processors...
Creating overviews for file: /data1/test/out/orthoigen-test/test3/t2.tif
creating r1...
100%
creating r2...
100%
creating r3...
100%
creating r4...
100%
creating r5...
100%
creating r6...
100%
creating r7...
100%
Finished...
Elapsed time: 1
test3[0]: PASSED
test3[1]: FAILED
Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216110702.txt

Contents of log file:

// ---
// ossim-batch-test log:
// date format = yyyymmddhhmmss
//---
start_time: 20111216110702
config_file: /data1/test/orthoigen-test.kwl

----------------------------------------------------------------------
processing_test: test1
description: Test height for the center of the image for test2 and test3.
preProcessFlag:  0
expectedFlag:    0
testFlag:        1
postProcessFlag: 1
cleanFlag:       0
executing command: ossim-info --height -42.8508 147.2537 > /data1/test/out/orthoigen-test/test1/height.txt
begin: 20111216110702
end: 20111216110703
test1[0]: elapsed time in seconds: 0.2289
return status: 0
executing command: diff -w /data1/test/exp/orthoigen-test/test1/height.txt /data1/test/out/orthoigen-test/test1/height.txt > /tmp/tmp.txt
return status: 0
test1[0]: PASSED
end_test: test1
----------------------------------------------------------------------

----------------------------------------------------------------------
processing_test: test2
description: Test geometry for GeoEye NITF with RPC model.
preProcessFlag:  0
expectedFlag:    0
testFlag:        1
postProcessFlag: 1
cleanFlag:       0
executing command: ossim-info -d -i -p /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf > /data1/test/out/orthoigen-test/test2/info.txt
begin: 20111216110703
end: 20111216110703
test2[0]: elapsed time in seconds: 0.2582
return status: 0
executing command: diff -w /data1/test/exp/orthoigen-test/test2/info.txt /data1/test/out/orthoigen-test/test2/info.txt > /tmp/tmp.txt
return status: 0
test2[0]: PASSED
end_test: test2
----------------------------------------------------------------------

----------------------------------------------------------------------
processing_test: test3
description: Test orthorectification of a GeoEye NITF with RPC model.
preProcessFlag:  0
expectedFlag:    0
testFlag:        1
postProcessFlag: 1
cleanFlag:       0
executing command: ossim-orthoigen -t 1024 --geo-scaled -42.85 --resample-type bilinear /data1/test/source/GeoEye/GE1_Hobart_GeoStereo_NITF-NCD/5V090205M0001912264B220000100072M_001508507.ntf /data1/test/out/orthoigen-test/test3/t2.tif
begin: 20111216110703
end: 20111216110713
test3[0]: elapsed time in seconds: 10.5560
return status: 0
executing command: ossim-img2rr -r --create-histogram /data1/test/out/orthoigen-test/test3/t2.tif
begin: 20111216110713
end: 20111216110714
test3[1]: elapsed time in seconds: 0.5340
return status: 0
executing command: ossim-info -d -i -p /data1/test/out/orthoigen-test/test3/t2.tif > /data1/test/out/orthoigen-test/test3/t2-info.txt
begin: 20111216110714
end: 20111216110714
test3[2]: elapsed time in seconds: 0.2372
return status: 0
executing command: diff -w /data1/test/out/orthoigen-test/test3/t1.his /data1/test/out/orthoigen-test/test3/t2.his > /tmp/tmp.txt
return status: 0
test3[0]: PASSED
executing command: diff -w /data1/test/out/orthoigen-test/test3/t1-info.txt /data1/test/out/orthoigen-test/test3/t2-info.txt > /tmp/tmp.txt
return status: 256
test3[1]: FAILED
diff -w /data1/test/out/orthoigen-test/test3/t1-info.txt /data1/test/out/orthoigen-test/test3/t2-info.txt > /tmp/tmp.txt
output follows:
83c83
< image0.projection.tie_point_xy:  (147.1657,-42.7826)
---
> image0.projection.tie_point_xy:  (147.165760750085,-42.7826084701764)

end_test: test3
----------------------------------------------------------------------

stop_time: 20111216110714
total elapsed time in seconds: 11.8294

Accept the new tie point as valid:

$ ossim-batch-test --accept-test test3 orthoigen-test.kwlLogging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111011.txt
test1: disabled
test2: disabled
100%
100%
Time elapsed : 12.526
MPI running with 1 processors...
Creating overviews for file: /data1/test/out/orthoigen-test/test3/t1.tif
creating r1...
100%
creating r2...
100%
creating r3...
100%
creating r4...
100%
creating r5...
100%
creating r6...
100%
creating r7...
100%
Finished...
Elapsed time: 0
Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111011.txt

Re-run test3:

$ ossim-batch-test --run-test test3 orthoigen-test.kwl
Logging to file: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111102.txt
test1: disabled
test2: disabled
100%
100%
Time elapsed : 12.504
MPI running with 1 processors...
Creating overviews for file: /data1/test/out/orthoigen-test/test3/t2.tif
creating r1...
100%
creating r2...
100%
creating r3...
100%
creating r4...
100%
creating r5...
100%
creating r6...
100%
creating r7...
100%
Finished...
Elapsed time: 1
test3[0]: PASSED
test3[1]: PASSED
Wrote: /data1/test/logs/osgeo/ossim-batch-test-log-20111216111102.txt
Last modified 8 years ago Last modified on Dec 16, 2011 8:22:18 AM