| | 996 | === Final report === |
| | 997 | |
| | 998 | It was not required by GSoC but I wanted to do some final report to summarize what was done and what could be done in the future. So, here it is. |
| | 999 | |
| | 1000 | Ways to run tests: |
| | 1001 | * each file separately in the current location and mapset you are in |
| | 1002 | * good for testing the code you are just writing |
| | 1003 | * good for writing the test |
| | 1004 | * running all tests in subdirectories (all test files in all testsuite directories) |
| | 1005 | * generates a report |
| | 1006 | * runs in GRASS session but does not touch the current mapset |
| | 1007 | * switches to specified location and mapset |
| | 1008 | * generates detailed report |
| | 1009 | * using higher level script to run tests from outside GRASS session |
| | 1010 | |
| | 1011 | Ways to write tests: |
| | 1012 | * using GRASS Python package `gunittest` |
| | 1013 | * works in the same way as `unittest` but adds GRASS-specific functionality |
| | 1014 | * good tools for test author |
| | 1015 | * allows generation of HTML reports |
| | 1016 | * enables fine control over execution of GRASS modules in the future |
| | 1017 | * this is the preferred way, other ways are discouraged |
| | 1018 | * using standard Python `unittest` |
| | 1019 | * only general testing functionality available |
| | 1020 | * rich report will not be generated |
| | 1021 | * no particular reasons for using this |
| | 1022 | * any Python script |
| | 1023 | * Python syntax is used which is better then Shell |
| | 1024 | * no special reports or functions are provided |
| | 1025 | * not recommended |
| | 1026 | * Shell scripts |
| | 1027 | * provided to accommodate existing tests |
| | 1028 | * better Shell test then no test |
| | 1029 | * probably won't run on MS Windows |
| | 1030 | * not recommended |
| | 1031 | |
| | 1032 | When writing tests, `gunittest` package allows in a convenient way to: |
| | 1033 | * compare map statistics or info to a given reference |
| | 1034 | * compare raster and 3D raster maps with each other |
| | 1035 | * compare vector maps using different ways |
| | 1036 | * compare numbers and texts |
| | 1037 | |
| | 1038 | Generated test reports contain: |
| | 1039 | * key-value file with results for each test file |
| | 1040 | * HTML report with: |
| | 1041 | * detailed page with important information for each test file |
| | 1042 | * summary pages for each `testsuite` directory |
| | 1043 | * summary pages for all tests according to test files, `testsuite` directories, ... |
| | 1044 | |
| | 1045 | |
| | 1046 | Things which should be solved in the short term: |
| | 1047 | * tests should specify the location they can run in |
| | 1048 | * some code already exists for this but parsing of LOCATION variable or tag needs to added |
| | 1049 | * the missing piece is determination of which ''location types'' we need (e.g., same maps but different projections) |
| | 1050 | * encapsulate removal of maps and other files |
| | 1051 | * test tests on more platform |
| | 1052 | * Mac OS X should just work |
| | 1053 | * MS Windows might be challenging at the beginning but everything is in Python, so at least basics should work |
| | 1054 | * Python 2.6 compatibility |
| | 1055 | * there is `unittest2` for backwards compatibility but it seems not trivial |
| | 1056 | |
| | 1057 | Long-term plans: |
| | 1058 | * integrate `valgrind`, and other tools such as profiling tools |
| | 1059 | * provide coverage for Python code, C modules and library |
| | 1060 | * introduce static test of source code (probably separately) such as `pylint`, `cppcheck` or custom regular expression |
| | 1061 | |
| | 1062 | Smaller or bigger things which can be done but are not necessary: |
| | 1063 | * parallelization (can be done at different levels, higher levels are easier and might give better speedup) |
| | 1064 | * store test results in a database |
| | 1065 | * store test results in a standardized XML for better integration with other systems |
| | 1066 | * use UTC times everywhere |
| | 1067 | * add more information about system and environment to test reports |
| | 1068 | * create latest test results page in report |
| | 1069 | * create per-module result page in report with list of not tested modules |
| | 1070 | * link the tested code in reports |
| | 1071 | * output maps and their differences as images to report |
| | 1072 | * make removal or maps and other files optional |
| | 1073 | * standardize precision handling, now every test needs to decide individually (relative, absolute, float, integer) |
| | 1074 | * create better interface for running individual tests, all tests and make with appropriate defaults |
| | 1075 | * create interface for test filtering or running list of tests |
| | 1076 | * create method for running part of the tests and integrating results into a bigger report (to test just the part of the source code which has changed) |
| | 1077 | * check for invalid tests: test files without "main" or test methods without assert function calls |
| | 1078 | * use some !JavaScript interactive graphs for the overviews |
| | 1079 | * create a image with test results which can be included to other websites |
| | 1080 | * support report with different system configurations (e.g. compiler) |
| | 1081 | |
| | 1082 | It would be also interesting to compare this testing framework to the older "test suite", for example which features are missing. |
| | 1083 | |
| | 1084 | It is probably good to mention that documentation with examples is in Python API Sphinx documentation. You can get it by `make sphinxdoc`. Also there is a lot of tests already written, so you can get inspiration there (search for `testsuite` directories) but read the documentation too, to understand basic concepts. |
| | 1085 | |
| | 1086 | And there is probably no need to mention that although there is a lot of tests which can serve you as examples there is not enough tests to cover all GRASS functionality, so please contribute. Start with the modules, you are using. You can also contribute by running the tests on your platform and specific system configuration. I'm running the tests once a day and the HTML report is available online. |
| | 1087 | |
| | 1088 | Thanks to the test, we have already discovered several bugs soon after they were introduced (the day of after the commit). Additionally, some older bugs were discovered by thorough testing. This is very encouraging to me and I hope also to others. |
