Changes between Version 6 and Version 7 of GSoC/2014/TestingFrameworkForGRASS

Timestamp:

May 22, 2014, 10:27:40 PM (10 years ago)

Author:

wenzeslaus

Comment:

layout, basic invoking and others (from conversation with Soeren)

GSoC/2014/TestingFrameworkForGRASS

@@ -94 +94 @@
         # or both if self.longMessage is True (unittest.TestCase.longMessage)
         # precision should be considered too (for FCELL and DCELL but perhaps also for CELL)
+        # the actual implementation will be in separate module, so it can be reused by doctests or elsewhere
+        # this is necessary considering the number and potential complexity of functions
+        # and it is better design anyway
+
         if check sums not equal:
             self.fail(...)  # unittest.TestCase.fail
@@ -126 +130 @@
 }}}
 
-Compared to suggestion in ticket:2105#comment:4 it does not solve everything in `test_module` (`run_module`) function but it uses `self.assert*` similarly to `unittest.TestCase` which (syntactically) allows to check more then one thing.
+Compared to suggestion in ticket:2105#comment:4 it does not solve everything in `test_module` (`run_module`) function but it uses `self.assert*` similarly to `unittest.TestCase` which (syntactically) allows to check more then one thing. 
+
+Test scripts must have module/package character (as unittests requires for test discovery). This applies true for unittests and doctests, no exceptions. Doctests (inside normal module code or in separate doc) will be wrapped as `unittest` test cases (in `testsuite` directory). There is a [https://docs.python.org/2/library/doctest.html#unittest-api standard way] to do it. To have the possibility of import, all the GRASS Python libraries shouldn't do anything fancy at import time. For example, doctests currently don't work with `grass.script` unless you call [source:grass/trunk/gui/wxpython/core/toolboxes.py?rev=60218#L630 a set of functions] to deal with function `_` because of installing translate function as buildin `_` function while `_` is used also in `doctest`. (This is fixed for GUI but still applies to Python libraries).
+
+
+=== Analyzing module run using valgrind or others ===
 
 Modules (or any tests?) can run with `valgrind` (probably `--tool=memcheck`). This could be done on the level of testing classes but the better option is to integrate this functionality (optional running with `valgrind`). Environmental variable (GRASS_PYGRASS_VALGRIND) or additional option `valgrind_=True` (similarly to overwrite) would invoke module with `valgrind` (works for both binaries and scripts). Additional options can be passed to `valgrind` using `valgrind`'s environmental variable `$VALGRIND_OPTS`. Output would be saved in file to not interfere with module output.
 
 We may want to use also some (runtime checking) tools other than `valgrind`, for example clang/LLVM sanitizers (as for example [https://docs.python.org/devguide/clang.html Python does]). However, it is unclear how to handle more than one tool as well as it is unclear how to store the results for any of these (including `valgrind`) because one test can have multiple module calls (or none), module calls can be indirect (function in Python lib which calls a module or module calling module) and there is no standard way in `unittest` to pass additional test details.
+
+PyGRASS or specialized PyGRASS module runner (function) in testing framework can have function, global variable, or environmental variable which would specify which tool should run a module (if any) and what are the parameters (besides the possibility to set parameters by environmental variable defined by the tool). The should ideally be separated from the module output and go to a file in the test output directory (and it could be later linked from/included into the main report).
 
 
@@ -156 +167 @@
 The [https://docs.python.org/2/library/unittest.html#unittest.TestLoader.discover unittest.TestLoader.discover] function requires that module names ''are importable (i.e. are valid Python identifiers)''. Consequently, names of files with tests should contain dots (except for the `.py` suffix).
 
-Methods with tests must start with `test_` to be recognized by the [https://docs.python.org/2/library/unittest.html#organizing-test-code unittest] framework.
+Methods with tests must start with `test_` to be recognized by the [https://docs.python.org/2/library/unittest.html#organizing-test-code unittest] framework (with default setting but there is no reason to not keep this convention).
+
+Name for directory with test is "testsuite". It also fits to how `unittest` is using this term (set of test cases and other test suites). "test" and "tests" is simpler and you can see it, for example [http://hg.python.org/cpython/file/2145593d108d/Lib/unittest/test in Python], but might be too general. "unittest" would confuse with the module `unittest`.
+
+== Layout of directories and files ==
+
+Test scripts are located in a dedicated directory within module or library directories. All reference files or additionally needed data should be located there.
+
+The same directory as tested would work well for one or two Python files but not for number of reference files. In case of C/C++ this would mean mixing Python and C/C++ files in one directory which is strange. One directory in root with separate tree is something which would not work either because tests are not close enough to actual code to be often run and extended when appropriate.
+
+== Invoking tests ==
+
+Test scripts will work when directly executed from command line and when executed from the make system.
+
+Tests should be executable by itself (i.e. should they have `main()` function) to encourage running them often. It is not clear if this should be be used by the testing framework when running multiple tests or if it is better to import.
+
+
+=== Example run ===
+
+{{{
+cd lib/python/temporal
+}}}
+
+Now there are two options to run the tests. First execution by hand in my current location:
+
+{{{
+cd testsuite
+for test in `ls *.py` ; do
+    python ${test} >> /tmp/tgis_lib.txt 2>&1
+done
+}}}  
+
+The test output will be written to stdout and stderr (piped to a file in this case).
+
+Second option is an execution by the make system (still in `lib/python/temporal`):
+
+{{{
+make tests
+}}}
+
+
+== Locations, mapsets and data ==
+
+== Weekly reports ==