Changes between Version 21 and Version 22 of GSoC/2014/TestingFrameworkForGRASS

Timestamp:

05/26/14 20:12:49 (11 years ago)

Author:

wenzeslaus

Comment:

how the test methods are invoked

GSoC/2014/TestingFrameworkForGRASS

@@ -141 +141 @@
 There is no easy way how to test that (GRASS) fatal errors are invoked when appropriate. Even if the test method (`test_*`) itself would run in separate process (not only the whole script) the process will be ended without proper reporting of the test result (considering we want detailed test results). However, since this applies also to fatal errors invoked by unintentional failure and to fatal errors, it seems that it will be necessary to invoke the test methods (`test_*`) in a separate process to at least finish the other tests and not break the final report. This might be done by function decorator so that we don't invoke new process for each function but only for those who need it (the ones using things which use `ctypes`).
 
+=== How the individual test methods are invoked ===
+
+This describes how `unittest` is doing it and we want to use as many classes (directly or by inheriting) from `unittest` as possible, so it will be probably also our system.
+
+Loading of tests:
+
+{{{
+# finds all test methods of a given test case and creates test suite
+# suite class will get a list of instances of test case class with a method name (test method)
+# getTestCaseNames == test method names
+TestLoader.loadTestsFromTestCase()
+        testCaseNames = filter(isTestMethod, dir(testCaseClass))
+    loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
+
+# finds all test cases in the module/script
+TestLoader.loadTestsFromModule()
+    for each object in module:
+        if isinstance(obj, type) and issubclass(obj, case.TestCase):
+                        tests.append(self.loadTestsFromTestCase(obj))
+    suiteClass = suite.TestSuite
+    tests = self.suiteClass(tests)
+
+# finds all tests passes them to test runner
+TestProgram.runTests()
+    self.test = self.testLoader.loadTestsFromModule(self.module)
+    self.result = testRunner.run(self.test)
+
+# triggers the top level test suite
+TextTestRunner.run(test)
+    test(result)
+}}}
+
+Running of tests (test suite can contain test suite or test case instances):
+
+{{{
+# invokes the separate tests
+# does setup and teardown for class
+TestSuite.run(result)
+    for each test:
+        test(result)
+
+# runs the test method and adds the information to the result
+# does setup and teardown for instance
+TestCase.run(result)
+}}}
+
 === Analyzing module run using Valgrind or others ===
 
 Modules (or perhaps any tests) can run with `valgrind` (probably `--tool=memcheck`). This could be done on the level of testing classes but the better option is to integrate this functionality (optional running with `valgrind`) into PyGRASS, so it could be easily usable through it. Environmental variable (GRASS_PYGRASS_VALGRIND) or additional option `valgrind_=True` (similarly to overwrite) would invoke module with `valgrind` (works for both binaries and scripts). Additional options can be passed to `valgrind` using `valgrind`'s environmental variable `$VALGRIND_OPTS`. Output would be saved in file to not interfere with module output.
 
-We may want to use also some (runtime checking) tools other than `valgrind`, for example clang/LLVM sanitizers (as for example [https://docs.python.org/devguide/clang.html Python does]). However, it is unclear how to handle more than one tool as well as it is unclear how to store the results for any of these (including `valgrind`) because one test can have multiple module calls (or none), module calls can be indirect (function in Python lib which calls a module or module calling module) and there is no standard way in `unittest` to pass additional test details.
+We may want to use also some (runtime checking) tools other than `valgrind`, for example clang/LLVM sanitizers (as for example [https://docs.python.org/devguide/clang.html Python does]) or profiling (these would be different for C/C++ and Python). However, it is unclear how to handle more than one tool as well as it is unclear how to store the results for any of these (including `valgrind`) because one test can have multiple module calls (or none), module calls can be indirect (function in Python lib which calls a module or module calling module) and there is no standard way in `unittest` to pass additional test details.
 
 PyGRASS or specialized PyGRASS module runner (function) in testing framework can have function, global variable, or environmental variable which would specify which tool should run a module (if any) and what are the parameters (besides the possibility to set parameters by environmental variable defined by the tool). The should ideally be separated from the module output and go to a file in the test output directory (and it could be later linked from/included into the main report).