Changes between Version 13 and Version 14 of GSoC/2014/TestingFrameworkForGRASS

Timestamp:

May 25, 2014, 5:35:24 PM (10 years ago)

Author:

wenzeslaus

Comment:

explain more valgrind for libraries

GSoC/2014/TestingFrameworkForGRASS

@@ -142 +142 @@
 === Analyzing module run using Valgrind or others ===
 
-Modules (or any tests?) can run with `valgrind` (probably `--tool=memcheck`). This could be done on the level of testing classes but the better option is to integrate this functionality (optional running with `valgrind`). Environmental variable (GRASS_PYGRASS_VALGRIND) or additional option `valgrind_=True` (similarly to overwrite) would invoke module with `valgrind` (works for both binaries and scripts). Additional options can be passed to `valgrind` using `valgrind`'s environmental variable `$VALGRIND_OPTS`. Output would be saved in file to not interfere with module output.
+Modules (or perhaps any tests) can run with `valgrind` (probably `--tool=memcheck`). This could be done on the level of testing classes but the better option is to integrate this functionality (optional running with `valgrind`) into PyGRASS, so it could be easily usable through it. Environmental variable (GRASS_PYGRASS_VALGRIND) or additional option `valgrind_=True` (similarly to overwrite) would invoke module with `valgrind` (works for both binaries and scripts). Additional options can be passed to `valgrind` using `valgrind`'s environmental variable `$VALGRIND_OPTS`. Output would be saved in file to not interfere with module output.
 
 We may want to use also some (runtime checking) tools other than `valgrind`, for example clang/LLVM sanitizers (as for example [https://docs.python.org/devguide/clang.html Python does]). However, it is unclear how to handle more than one tool as well as it is unclear how to store the results for any of these (including `valgrind`) because one test can have multiple module calls (or none), module calls can be indirect (function in Python lib which calls a module or module calling module) and there is no standard way in `unittest` to pass additional test details.
@@ -148 +148 @@
 PyGRASS or specialized PyGRASS module runner (function) in testing framework can have function, global variable, or environmental variable which would specify which tool should run a module (if any) and what are the parameters (besides the possibility to set parameters by environmental variable defined by the tool). The should ideally be separated from the module output and go to a file in the test output directory (and it could be later linked from/included into the main report).
 
-Having output from many modules can be confusing (e.g. we run `r.info` before actually running our module). It would be ideal if it would be possible to specify which modules called in the test should run with `valgrind` or other tool. API for this may, however, interfere with the API for global settings of running with these tools. It is not clear if `valgrind` would be applied even for library tests. This would require to run the testing process with `valgrind`. But since it needs to run separately anyway, this can be done.
+Having output from many modules can be confusing (e.g. we run `r.info` before actually running our module). It would be ideal if it would be possible to specify which modules called in the test should run with `valgrind` or other tool. API for this may, however, interfere with the API for global settings of running with these tools.
+
+It is not clear if `valgrind` would be applied even for library tests. This would require to run the testing process with `valgrind`. But since it needs to run separately anyway, this can be done. In case we would like to run with `valgrind` test function (`test_*`), the testing framework would have to contain the `valgrind` running function anyway. The function would run the test function as subprocess (which is anyway necessary to deal with process termination). The advantage of integration into PyGRASS wouldn't be so big in this case. But even in the case of separate function for running subprocess, a PyGRASS `Module` class will be used to pass the parameters.
+