Changes between Version 33 and Version 34 of GSoC/2014/TestingFrameworkForGRASS

Timestamp:

Jun 4, 2014, 7:42:35 PM (10 years ago)

Author:

wenzeslaus

Comment:

code coverage as an available addition

GSoC/2014/TestingFrameworkForGRASS

@@ -230 +230 @@
 It is not clear if `valgrind` would be applied even for library tests. This would require to run the testing process with `valgrind`. But since it needs to run separately anyway, this can be done. In case we would like to run with `valgrind` test function (`test_*`), the testing framework would have to contain the `valgrind` running function anyway. The function would run the test function as subprocess (which is anyway necessary to deal with process termination). The advantage of integration into PyGRASS wouldn't be so big in this case. But even in the case of separate function for running subprocess, a PyGRASS `Module` class will be used to pass the parameters.
 
+=== Other source code analyses ===
+
 Concerning static source code analysis, it seems that it is better to do it separately because it depends more on source code files or, in case of Python, modules and packages while tests seems to have their own structure. Also static source code analysis is not related to testing itself, although it is, for sure, part of quality assurance. However, it must be noted that some part of analysis can be one test case as suggested by [http://pep8.readthedocs.org/en/latest/advanced.html#automated-tests pep8 documentation] (test fails if code style requirements are not fulfilled). This could integrate `pep8` with testing framework in an elegant way but it would require each `testsuite` to explicitly request this check.
+
+The determining code coverage of the tests must be done during testing. Speaking about Python, some more advanced testing frameworks includes code coverage tools in them, namely [http://nedbatchelder.com/code/coverage/ coverage]. Since `unittest` does not include `coverage`, GRASS will have to incorporate `coverage` into its testing framework. The basic usage is rather simple. If we consider that all is running in one Python process, we can just put the following (rather messy) code into some module which will be loaded before the tests will start (the main invoking script would be of course the best place):
+{{{
+#!python
+import coverage
+cov = coverage.coverage(omit="*testsuite*")
+cov.start()
+
+def endcov(cov):
+    cov.stop()
+    cov.html_report(directory='testcodecoverage')
+
+import atexit
+atexit.register(endcov, cov)
+}}}
+However, the HTML reports needs some polishing (especially the names of modules and paths) and moreover, the reports will be spitted by running different tests as separate processes. It it possible to link to these subreports and it might be even advantageous. But what is really an issue is running separate methods as separate processes, so the report is split in the way that it we have two (or more) reports, each of them reporting incomplete coverage but if merged together the coverage might be full. The merging seems like a challenge and would be probably hard on the level of HTML, so this would require using more low level API which is of course much harder. The default HTML report would be separate from the testing report, however this is not an issue and it might be even possible to add some links (with percentages) at proper places into the testing report.
 === Dependencies ===