Missing guidelines for testing

[wenzeslaus]

I miss any kind of guide how to create test. This applies for the main source code and the addons, C/C++ and Python, modules and code itself.

There are tests of temporal modules but it is not currently described anywhere how they are organized.

There is also a testsuite directory in trunk but it is not maintained.

I've also noticed some tests running during compilation but I was never able to find out what it is.

Some Python functions can be tested using doctest and Python developers usually know how to use it. However, it is not clear how to ensure calling from make.

There is a page about the test framework we wish to have:

• ​http://grasswiki.osgeo.org/wiki/Test_Suite

But since it does not exists the page creates just confusion.

There is also a mailing list about testing:

• ​http://lists.osgeo.org/mailman/listinfo/grass-qa

But it is not active.

We probably also miss the testing infrastructure itself but that's another bigger ticket. But some things exists already, so we can use them.

Maybe, we need Tests component in Trac?

[huhabla]

Replying to wenzeslaus:

I miss any kind of guide how to create test. This applies for the main source code and the addons, C/C++ and Python, modules and code itself.

Have a look at: ​http://grasswiki.osgeo.org/wiki/Test_Suite

It is still an idea and guideline, but there are several tests scripts in GRASS that implement the guideline concept for GRASS modules.

There are tests of temporal modules but it is not currently described anywhere how they are organized.

The temporal module tests follow the test ideas described in: ​http://grasswiki.osgeo.org/wiki/Test_Suite

There is also a testsuite directory in trunk but it is not maintained.

I've also noticed some tests running during compilation but I was never able to find out what it is.

I am not sure what tests you mean? There are three C-libraries that implement unit tests for several library functions: gpde, gmath and raster3d libraries. These tests are not compiled by default. They must be compiled by hand and then executed in a GRASS location.

Some Python functions can be tested using doctest and Python developers usually know how to use it. However, it is not clear how to ensure calling from make.

There is no test mechanism in the make system yet and no guideline how to implement Python doctests.

There is a page about the test framework we wish to have:

• ​http://grasswiki.osgeo.org/wiki/Test_Suite

But since it does not exists the page creates just confusion.

This page describes in very detail how the testsuite can be implemented for modules and libraries. Python doctests and how to implement automated testing in the make system are not covered. It includes a guideline howto implement tests for modules based on shell scripts. Hence it is unclear to me how this leads to confusion?

However, i am all for a complete new test suite design. I would suggest to implement all module and Python library tests in Python, so that these tests can be executed independently from the OS or the used command line interpreter, with or without the make system. The next big issue is how to automatically test the GUI?

I would like to mentor this as a GSoC project in 2014. I have some experience with test suites ... .[1] And i have the experience that nobody cares to implement module or library tests ... .

Best regards Soeren

[1] ​https://code.google.com/p/grass6-test-suite/

There is also a mailing list about testing:

• ​http://lists.osgeo.org/mailman/listinfo/grass-qa

But it is not active.

We probably also miss the testing infrastructure itself but that's another bigger ticket. But some things exists already, so we can use them.

Maybe, we need Tests component in Trac?

[neteler]

Replying to huhabla:

And i have the experience that nobody cares to implement module or library tests ... .

It would be great fun to get this Makefile target globally implemented:

make test

See ​http://grasswiki.osgeo.org/wiki/Test_Suite#Make_system

Replying to wenzeslaus:

There is also a mailing list about testing:

• ​http://lists.osgeo.org/mailman/listinfo/grass-qa

But it is not active.

It was active for some years being the outcome of a series of articles:

A Novel approach to Optimize Clone Refactoring Activity ​http://doi.acm.org/10.1145/1143997.1144312

A Language-Independent Software Renovation Framework ​http://dx.doi.org/10.1016/j.jss.2004.03.033

Moving to smaller libraries via clustering and genetic algorithms ​http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1192439

Maybe, we need Tests component in Trac?

Added.

[wenzeslaus]

Replying to huhabla:

Replying to wenzeslaus:

There is a page about the test framework we wish to have:

• ​http://grasswiki.osgeo.org/wiki/Test_Suite

But since it does not exists the page creates just confusion.

This page describes in very detail how the testsuite can be implemented for modules and libraries. Python doctests and how to implement automated testing in the make system are not covered. It includes a guideline howto implement tests for modules based on shell scripts. Hence it is unclear to me how this leads to confusion?

However, i am all for a complete new test suite design. I would suggest to implement all module and Python library tests in Python, so that these tests can be executed independently from the OS or the used command line interpreter, with or without the make system.

For me it seems that it is not worth to build our own tests system and bash-like language interpreter (as proposed on the wiki page). I vote for writing module tests directly in Python. From my experience I was writing module test in bash and it is easier at the beginning but if you want to do something more special than calling a module and check the return code bash-like language is not sufficient. Moreover, we want cross platform tests and I think we don't want to implement bash-like syntax parser and interpreter for variables, if-statements, for loops in Python, although it sounds interesting. When writing directly in Python, the tests dependents on GRASS or Python process calling capabilities but they would depend anyway if the interpreter would be build. The only advantage in writing in bash-like syntax is that it can run as bash/sh... script, but Python script is great too, isn't it?

For the test of the library code it is clear that it must be done in Python as you say. I suppose that it would be tested using ctypes, am I right? The problem I can see is that I would like to test also static functions and this is not possible I guess.

The next big issue is how to automatically test the GUI?

There are currently several ways we are using. You should split the GUI to some logic/representation (MVC etc.), so for some functions you can just use standard tests, we are using doctest.

Sometimes, logic requires you to write some pseudo-GUI (e.g. text outputting classes, or generally test code) to be able to test it. This is usually combined with the approach of tests which run as standard script, produce some output and this is compared manually or by some diff. So far, no testing framework such as unitest was used for this.

Than there are completely non-automatic tests which finally contain the GUI components/classes. You just run a script/module with some parameters and you try how the window behaves without the burden of running the whole g.gui and setting things manually. The tandalone GUI modules (g.gui.*) are in this category and then there are some Python files which enables you to run them and they show some GUI which is not useful for users but just for testing. The problem of the last ones is that they are not well maintained because they are not listed or documented anywhere and are harder to use comparing to doctest.

For what I heard the automated GUI tests are not worth to do. But I'm open to any ideas about is as well as about improvements of currently used techniques.

I would like to mentor this as a GSoC project in 2014. I have some experience with test suites ... And i have the experience that nobody cares to implement module or library tests ... .

GSoC project for tests of any kind would be great.

It seems to me that the server-side automatic test are important. There are even some services for this. But this goes really far from the original ticket problem.

And by the way, Trac wiki synatax does not support our tradition of references done through [1]. The trac syntax is becoming an issue for us, I would say.

[huhabla]

Replying to wenzeslaus:

Replying to huhabla:

Replying to wenzeslaus:

There is a page about the test framework we wish to have:

• ​http://grasswiki.osgeo.org/wiki/Test_Suite

But since it does not exists the page creates just confusion.

This page describes in very detail how the testsuite can be implemented for modules and libraries. Python doctests and how to implement automated testing in the make system are not covered. It includes a guideline howto implement tests for modules based on shell scripts. Hence it is unclear to me how this leads to confusion?

However, i am all for a complete new test suite design. I would suggest to implement all module and Python library tests in Python, so that these tests can be executed independently from the OS or the used command line interpreter, with or without the make system.

For me it seems that it is not worth to build our own tests system and bash-like language interpreter (as proposed on the wiki page). I vote for writing module tests directly in Python. From my experience I was writing module test in bash and it is easier at the beginning but if you want to do something more special than calling a module and check the return code bash-like language is not sufficient. Moreover, we want cross platform tests and I think we don't want to implement bash-like syntax parser and interpreter for variables, if-statements, for loops in Python, although it sounds interesting. When writing directly in Python, the tests dependents on GRASS or Python process calling capabilities but they would depend anyway if the interpreter would be build. The only advantage in writing in bash-like syntax is that it can run as bash/sh... script, but Python script is great too, isn't it?

It is. Writing module tests directly in Python will give us much more flexibility. But we have to implement a GRASS specific Python test framework to test and validate GRASS modules. This framework should provide classes and functions to test different module configurations and check their output with reference data automatically. The suggested framework in [1] is still valid in several aspects (location creation, cleaning, data check). I would suggest to use the PyUnit framework for module testing. The PyGRASS Module interface should be used to configure module runs. Configured modules should be run by specific test functions that check the module output and stdout/stderr with reference data that is located in the modules test directory. It should be possible to configure the test suite so that all modules are executed in a virtual environment like valgrind for memory leak checks.

Here an PyUnit example of a r.info test:

#!/usr/bin/python

import unittest
import grass.pygrass.modules as pymod
import grass.test_suite as test_suite

class r_info_test(unittest.TestCase):
    
    @classmethod
    def setUpClass(cls):   
        """! Set up the test mapset and create test data for all tests
        """             
        # Create the temporary test mapset and set up the test environment
        test_suite.init()
        
        # Create test data for each test
        m = pymod.Module("r.mapcalc", expr="test = 1", run_=False)
        # We simply run this module, if the module run fails, 
        # the whole test will be aborted
        test_suite.run_module(module=m)

    def test_flag_g(self):
        """! Test to validate the output of r.info using flag "g"
        """
        # Configure a r.info test 
        m = pymod.Module("r.info", map="test", flags="g", run_=False)
        
        # Run the test using the test suite functionality and check
        # the output on stdout with reference data that is located in the
        # r.info test directory
        # This function will call an exception if the module run fails
        # or if the output in comparison to the reference data is incorrect
        test_suite.test_module(module=m, check="stdout", reference="r_info_g.ref")
        
    def test_flag_e(self):
        """! Test to validate the output of r.info using flag "e"
        """
        # Configure a r.info test 
        m = pymod.Module("r.info", map="test", flags="e", run_=False)
        
        # Run the test using the test suite functionality and check
        # the output on stdout with reference data that is located in the
        # r.info test directory.
        # This function will call an exception if the module run fails
        # or if the output in comparison to the reference data is incorrect
        test_suite.test_module(module=m, check="stdout", reference="r_info_e.ref")
        
    @classmethod
    def tearDownClass(cls):
        """! Remove the temporary mapset """
        test_suite.clean_up()
        
        
if __name__ == '__main__':
    unittest.main()

[1] ​http://grasswiki.osgeo.org/wiki/Test_Suite

For the test of the library code it is clear that it must be done in Python as you say. I suppose that it would be tested using ctypes, am I right? The problem I can see is that I would like to test also static functions and this is not possible I guess.

I see no problem in using doctests [1] and PyUnit tests [2] for Python library and C-library testing. There are C-library tests for the gmath, gpde and raster3d libraries [3,4,5]. These library tests will compile/create dedicated GRASS modules that can be executed in a grass environment ... using a PyUnit test for example. All C-library functions can be tested in this way.

PyUnit tests for GRASS library functions can be implemented if the C-libraries are accessible using the ctype bindings. Here a PyUnit test example that calls C++ library functions from the vtk-grass-bridge project [6]. Doctests are already widely used in the temporal framework, the temporal algebra and in PyGRASS, example [7].

[1] ​http://docs.python.org/2/library/doctest.html

[2] ​http://docs.python.org/2/library/unittest.html

[3] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/gmath/test

[4] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/gpde/test

[5] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/raster3d/test

[6] ​https://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Raster/Testing/Python/GRASSRasterMapReaderWriterTest.py

[7] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/temporal/space_time_datasets.py

The make system must be extended to allow automated test runs -> ​http://grasswiki.osgeo.org/wiki/Test_Suite#Make_system

This example should run the library tests.

cd grass_trunk/lib
make test

Library tests are PyUnit tests in case of C-libraries that call ctypes functions or library module like "test.raster3d.lib". Doctests should be implemented in case of Python libraries like the temporal framework or PyGRASS.

This example should only run raster module tests

cd grass_trunk/raster
make test

The make system should use a dedicated test location in which all needed global test-specific data is located in the PERMANENT mapset (minimized NC location?). It should run all tests within a grass session using this location. Each test should create its dedicated temporary mapset in the test location by calling the "test_suite.init()" function in the class setup. This hopefully allows the parallel execution of tests. Hence, "make -j4 test" should test 4 modules in parallel. Each test will create its own HTML output.

The result of the tests should be a compiled HTML document that shows every test run in detail. In addition, the output of valgrind about memory leaks should be available for each module and library test. Such HTML documents can be hosted on the test server that performs automatic test runs.

The make system will remove all remaining temporary mapsets from the test location after all tests are finished.

What a nice GSoC 2014 project. :D

[huhabla]

Any feedback regarding the proposed test approach and framework?

Replying to huhabla:

Replying to wenzeslaus:

Replying to huhabla:

Replying to wenzeslaus:

There is a page about the test framework we wish to have:

• ​http://grasswiki.osgeo.org/wiki/Test_Suite

But since it does not exists the page creates just confusion.

This page describes in very detail how the testsuite can be implemented for modules and libraries. Python doctests and how to implement automated testing in the make system are not covered. It includes a guideline howto implement tests for modules based on shell scripts. Hence it is unclear to me how this leads to confusion?

However, i am all for a complete new test suite design. I would suggest to implement all module and Python library tests in Python, so that these tests can be executed independently from the OS or the used command line interpreter, with or without the make system.

For me it seems that it is not worth to build our own tests system and bash-like language interpreter (as proposed on the wiki page). I vote for writing module tests directly in Python. From my experience I was writing module test in bash and it is easier at the beginning but if you want to do something more special than calling a module and check the return code bash-like language is not sufficient. Moreover, we want cross platform tests and I think we don't want to implement bash-like syntax parser and interpreter for variables, if-statements, for loops in Python, although it sounds interesting. When writing directly in Python, the tests dependents on GRASS or Python process calling capabilities but they would depend anyway if the interpreter would be build. The only advantage in writing in bash-like syntax is that it can run as bash/sh... script, but Python script is great too, isn't it?

It is. Writing module tests directly in Python will give us much more flexibility. But we have to implement a GRASS specific Python test framework to test and validate GRASS modules. This framework should provide classes and functions to test different module configurations and check their output with reference data automatically. The suggested framework in [1] is still valid in several aspects (location creation, cleaning, data check). I would suggest to use the PyUnit framework for module testing. The PyGRASS Module interface should be used to configure module runs. Configured modules should be run by specific test functions that check the module output and stdout/stderr with reference data that is located in the modules test directory. It should be possible to configure the test suite so that all modules are executed in a virtual environment like valgrind for memory leak checks.

Here an PyUnit example of a r.info test:

#!/usr/bin/python

import unittest
import grass.pygrass.modules as pymod
import grass.test_suite as test_suite

class r_info_test(unittest.TestCase):
    
    @classmethod
    def setUpClass(cls):   
        """! Set up the test mapset and create test data for all tests
        """             
        # Create the temporary test mapset and set up the test environment
        test_suite.init()
        
        # Create test data for each test
        m = pymod.Module("r.mapcalc", expr="test = 1", run_=False)
        # We simply run this module, if the module run fails, 
        # the whole test will be aborted
        test_suite.run_module(module=m)

    def test_flag_g(self):
        """! Test to validate the output of r.info using flag "g"
        """
        # Configure a r.info test 
        m = pymod.Module("r.info", map="test", flags="g", run_=False)
        
        # Run the test using the test suite functionality and check
        # the output on stdout with reference data that is located in the
        # r.info test directory
        # This function will call an exception if the module run fails
        # or if the output in comparison to the reference data is incorrect
        test_suite.test_module(module=m, check="stdout", reference="r_info_g.ref")
        
    def test_flag_e(self):
        """! Test to validate the output of r.info using flag "e"
        """
        # Configure a r.info test 
        m = pymod.Module("r.info", map="test", flags="e", run_=False)
        
        # Run the test using the test suite functionality and check
        # the output on stdout with reference data that is located in the
        # r.info test directory.
        # This function will call an exception if the module run fails
        # or if the output in comparison to the reference data is incorrect
        test_suite.test_module(module=m, check="stdout", reference="r_info_e.ref")
        
    @classmethod
    def tearDownClass(cls):
        """! Remove the temporary mapset """
        test_suite.clean_up()
        
        
if __name__ == '__main__':
    unittest.main()

[1] ​http://grasswiki.osgeo.org/wiki/Test_Suite

For the test of the library code it is clear that it must be done in Python as you say. I suppose that it would be tested using ctypes, am I right? The problem I can see is that I would like to test also static functions and this is not possible I guess.

I see no problem in using doctests [1] and PyUnit tests [2] for Python library and C-library testing. There are C-library tests for the gmath, gpde and raster3d libraries [3,4,5]. These library tests will compile/create dedicated GRASS modules that can be executed in a grass environment ... using a PyUnit test for example. All C-library functions can be tested in this way.

PyUnit tests for GRASS library functions can be implemented if the C-libraries are accessible using the ctype bindings. Here a PyUnit test example that calls C++ library functions from the vtk-grass-bridge project [6]. Doctests are already widely used in the temporal framework, the temporal algebra and in PyGRASS, example [7].

[1] ​http://docs.python.org/2/library/doctest.html

[2] ​http://docs.python.org/2/library/unittest.html

[3] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/gmath/test

[4] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/gpde/test

[5] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/raster3d/test

[6] ​https://code.google.com/p/vtk-grass-bridge/source/browse/trunk/Raster/Testing/Python/GRASSRasterMapReaderWriterTest.py

[7] ​http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/temporal/space_time_datasets.py

The make system must be extended to allow automated test runs -> ​http://grasswiki.osgeo.org/wiki/Test_Suite#Make_system

This example should run the library tests.

cd grass_trunk/lib
make test

Library tests are PyUnit tests in case of C-libraries that call ctypes functions or library module like "test.raster3d.lib". Doctests should be implemented in case of Python libraries like the temporal framework or PyGRASS.

This example should only run raster module tests

cd grass_trunk/raster
make test

The make system should use a dedicated test location in which all needed global test-specific data is located in the PERMANENT mapset (minimized NC location?). It should run all tests within a grass session using this location. Each test should create its dedicated temporary mapset in the test location by calling the "test_suite.init()" function in the class setup. This hopefully allows the parallel execution of tests. Hence, "make -j4 test" should test 4 modules in parallel. Each test will create its own HTML output.

The result of the tests should be a compiled HTML document that shows every test run in detail. In addition, the output of valgrind about memory leaks should be available for each module and library test. Such HTML documents can be hosted on the test server that performs automatic test runs.

The make system will remove all remaining temporary mapsets from the test location after all tests are finished.

What a nice GSoC 2014 project. :D

[hamish]

Running a test of a C module is very easy and clear in shell script, any added complication which Bourne scripting may have trouble with quickly becomes a test of that complication and not the module you are trying to test, regardless of the language, so any non-module complications should be avoided, and thus shouldn't be a factor in the test_suite design.

Follow the K.I.S.S. principal, and for the C module tests ~ three lines of shell script + comments are much preferable and readable than 10-30 lines of python boilerplate, IMHO.

The Makefiles and build system require the UNIX command line tools already so the cross-platform argument is moot, especially if you want to run it from a 'make test'.

That's not to say that results couldn't be written to a file like the build failures are, and then some python script or otherwise collect them into a html page. (I'd just focus on the failures, scrolling past 350 passing "green" modules to find the one "red" one serves little purpose IMO) or just print failures to the end of the build log like missing man pages and failed module builds.. Simple simple simple, visible, and to the point.

For valgrind results, sure, a webpage table on adhoc or a new osgeo buildbot VM would be great. see e.g. ​http://www.gpsdrive.de/build_cluster/results.shtml

for inspiration you will find various test_suite/ directories in raster/, with example runs based on either standard datasets or synthetic maps. e.g. see r.cost, r.stats, and r.mapcalc. An exhaustive set of tests of all flag and option combinations is probably not possible, we just have to select something representative for each module. / the idea so far with those was to just collect some working example for each of the C modules, & decide on how to automate it all later.

the grass-addons/grass6/general/g.md5sum module was written with test suites in mind, see the help page. There is also grass-addons/grass7/general/g.compare.md5 too look at, but beware there is the problem that different architectures, CPU models, compilers, OSs, etc. may handle float point precision in slightly different ways, so instead of a results-hash like g.md5sum could store you'd want to store a copy of a full results map for each module, then compare with the new output and check that it doesn't result in some epsilon greater than <something>. The disk use could be big.. And what should the threshold be? Should the double precision be rounded from %.17g to %.15g to mask any local FP implementation differences? It's all a bit sloppy.. For G6's g.md5sum I took the slightly reduced precision approach, but since r.out.ascii is using %.*f not %.*g for dp=, that's not right yet either.

Trac wiki synatax does not support our tradition of references done through [1]. The trac syntax is becoming an issue for us,

not sure what to say except try to use something else like "1." or (1) when in the trac.. easy to do and helps to keep the mind flexible. :-) Another thing to remember to backtick any mention of r3.* modules, since rev 3 of the svn (CVS) was a huge commit.

2c & regards, Hamish

[wenzeslaus]

I agree with huhabla (comment 4) that the tests should be primarily in Python. We actually already did the decision: GRASS modules which are scripts are in Python. The reasons for this were probably stated at that point but let's recall them:

• it's easier than C/C++ to write and run and more flexible if necessary when doing special things (might apply to tests)
• once you need to write an if statement, Python is simpler to write and read than Bash (and same applies to strings and sed, awk, ...)
• although most of the GRASS developers and power users writes in Bash a lot, the common language for most of the developers is Python (today and probably in the future too)
• Python is a general purpose language (both scripts and complicated applications are possible)

And as for the additional complexity of Python comparing to Bash (comment:5), the module test is not only a call of the module which would be easier to write in Bash. If the module returned 0 or not, is not enough. It is usually needed to set region, prepare data, call the module and test the output. Which is actually the same as it is said about writing unit tests in general: The test for the function can be longer than the function itself. So, I don't expect tests to be simple, although I wish them to be.

However, I would like to point out that we have quite different units to test, basically modules, C function and Python functions. For some of them the choices are unclear but for Python libraries, Python testing is just the best choice. There are even unittest and doctest in the standard distribution (doctest is compatible with unittest), so we don't need to use something external. No need to write some new things.

C/C++, on the other hand, requires some more thinking and unfortunately, it is hard to follow KISS principle, because it is not simple with them. There is no standard or build-in way of testing, so writing some special code is inevitable. You can write some testing programs but sooner or later you need some special code to compare results, manage tests, etc.

Besides some classic unit test frameworks for C/C++ there could be something which uses Doxygen to extract tests from doc (similarly to Python doctest) and also something which uses abilities of LLVM/clang (JIT compiler, C++ interpreter) but I have never searched for it. However, we would have probably a lot of problems with using some 3rd party solution or, so all these options are not usable. See also ​QGIS Unit Testing which uses Qt and CTest testing frameworks.

Especially for the C/C++ modules, instead of writing testing programs to test their internal functions, we can also implement something like a --test option which would be available only when certain #define would be active (using make/configure) and testing functions would be directly in the module and dependent on the #define value. Common testing functions can be in some library.

Here is the summary of options I see as doable, probable and acceptable.

Python libraries Python:

• unittest and doctest

Functions in Python modules:

• unittest and doctest

Interface of C/C++ libraries:

• interface by unittest and doctest through ctypes

Internal (static, private, ...) functions in C/C++ libraries:

• testing programs
• generate ctypes interface and then unittest and doctest

Functions in C/C++ modules:

• testing programs
• generate ctypes interface and then unittest and doctest
• --test parameter for module

C/C++ modules and Python modules:

• unittest and doctest
• Bash/shell scripts

I don't know if the ctypes generation for testing purposes would be possible but if yes, it seems to be a good option. However, you still cannot test static functions in that way.

There are five other main areas of the testing topic:

• recursive running of tests (make test)
• reports/outputs
  • output after make test is necessary
  • HTML or parsable text or XML would be really useful for online presentation
• automatic/online running
  • automatic running is not necessary but important (run of the test after commit would be even better)
  • there are some online testing services which we might use or be compatible with
• supporting functionality especially that for comparing (maps, numbers)
• data for testing
  • from my experience random are not enough
  • minimal NC sounds good
  • some artificial data would be good
  • no idea about LL locations, sometimes important but this might apply to other projections, too

I cannot tell about any of these topics anything more now but they should be discussed as well as the way of testing itself.

But back to the ticket original question. What is the way to write tests now?

• For Python, it seems that you should use doctest or unittest and invoke it in whatever way you want.
• For modules, you can write test Python or Bash scripts both are currently legal to use but Python might be the only option in the future (have a look to the comment:5 for some possible ways of testing computational results).
• For C/C++ library functions and module functions, the current way is to write special program which will test the functions, set up a build by copy and paste from existing tests (good luck).

However, question actually was Where do I find the information about tests? We must say that the best (and only) place is now this ticket (#2105).

[wenzeslaus]

Some software projects require to have a test for each closed ticked, fixed bug or commit. I'm wondering how to do it for platform dependent or system configuration specific issues such as r58200 discussed at ​GRASS-dev. Maybe in the same way as GUI part of the GUI can be tested, i.e. document with steps what to do (how to reproduce).

[wenzeslaus]

There was a GSoC:

• wiki:GSoC/2014/TestingFrameworkForGRASS

And documentation is now available:

• ​http://grass.osgeo.org/grass71/manuals/libpython/gunittest_testing.html
• source:grass/trunk/lib/python/docs/src/gunittest_testing.rst?rev=61572​

This ticket was asking for documentation but new possible tool for testing was discussed too, so here are the points from comment:7 applied to current state of testing framework.

Python libraries Python:

• gunittest
• unittest use its assert methods through gunittest
• doctest use only for documentation purposes and then test this documentation (i.e. do complex tests using gunittest), there is a special way how to import tests

Functions in Python modules:

• doctest should at least partially work, currently best to invoke directly (without testing framework)
• to standardize/promote --test parameter for GRASS modules might be useful here
• others under investigation

Interface of C/C++ libraries:

• test using special (test) GRASS modules, invoke these separately for benchmark and as standard GRASS modules through gunittest
• test as Python libraries through ctypes

Internal (static, private, ...) functions in C/C++ libraries:

• N/A

Functions in C/C++ modules:

• use special (test) GRASS modules as for C/C++ libraries
• to standardize/promote --test parameter for GRASS modules might be useful here
• static and private cannot be tested

C/C++ modules and Python modules:

• gunittest

There are five other main areas of the testing topic:

• recursive running of tests
  • e.g., python -m grass.gunittest.main --location nc_spm_grass7 --location-type nc
  • make test not implemented
• reports/outputs
  • textual output on stdout
  • HTML
  • parsable key-value files with summaries
  • XML or database not implemented
  • automatic and online running supported using custom scripts and documented for custom server
    • tests currently runs every day
    • no online testing services explicitly supported or tested
• supporting functionality especially that for comparing (maps, numbers) implemented or using unittest
• data for testing is a complex issue but most important are random data, data included in tests and NC SPM sample location, other can be used too but this was not much tested yet

Closing this ticket as more then fixed, create new tickets for further issues with testing (use component Tests).

[neteler]

Milestone renamed