Functional Tests¶

The next level up from unit tests are functional tests. Functional tests ensure that a set of code units work together for a desired outcome. They examine a specific feature/function of the software, typically by testing APIs and the user interface, to ensure requirements are met.

Whenever possible, functional tests should be automated and incorporated into our continuous integration process. There is no single best test framework to automate functional tests. The method used is dependent on several factors, including the language being tested, the preconditions or state needed to run a test, the results generated by a test and the method of verification.

There is no hard and fast rule as to how many functional test cases to write. Usually functional test cases are derived from user requirements, stories, or external documentation. The challenge in testing FOSSology is to understand how the system is supposed to work and then design tests to prove/disprove that it works that way.

At a minimum, functional tests should test:

• all valid inputs in the ranges supported.
• invalid inputs should be supplied, and the code should error or die gracefully.
• all APIs (if they exist)
• all cli options. For example, if a program has a cli interface like

foo -h [-d dir] [-c comment] [-x] -f <file>

Then every option should be tested for valid values, and then every option should be tested for invalid values.
For example:

• foo <no parameters> (expect help message)
• foo -h (expect help message)
• foo -d <invalid dir> (should produce an error)
• foo -f <no file> (should produce an error)
• foo -c <no comment> (should produce an error)

etc...

Keep your test cases small, and have them test only one thing (if possible). There should always be more than one test for any given module.
show examples and give recommendations for each of the different methods

Examples of automated functional testing in fossology using phpUnit , python and shell unit (shunit2) are in our subversion source tree. Tests utilizing PHPunit and shunit2 demonstrate how to use the builtin assert tests to check for an expected test result. The basic structure looks like this:

1. Setup - initialize variables, data, etc.
2. Test - execute the functional test(s)
3. Verify - verify results using the built-in asserts
4. Teardown - perform cleanup and trigger the built-in reporting mechanisms

Using shell unit, steps 1,3 & 4 are provided by shunit2:

#! /bin/sh
#
# Copyright 2008 Kate Ward. All Rights Reserved.
# Released under the LGPL (GNU Lesser General Public License)
#
# shUnit2 -- Unit testing framework for Unix shell scripts.
# http://code.google.com/p/shunit2/
#
# Author: kate.ward@forestent.com (Kate Ward)
#

< shunit2 specific initialization> 
...
SHUNIT_TRUE=0
SHUNIT_FALSE=1
SHUNIT_ERROR=2
...

# constants
__SHUNIT_ASSERT_MSG_PREFIX='ASSERT:'
...

# variables
__shunit_lineno=''  # line number of executed test
...

# counts of tests
__shunit_testSuccess=${SHUNIT_TRUE}
__shunit_testsTotal=0
__shunit_testsPassed=0
__shunit_testsFailed=0

# counts of asserts
__shunit_assertsTotal=0
...

Here's an example of step 2, the tests to execute:

#! /bin/sh
testOneShotaffero()
{
# test to see if the file exists
  if [ ! -f '../../../testing/dataFiles/TestData/licenses/Affero-v1.0' ]; then
    fail "ERROR: test file not found...aborting test" 
  fi

  out=`/usr/local/etc/fossology/mods-enabled/nomos/agent/nomos ../../../testing/dataFiles/TestData/licenses/Affero-v1.0`
  assertEquals "File Affero-v1.0 contains license(s) Affero_v1" "${out}" 
}

testOneShotempty() 
{
# test to see if the file exists
  if [ ! -f '../testdata/empty' ]; then
    fail "ERROR: test file not found...aborting test" 
  fi

# echo "starting testOneShotempty" 
  out=`../../agent/nomos ../testdata/empty`
  assertEquals "File empty contains license(s) No_license_found" "${out}" 
}

testOneShotgpl3() 
{
# test to see if the file exists
  if [ ! -f '../../../testing/dataFiles/TestData/licenses/gpl-3.0.txt' ]; then
    fail "ERROR: test file not found...aborting test" 
  fi

# echo "starting testOneShotgpl3" 
  out=`../../agent/nomos ../../../testing/dataFiles/TestData/licenses/gpl-3.0.txt`
  assertEquals "File gpl-3.0.txt contains license(s) FSF,GPL_v3,Public-domain" "${out}" 
}

Alex's homegrown python test harness goes a step further by allowing one to define actions. He then creates 5 basic actions tailored to test the scheduler.