Functional Tests¶

The next level up from unit tests are functional tests. Functional tests ensure that a set of code units work together for a desired outcome. They examine a specific feature/function of the software, typically by testing APIs and the user interface, to ensure requirements are met.

Whenever possible, functional tests should be automated and incorporated into our continuous integration process. There is no single best test framework to automate functional tests. The method used is dependent on several factors, including the language being tested, the preconditions or state needed to run a test, the results generated by a test and the method of verification.

There is no hard and fast rule as to how many functional test cases to write. Usually functional test cases are derived from user requirements, stories, or external documentation. The challenge in testing FOSSology is to understand how the system is supposed to work and then design tests to prove/disprove that it works that way.

At a minimum, functional tests should test:

• all valid inputs in the ranges supported.
• invalid inputs should be supplied, and the code should error or die gracefully.
• all APIs (if they exist)
• all cli options. For example, if a program has a cli interface like

foo -h [-d dir] [-c comment] [-x] -f <file>

Then every option should be tested for valid values, and then every option should be tested for invalid values.
For example:

• foo <no parameters> (expect help message)
• foo -h (expect help message)
• foo -d <invalid dir> (should produce an error)
• foo -f <no file> (should produce an error)
• foo -c <no comment> (should produce an error)

etc...

Keep your test cases small, and have them test only one thing (if possible). There should always be more than one test for any given module.
show examples and give recommendations for each of the different methods

Examples of automated functional testing in fossology using phpUnit , python and shell unit (shunit2) are in our subversion source tree. Tests utilizing PHPunit and shunit2 demonstrate how to use the builtin assert tests to check for an expected test result. The basic structure looks like this:

1. Setup - initialize variables, data, etc.
2. Test - execute the functional test(s)
3. Verify - verify results using the built-in asserts
4. Teardown - perform cleanup and trigger the built-in reporting mechanisms

Using shell unit, steps 1,3 & 4 are provided by shunit2. Here is a greatly simplified example:

#! /bin/sh
#
# Copyright 2008 Kate Ward. All Rights Reserved.
# Released under the LGPL (GNU Lesser General Public License)
#
# shUnit2 -- Unit testing framework for Unix shell scripts.
# http://code.google.com/p/shunit2/
#
# Author: kate.ward@forestent.com (Kate Ward)
#

shunit2 specific initialization happens first

SHUNIT_TRUE=0
SHUNIT_FALSE=1
SHUNIT_ERROR=2
...
# constants
__SHUNIT_ASSERT_MSG_PREFIX='ASSERT:'
...
# variables
__shunit_lineno=''  # line number of executed test
...
# counts of tests
__shunit_testsTotal=0
__shunit_testsPassed=0
__shunit_testsFailed=0
...
# counts of asserts
__shunit_assertsTotal=0
...

asserts and helper functions

#-----------------------------------------------------------------------------
# assert functions
#
...
assertEquals()
assertNotEquals()

#-----------------------------------------------------------------------------
# failure functions
# Records a test failure.
fail()
failNotEquals()
...

#-----------------------------------------------------------------------------
# suite functions
#
#suite() { :; }  # DO NOT UNCOMMENT THIS FUNCTION
# Adds a function name to the list of tests schedule for execution.
suite_addTest()
...

# Stub. This function will be called before each test is run.
# Common environment preparation tasks shared by all tests can be defined here.
# This function should be overridden by the user in their unit test suite.
#
#setUp() { :; }  # DO NOT UNCOMMENT THIS FUNCTION
...

# Stub. This function will be called after each test is run.
# Common environment cleanup tasks shared by all tests can be defined here.
# This function should be overridden by the user in their unit test suite.
#
#tearDown() { :; }  # DO NOT UNCOMMENT THIS FUNCTION

These next functions do all the grunt work for you.

#------------------------------------------------------------------------------
# internal shUnit2 functions
#

_shunit_mktempDir()
_shunit_mktempFunc()
_shunit_cleanup()
_shunit_execSuite()
_shunit_generateReport()
_shunit_assertPass()
_shunit_assertFail()
...

Here's where the magic happens...

#------------------------------------------------------------------------------
# main
#

# provide a public temporary directory for unit test scripts
SHUNIT_TMPDIR="${__shunit_tmpDir}/tmp" 
mkdir "${SHUNIT_TMPDIR}" 

# some other housekeeping takes place here ...

# execute the oneTimeSetUp function (if it exists)
oneTimeSetUp

# dynamically build a list of tests to execute
if [ -z "${__shunit_suite}" ]; then
  shunit_funcs_=`_shunit_extractTestFunctions "${__shunit_script}"`
  for shunit_func_ in ${shunit_funcs_}; do
    suite_addTest ${shunit_func_}
  done
fi
...
# execute the tests
_shunit_execSuite

# ask the plugin to do any final reporting
shunit_plugin_final_report

# ask the plugin to finish itself
shunit_plugin_finish

# execute the oneTimeTearDown function (if it exists)
oneTimeTearDown

# that's it folks
exit $?

Here's an example of step 2, the tests to execute:

#! /bin/sh
testOneShotaffero()
{
# test to see if the file exists
  if [ ! -f '../../../testing/dataFiles/TestData/licenses/Affero-v1.0' ]; then
    fail "ERROR: test file not found...aborting test" 
  fi

  out=`/usr/local/etc/fossology/mods-enabled/nomos/agent/nomos ../../../testing/dataFiles/TestData/licenses/Affero-v1.0`
  assertEquals "File Affero-v1.0 contains license(s) Affero_v1" "${out}" 
}

testOneShotempty() 
{
# test to see if the file exists
  if [ ! -f '../testdata/empty' ]; then
    fail "ERROR: test file not found...aborting test" 
  fi

# echo "starting testOneShotempty" 
  out=`../../agent/nomos ../testdata/empty`
  assertEquals "File empty contains license(s) No_license_found" "${out}" 
}

testOneShotgpl3() 
{
# test to see if the file exists
  if [ ! -f '../../../testing/dataFiles/TestData/licenses/gpl-3.0.txt' ]; then
    fail "ERROR: test file not found...aborting test" 
  fi

# echo "starting testOneShotgpl3" 
  out=`../../agent/nomos ../../../testing/dataFiles/TestData/licenses/gpl-3.0.txt`
  assertEquals "File gpl-3.0.txt contains license(s) FSF,GPL_v3,Public-domain" "${out}" 
}

Alex's homegrown python test harness goes a step further by allowing one to define actions. He then creates 5 basic actions tailored to test the scheduler.