Feature #2979

Determine best functional test framework (if not PHPUnit)

Added by Paul Holland 11 months ago. Updated 9 months ago.

Status:Closed Start date:06/18/2012
Priority:High Due date:
Assignee:Paul Holland % Done:

0%
Category:Testing
Target version:-
Rank:2 Tester:

Description

PHPUnit has some limitations re: executing functional tests since it expects to set up and tear down after each test. We should determine if this is OK or select a different test framework for functional tests and provide guidance to the team.

History

#1
Updated by Mary Laser 11 months ago

  • Status changed from New to In Progress
  • Estimate set to 9

looking at options

#2
Updated by Mary Laser 10 months ago

An example of running nomos one shot tests using shunit2 has been checked in http://fossology.svn.sourceforge.net/viewvc/fossology/trunk/fossology/src/nomos/agent_tests/Functional/

Similarly, Alex has demonstrated how to use a python based test harness to execute functional tests for the scheduler (http://fossology.svn.sourceforge.net/viewvc/fossology/trunk/fossology/src/scheduler/agent_tests/Functional/).

The Functional Testing page (http://www.fossology.org/projects/fossology/wiki/Automation_Testing_Functional_Tests) will be updated to reflect these strategies for automating functional tests.

#3
Updated by Mary Laser 10 months ago

  • IterNum changed from 3 to 5

I've concluded there is no single best test framework to automate functional tests. The method used is dependent on several factors, including the language being tested, the preconditions or state needed to run a test, the results generated by a test and the method of verification. Examples of automating functional tests using PHPunit and shunit2 demonstrate how to use the builtin assert tests to check for an expected test result.

Alex's homegrown python test harness goes a step further by allowing one to define actions. He then creates 5 basic actions tailored to test the scheduler.

To close this issue, I will update the Automating functional tests page (http://www.fossology.org/projects/fossology/wiki/Automation_Testing_Functional_Tests)to show examples and give recommendations for each of the different methods.

#4
Updated by Paul Holland 10 months ago

  • Target version set to 2.0.1

#5
Updated by Mary Laser 10 months ago

  • IterNum changed from 5 to 6

#6
Updated by Mary Laser 9 months ago

  • Status changed from In Progress to Resolved

wiki page with examples completed.

#7
Updated by Mary Laser 9 months ago

  • Assignee changed from Mary Laser to Bob Gobeille

please verify for completeness and close

#8
Updated by Bob Gobeille 9 months ago

The functional test page http://www.fossology.org/projects/fossology/wiki/Automation_Testing_Functional_Tests does not give enough information on how to write tests. There is not even a link to shunit or phpunit reference docs.

Alex's test code reminds me a lot of ant. I don't see why he just didn't use ant rather than write his own.

The original premise of this issue, that phpunit has setup and teardown functions is unclear. All the methods discussed here have optional setup and teardown functions.

#9
Updated by Bob Gobeille 9 months ago

  • Assignee changed from Bob Gobeille to Mary Laser

#10
Updated by Mary Laser 9 months ago

  • Target version deleted (2.0.1)

#11
Updated by Mary Laser 9 months ago

  • Assignee changed from Mary Laser to Paul Holland

per bob's feedback: added a link to phpunit docs & added references for test writing

please verify for completeness and close

#12
Updated by Paul Holland 9 months ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF