• 2.0-specific Test Suite
  • Component Tests (Candidates for Automation if Not Already Covered)
    • Module Tests
    • Common UI Components
    • nomos license and copyright verification tests
    • CLI Tests
  • Basic Funtionality via the UI
    • Basic functionality
    • Web Site sanity checks
    • Basic FOSSology functionality based on the top menu items
  • RESTful UI test cases (in support of PTS requirement)
  • Install Tests
    • Source Installs
    • Package Installs
  • Cluster Testing
    • Install and Config Tests
    • Functionality
    • Cluster Ops
    • Stress Tests
    • Other items
  • Performance Testing
  • Concurrent User Testing
  • Migration Testing
• Post-Beta Testing to get to RC1
• Testing for RC2
• Testing for RC3

2.0-specific Test Suite¶

FOSSology test suite tracking for 2.0

Note for details on specific test cases, see Manual_Test_Cases

Component Tests (Candidates for Automation if Not Already Covered)¶

Module Tests¶

Module	Owner	Status/Notes
pkgagent	AUTO	unit tests: package agent unit tests
Scheduler	bob	#849 #873 #789 #561 #861
fossconfig	AUTO	unit tests: fossconfig unit tests
write 3 automated unit tests to test functions in libfossagent (fo_GetAgentKey, fo_WriteARS, fo_CreateARSTable)	bob
libfossdb	AUTO	unit tests: libfossdb unit tests
write automated unit tests to test functions in libfossrepo	bob
libfossscheduler	AUTO	unit tests: libfossscheduler unit tests
sqlCopy	AUTO	unit tests: slqCopy unit tests
adj2nest	bob
buckets	larry	done
nomosdiff	larry	done
bucketsdiff	larry	done #1581
tagging	larry	done #1389
copyright	mark	All cli parameters tested, waiting for Alex to verify latest accuracy numbers. phpunit tests written to capture cli test cases. copyright Functional Tests
delagent	AUTO	unit tests: delagent unit tests
mimetype	larry	done
nomos	mark	All cli parameters tested, phpunit tests written to capture test cases Nomos Functional Tests
ununpack	mark	#915 Issue is not critical, and can be solved by documentation. Larry helped port the 1.4.1 functional tests over to the 2.0 branch and Mark incorporated them into the nightly functional tests. All Tests pass. Unpack Functional Tests
wget_agent	paul	#879 #867

Common UI Components¶

Testing for..	Owner	Status/Notes
common-active.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/d7/dc5/test__common__active_8php.html
common-agents.php	DUP	Already tested elsewhere (upload tests)
common-auth.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/d8/d1d/test__common__auth_8php.html
common-cache.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/d4/dee/test__common__cache_8php.html
common-cli.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/dd/d9d/test__common__cli_8php.html
common-buckets.php	DUP	Already tested elsewhere (nomos diff, buckets)
common-compare.php	DUP	Already tested elsewhere (nomos diff)
common-db.php	DUP	Already tested elsewhere (various)
common-dir.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/d1/d53/test__common__dir_8php.html
common-folders.php	DUP	Already tested elsewhere (various)
common-job.php	DUP	Already tested elsewhere (upload)
common-license-file.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/d8/d06/test__common__license__file_8php.html
common-menu.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/da/d65/test__common__menu_8php.html
common-parm.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/df/d6e/test__common__parm_8php.html
commonphp	DUP	Already tested elsewhere (various)
common-pkg.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/d5/d50/test__common__pkg_8php.html
common-plugin.php	DUP	Already tested elsewhere (various)
common-repo.php	DUP	Already tested elsewhere (various)
common-scheduler.php	DUP	Already tested elsewhere (various)
common-sysconfig.php	AUTO	http://fonightly.usa.hp.com/fossologyCodeDocs/d7/df3/test__common__sysconfig_8php.html
common-tags.php	DUP	Already tested elsewhere (tagging)
common-ui.php	DUP	Already tested elsewhere (various)
bootstrap.php	DUP	Already tested elsewhere (various)
libschema.php	DUP	Already tested elsewhere (various)

nomos license and copyright verification tests¶

Functionality to test	Owner	Status/Notes
nomos cli oneshots	mark	5 new functional tests for nomos: Nomos Functional Tests
verifyFoss23D1F1L.php	mark	This test and the tests below need to have an upload loaded first. There is no way to do that today without driving the ui. Or by rewriting the tests.
verifyFossDirsOnly.php	mark
verifyFossI16L335U29.php	mark

CLI Tests¶

Command Line utility to test	Owner	Status/Notes
cp2foss	larry
fossjobs	larry
fo_cli	paul
delagent	vincent
pkgagent	vincent
schema-export	vincent	#1155

Basic Funtionality via the UI¶

Basic functionality¶

Functionality to test	Owner	Status/Notes	Post-Beta test status (Larry)
Homepage	Larry	done	done cluster
Upload	Larry	done #861 #1365	done #1191 (not 2.0)
Jobs	Larry	done	done
Search	Larry	done	done
Browse	Larry	done	done
Organize	Larry	Done	done #1185 (not 2.0)
User Permissions	Larry	done User Permission function is not finished in my opinion, including ui and cli, firstly I need one user doc about this part, sencondly I do not think it is necessary to file all bugs about this part, I will be too much, execpt Admin almost each role has defects for now	done
Admin	Larry	done	done #1833

Please add tests as needed.

Web Site sanity checks¶

Functionality to test	Owner	Status/Notes
AboutMenuTest.php	DUP	Already tested in UI test cases
login.php	DUP	Already tested in UI test cases
OrgFoldersMenuTest-Create.php	DUP	Already tested in UI test cases
OrgFoldersMenuTest-Delete.php	DUP	Already tested in UI test cases
OrgFoldersMenuTest-Edit.php	DUP	Already tested in UI test cases
OrgFoldersMenuTest-Move.php	DUP	Already tested in UI test cases
OrgUploadsMenuTest-Delete.php	DUP	Already tested in UI test cases
OrgUploadsMenuTest-Move.php	DUP	Already tested in UI test cases
SearchMenuTest.php	DUP	Already tested in UI test cases
UploadFileMenuTest.php	DUP	Already tested in UI test cases
UploadInstructMenuTest.php	DUP	Already tested in UI test cases
UploadOne-ShotMenuTest.php	DUP	Already tested in UI test cases
UploadServerMenuTest.php	DUP	Already tested in UI test cases
UploadUrlMenuTest.php	DUP	Already tested in UI test cases

Basic FOSSology functionality based on the top menu items¶

Functionality to test	Owner	Status/Notes
CreateFolderTest.php	DUP	Already tested in UI test cases
DeleteFolderTest.php	DUP	Already tested in UI test cases
editFolderDescriptionOnlyTest.php	DUP	Already tested in UI test cases
editFolderNameOnlyTest.php	DUP	Already tested in UI test cases
editFolderTest.php	DUP	Already tested in UI test cases
moveFolderTest.php	DUP	Already tested in UI test cases
DupFolderTest.php	DUP	Already tested in UI test cases
moveUploadTest.php	DUP	Already tested in UI test cases
can create duplicate upload	DUP	Already tested in UI test cases
upload properties	DUP	Already tested in UI test cases
upload removal	DUP	Already tested in UI test cases
UploadFileTest.php	DUP	Already tested in UI test cases
UploadUrlTest.php	DUP	Already tested in UI test cases
uploadSrvFileTest.php	DUP	Already tested in UI test cases
UploadSrvArchiveTest.php	DUP	Already tested in UI test cases
uploadSrvDirTest.php	DUP	Already tested in UI test cases
addUserTest.php	DUP	Already tested in UI test cases
dupUserTest.php	DUP	Already tested in UI test cases
emailUserTest.php	larry	done #849 **
noEmailUserTest.php	larry	done
uploadWEmail.php	larry	done
uploadWoutEmail.php	larry	done
Browse tests	DUP	Already tested in UI test cases
Schedule Agents tests	DUP	Already tested in UI test cases
Admin tests	DUP	Already tested in UI test cases

RESTful UI test cases (in support of PTS requirement)¶

Test	Owner	Status/Notes
One-Shot nomos license test for PTS support	bob

Install Tests¶

Install tests take two forms, source installs and package installs. Source installs are just that, the sources are checked out, utils/fo-installdeps is run, fossology is made, installed and fo-postinstall run. Tests are then run on the installed system.

Source Installs¶

Functionality to test	Owner	Status/Notes
Source Install	vincent	DONE
fo-installdeps	vincent	DONE #915

For details on testing a source install in a cluster, see Cluster Testing below.

Package Installs¶

The package installs are automated by a php script. The script expects to be run on a Virtual Machine that was set up for package install testing. See Package Install Testing for details.

Functionality to test	Owner	Status/Notes
test package installs	mark	Jenkins job is set up, will turn it on when packages are made/ready
Install Packages on Test Systems	mark	will install packages on 64 and 32 bit machines and stress test
Upgrade testing 1.4.1 -> 2.0	mark

Functionality to test	Owner	Status/Notes
verify agent.agent_rev gets bumped when we install a new agent	Larry	need an agent with a new version

Cluster Testing¶

Install and Config Tests¶

Test	Owner	Status/Notes
can a cluster be installed from source using existing documentation?	vincent	DONE
can a cluster be installed from a package using existing documentation?	vincent	DONE
Configuration: Scheduler/Web/Db Server, agent machines	vincent	DONE
Configuration: Scheduler/Web/Db/agents on server, agents on agent machine	vincent	DONE
Configuration: web server, db server, agent machines	---	unsupported configuration, will not test
Configuration: Others?	??
Can a cluster be installed without manually editing fossology *.conf files?	none	cannot be done in 2.0; enhancement filed for 2.1

Functionality¶

Test	Owner	Status/Notes	Post-Beta test status (Vincent)
Verify Upload from file works	vincent	DONE	Done
Verify a file can be uploaded using Upload from url	vincent	DONE	Done
Verify a directory & it's contents can be uploaded using Upload from url	vincent	DONE	Done
Verify wildcards work with Upload from url	vincent	DONE	Done
Verify Upload from server works	mark	#1047 #1413	Done
Verify one-shot nomos works	mark	DONE	Done
Verify Upload one shot copyright works	mark	DONE	Done
upload a tar with a known number of files in it (contents are known as well),verify files are in the gold and files repo's.	vincent	DONE	Done

Cluster Ops¶

Test	Owner	Status/Notes	Post-Beta test status (Bob)
cluster where jobs only run on agent machines, verify true.	vincent	DONE	DONE per Bob's verbal update
cluster where jobs are run on both web/db server and agents, verify agents run on both machines.	vincent	DONE	DONE per Bob's verbal update
cluster with multiple agent machines, are jobs run across the cluster using round robin load balancing?	vincent	DONE	DONE per Bob's verbal update
verify unpack only runs once in a cluster	mary		DONE per Bob's verbal update
cluster with multiple agent machines, do agents for multiple jobs run on only one machine or multiple?	vincent	DONE (redundant test, same as above)	DONE per Bob's verbal update

Stress Tests¶

Test	Owner	Status/Notes	Post-Beta test status (Mary)
upload a distro iso, e.g. debian, fedora, RH.	vincent	DONE	DONE
Busy the system up with multiple large uploads. All should be processed with no failures in any agents.	vincent	DONE #1041	DONE
While large uploads are running and being processed, upload a smaller job and make sure it gets run in a resonable time	larry(done)	works great, small upload was process to completion and large upload continued to be processed.	DONE
While jobs are running, delete other uploads and ensure they are processed when other jobs end.	vincent	DONE	DONE
Run a number of uploads at once... pause some, change priorities, resume, cancel some	vincent	DONE #1041
Delete a distro iso, testing delagent for big iso	mark	#1053	DONE

Other items¶

Since the way clusters work in 2.0 has changed, here are some questions that can turn into test cases.

Test	Owner	Status/Notes
Where are the agent logs in the repo? Db server? Web Server? or Agent machines?	vincent
What happens to the agents on the agent machine when the scheduler is shut down?	vincent
In a cluster with a separate web server (i.e. separate db server), stop the scheduler, does the ui report it stopped?	vincent

Performance Testing¶

A place holder

Concurrent User Testing¶

A place holder

Migration Testing¶

1. install a cluster with FO 1.4.1 (using the 1.4.1 packages)
2. upload and analyze some packages/distros to populate the db
   NOTE: be careful in your selection of packages and the analyses run. you need enough to populate all the tables, but not so much that it takes too long to genarate results quickly; no more than a couple hours. I DO NOT recommend using a large iso.
3. generate reports (nomos, buckets, copyright) and save as pdf or html
   for 1.4.1 -> 2.0 migration test, I saved
   Browse.htm (top level Browse menu)
   Bucket Browser.htm (for debian-6.0.4-source-DVD-1.iso)
   License Browser.htm (for debian-6.0.4-source-DVD-1.iso)
   Copyright.htm (for debian-6.0.4-source-DVD-1.iso)
   Created 3 Tag Namespaces; Manage Tag Namespace.htm(TagNS1, TagNS2, TagNS3)
   Tagged a single file (Tag.htm) RedHat.tar/RedHat/amtu-1.0.2-2.EL4.ia64.rpm
   Is this sufficient?
4. upgrade cluster with 2.0 packages
   edit /etc/apt/sources.list to comment out source for previous version and add an entry for the new version you wish to upgrade to.
   On the server: as root, run the apt commands; sudo apt-get clean; sudo apt-get update; sudo apt-get purge fossology fossology-agents-single fossology-scheduler fossology-scheduler-single fossology-web-single
   sudo apt-get upgrade fossology-db fossology-web fossology-scheduler fossology-common
   sudo apt-get install fossology-ununpack fossology-copyright fossology-buckets fossology-mimetype fossology-delagent fossology-wgetagent fossology-nomos fossology-pkgagent
   On the clients: as root, run the apt commands; sudo apt-get clean; sudo apt-get update; sudo apt-get upgrade fossology-common; sudo apt-get purge fossology-agents
   sudo apt-get install fossology-ununpack fossology-copyright fossology-buckets fossology-mimetype fossology-delagent fossology-wgetagent fossology-nomos fossology-pkgagent
   edit cluster config files: /etc/fossology/fossology.conf & /etc/fossology/Db.conf
5. BEFORE generating 2.0 results, look at the 1.4.1 results in the newly upgraded 2.0 UI and verify against the html (or pdf) files save in step 3. The numbers should be identical.
6. rescan the uploaded packages/distros with the 2.0 version of nomos
7. verify that the 1.4.1 and 2.0 nomos results are similar. (For example, if 1.4.1 reports 1000 GPL_v2+ licenses, then the 2.0 number should be roughly the same though not necessarily identical).

Post-Beta Testing to get to RC1¶

• All packages install successfully (Vincent & Mark) http://fossbuild.usa.hp.com/fossology/2.0.0/testing/
• test & close all 2.0 bugs (issue Authors): http://www.fossology.org/projects/fossology/issues?query_id=9
  Mary: #1653 #1515 #1467 #1641 #1665 #21 #1683
  Alex:
  Bob: #1035 #1077 #561 #1053 #999 #837
  Vincent: #1047 #855
  Larry #1605 #1299 #1389 #1401 #291 #807 #1581 #1455 #921 #1389
  Mark: #1545 #1101 #1539 #1323 #915 #969 #1257 #1263 #1125 #1131 #1587 #1233 #1239 #1329 #1575 #897 #1113 #1119 #1137 #1143 #1281 #1359 #1431 #1413 #957 #1275 #1689

• perform regression tests
  http://www.fossology.org/projects/fossology/wiki/20-specific_Test_Suite#Basic-functionality (Larry done on ubuntu 1110/64 and debian 6.0/64, rhel 6.0/64, rhel 6.0/32 till April 17, one cluster(rhel 6.0/64 and rhel 6.0/32) )
  http://www.fossology.org/projects/fossology/wiki/20-specific_Test_Suite#Functionality (Vincent)
  http://www.fossology.org/projects/fossology/wiki/20-specific_Test_Suite#Cluster-Ops (Bob)
  http://www.fossology.org/projects/fossology/wiki/20-specific_Test_Suite#Stress-Tests (Mary)

Testing for RC2¶

• test & close all "High" priority 2.0 bugs: http://www.fossology.org/projects/fossology/issues?query_id=9
  Bob: please test #777 #1857, #1917, #1875, #1887, #1899, #1911, #1923, #1965
  Mary: test #1965 (on fo.usa)

• perform scheduler regression tests
  http://www.fossology.org/projects/fossology/wiki/Scheduler_Test_Cases (whoever can get to it first: Mary, Vincent or Larry vincent and larry have done it on single system, cluster test is needed)

Testing for RC3¶

• perform regression tests to exercise code changes made post-RC2
1. Test substitution of "/" with "`" in ufile_name using this file as a test case: http://newbobg.ostt/~bobg/testfiles/testfiles.tar, fixed in branches/fossology2.0/fossology/src/ununpack/agent/utils.c, Completed by Larry
   http://fossology.svn.sourceforge.net/viewvc/fossology/branches/fossology2.0/fossology/src/ununpack/agent/utils.c?view=markup&pathrev=5877 &
   http://fossology.svn.sourceforge.net/viewvc/fossology/branches/fossology2.0/fossology/src/ununpack/agent/utils.c?view=markup&pathrev=5878 &
   http://fossology.svn.sourceforge.net/viewvc/fossology/branches/fossology2.0/fossology/src/ununpack/agent/utils.c?view=markup&pathrev=5879
2. Test memory leak fix in _RepGetHost using valgrind (Eg. valgrind --tool=memcheck --leak-check=yes ./nomos myfile.c) fixed in branches/fossology2.0/fossology/src/lib/c/libfossrepo.c, fossconfig.c & libfossscheduler.c Completed by Larry
   http://fossology.svn.sourceforge.net/viewvc/fossology/branches/fossology2.0/fossology/src/lib/c/libfossrepo.c?view=markup&pathrev=5876 &
   http://fossology.svn.sourceforge.net/viewvc/fossology/branches/fossology2.0/fossology/src/lib/c/fossconfig.c?view=markup&pathrev=5875 &
   http://fossology.svn.sourceforge.net/viewvc/fossology/branches/fossology2.0/fossology/src/lib/c/libfossscheduler.c?view=markup&pathrev=5875
3. Agent regression tests
   adj2nest (Mary) DONE testfiles.tar, RedHat.tar & all.zip passed
   buckets (Mary) DONE all.zip & RedHat.tar passed
   copyright (Mary) DONE all.zip & LTS_dev_acpi.tar.gz passed
   delagent (Mary) DONE testfiles.tar passed
   mimetype (Mary) DONE testfiles.tar, RedHat.tar & all.zip passed
   nomos (Mary) DONE testfiles.tar, RedHat.tar & all.zip passed
   pkgagent (Mary) DONE python-devel-2.6.0-8.6.x86_64.rpm
   ununpack (Mary) DONE testfiles.tar, RedHat.tar passed
   wget_agent (Mary) DONE testfiles.tar via upload_url passed; RedHat.tar via upload_url passed; all.zip via upload_file passed
4. Check to see garbage in email notification has been corrected (http://www.fossology.org/issues/1875) Deferred to 2.0.1 http://www.fossology.org/issues/1989
5. Verify the scheduler will accept and use "-c sysconfdir" passed on the command line (http://www.fossology.org/issues/2007)
   larry's test notes: For now, the scheduler can use the new sysconf to schedule agents, but the agents cannot finish successfully, please refer to http://www.fossology.org/issues/2025, so for now, I cannot say if issue #2007 is fixed or not.

• Test the package upgrade path from RC2 to RC3 (for fossology.usa upgrade) (Mary) DONE