• 2.0 Test Plan
  • Overview
  • Existing Tests
  • Test Framework Documents
    • CUnit
    • PHPUnit
  • Manual Tests
  • Unit Tests
    • Unit Test Status
  • Functional Tests
    • Criteria for writing functional tests
    • Test Documentation
    • No UI Functional Test
    • Common UI Functional Test
    • UI Function Test (SimpleTest)
    • Alpha testing feedback

2.0 Test Plan¶

Overview¶

Fossology release 2.0 is a major re-architecture of the internals of Fossology. In particular, the scheduler is being redesigned and rewritten. Other changes like how the system and agents are configured are also completely new. The scheduler changes require agent changes. Every agent will be modified in this release.

The sources are also being reorganized to allow each agent to have all the needed components for it's operation in it's source tree. This includes the unit and functional tests for that agent.

In previous releases continuous integration was used to build, install and test fossology on a nightly basis. The continuous integration effort will be expanded for release 2.0 by using the Jenkins CI application. Jenkins will run and report test results for build, install, unit and functional tests on a nightly basis. Additionally, unofficial product packages in debian and RPM format will be built each night and installed on Virtual Machines (VMs) to enable
end to end integration and testing.

In addition to the report format, all tests should be documented using doxygen. Test documentation is needed for a number of reasons. * Tests are code, code should be documented. For example, what does the test test? * Test documentation can serve as the test plan, so either a small test plan is needed or none at all depending on how well the test documentation is done. * Tests can have an API, the API should be documented so other tests can use it.

The tests are located along with the source module they test. See Source/Disk Structure for details.

All tests must produce output in junit format for test reporting. An email was sent July 25, 2011 announcing this requirement.

  Hi All,
  FOSSology has established test frameworks and standard reports.  The reporting standard is the unit report format. This varies depending on what is being tested:
    • c code: CUnit
    • PHP code: phpunit, or simpletest, phpunit is preferred for straight unit tests. By itself, phpunit cannot test UI's.
    • ui: simepletest.  Only simpletest has a way to test a browser, but it can't deal with javascript updates to the screen.       
  If you don’t use one of the existing test frameworks, then whatever you choose must produce a xml 
  report in junit format. The definition for the junit report format can be found on the fossology wiki at: 
  http://fossology.org/test:junit_format. 
  It is strongly suggested that you use one of the existing test frameworks to create tests.  
  If you choose not to, a suggested method to produce junit report formats is to wrap all test 
  cases in PHUnit.  By running PHPUnit with --log-junit flag, the results will be in junit format.
  Thanks for your support and compliance with these test reporting standards.
  Mary

Existing Tests¶

Existing tests should be moved by the developer working on that code for 2.0 to the new source structure. Much of the physical movement of the files has already been accomplished with the source code changes on the 2.0 branch.

Each developer should run the tests to see if the movement of code caused the test itself to break. Fix any test failures and retest till the tests run without test errors in the tests.

Test Framework Documents¶

To use the different test frameworks (CUnit, PHPUnit, SimpleTest) the framework documentation should be consulted. Below are links to the various frameworks and their documents.

CUnit¶

CUnit Documentation

CUnit Doxygen Documentation

Two specific pages of the above that are useful:

CUnit Suite Info

CUnit Code Annotations

PHPUnit¶

PHPUnit Documentation

Simpletest

SimpleTest has three main documents, documentation, the tutorial and the api reference which is a phpdocumenter output (like doxygen).

Simpletest Tutorial

Simpletest Documentation

Simpletest API

Manual Tests¶

If your tests are not automation and are done by hand, they must be documented in the manual test cases page. Add a sub page and use the templates.

• Manual Test Cases

Unit Tests¶

All back end agents, scheduler and libraries should have unit tests written for them. CUnit is the framework to use for this task if the code is in C. There are many examples and resources on the web for writing CUnit tests. See the above links for the documents for a specific test framework.

For a discussion of what to unit test, see the Unit Test Guidelines.

The number of tests to write is determined by the code coverage metric. Enough unit tests should be written to achieve 60-70% code coverage for a given module. Coverage is determined using lcov. If the coverage for the module is not between 60-70%, then more tests should be written till that metric is achieved.

Unit Test Status¶

Use the table below to track progress towards the 60-70% code coverage goal.

Module	Tested(lines)	Coverage Result	Owner	Notes
pkgagent	295/451	64.7%	vincent	Add'l ad hoc testing by Team
Scheduler		10%	norton	Add'l ad hoc testing by Team
fossconfig		100%	norton	Add'l ad hoc testing by Team
libfossagent		0%	bobg	Ad hoc testing by Team
libfossdb		0%	bobg	Ad hoc testing by Team
libfossrepo		0%	bobg	Ad hoc testing by Team
libfossscheduler		100%	norton	Add'l ad hoc testing by Team
sqlCopy		0%	bobg	Ad hoc testing by Team
adj2nest		0%	bobg	Ad hoc testing by Team
buckets		0%	bobg	Ad hoc testing by Team needs more testing
copyright		100%	norton	Add'l ad hoc testing by Team
delagent	195/292	66.8%	vincent	Add'l ad hoc testing by Team
example_wc_agent		0%	larry	will not test
mimetype	146/201	72.6%	larry	case count 6, Add'l ad hoc testing by Team
nomos		0%	bobg	Ad hoc testing by Team
Shell (if needed)		0%	vincent	will not test
sql agent (if needed)		0%	vincent	will not test
ununpack	1134/1901	59.7%	bobg	case count 56, Add'l ad hoc testing by Team
wget_agent	149/219	68%	larry	Add'l ad hoc testing by Team

Functional Tests¶

Functional tests must report their results in junit format. Functional tests should use some type of test framework to help with test automation and reporting. The existing frameworks are PHPUnit for non ui functional testing and simpletest for ui testing or regular functional testing.

If some other test framework is used, an easy method of generating junit report formats is to wrap the test in PHPUnit which produces junit format using the option below.
--log-junit <file>

Functional tests should be written for every major component/capability listed on the V2.0 Features and Tasks page. Owners for each component/capability are also responsible for creating the functional tests to test the component.

There is a lot of information available about planning and writing Functional Tests. Below are some links to articles discussing Functional (and Unit) testing.

Testing Fun

How to Write Test Cases

Tips for Design of Test Data

Criteria for writing functional tests¶

There is no hard and fast rule as to how many functional test cases to write. Usually functional test cases are derived from user requirements, stories, or external documentation. Fossology does not have a lot of written user requirements or documents. So the challenge in testing Fossology is to understand how the system is //supposed// to work and then design tests to prove/disprove that it works that way.

At a minimum, functional tests should test: * all valid inputs in the ranges supported. * invalid inputs should be supplied, and the code should error or die gracefully. * all API's (if they exist) * all cli options. For example, if a program has a cli interface like
foo -h [-d dir] [-c comment] [-x] -f <file>

Then every option should be tested for valid values, and then every option should be tested for invalid values.
For example: * foo <no parameters, expect help message> * foo -h <expect help message> * -d valid-dir -f valid-file * -c comment * -x

• foo -d <invalid dir> (should produce an error)
• foo -f <no file> (should produce an error)
• foo -c <no comment> (should produce an error)

etc...

Keep your test cases small, and have them test only one thing (if possible). There should always be more than one functional test for any given module.

Test Documentation¶

As mentioned above, tests should be documented using doxygen style formatting. The documentation for the tests can be found at Test Documentation. These documents are updated every night.

No UI Functional Test¶

The table below could be tested using PHPUnit or any other test framwork, the requirement is that the test results be in junit format.

Module	% Complete	Owner	Notes
Scheduler	10%	norton	Ad hoc testing by Team
fossconfig	100%	norton	Add'l ad hoc testing by Team
adj2nest	0%	bobg	Ad hoc testing by Team
buckets	0%	bobg	Ad hoc testing by Bob
copyright	0%	norton	Ad hoc testing by Team
delagent	60%	vincent	Add'l ad hoc testing by Team
mimetype	60%	larry	case count 3
nomos	0%	bobg	Ad hoc testing by Team
sql agent (if needed)	0%	vincent	will not be tested
pkgagent	60%	vincent	Add'l ad hoc testing by Team
ununpack	50%	larry	case count 18, Add'l ad hoc testing by Team
wget_agent	60%	larry	case count 8, Add'l ad hoc testing by Team

Common UI Functional Test¶

Mark is investigating how one would test the common modules. It appears that they assume the UI is up, which may complicate how they are tested.

DTC in the table below represents "Days To Complete"

% Unit Tests Done	%Func Tests Done	Owner	file	DTC
100		Larry	common-active.php	.25
Ad hoc testing by Team		bobg	common-agents.php	2
100		vincent	common-auth.php	25
Ad hoc testing by Bob * Mary will test *		bobg	common-buckets.php	25
100		larry	common-cache.php	.25
100		larry	common-cli.php	.5
Ad hoc testing by Bob * Mary will test *		bobg	common-compare.php	5
Ad hoc testing by Team		bobg	common-db.php	0
30		Vincent	common-dir.php	25
Ad hoc testing by Team		Vincent	common-folders.php	25
Ad hoc testing by Team		bobg	common-job.php	2
100		larry	common-license-file.php	.25
* will not test *		bobg	common-license.php (bsam only)
100		vincent	common-menu.php	25
100		vincent	common-parm.php	25
Ad hoc testing by Team		bobg	common.php	0
50		vincent	common-pkg.php	25
Ad hoc testing by Team		bobg	common-plugin.php	0
Ad hoc testing by Team		bobg	common-repo.php	25
Ad hoc testing by Team		bobg	common-scheduler.php	25
70		larry	common-sysconfig.php	.25
Ad hoc testing by Bob		vincent	common-tags.php	25
100		larry	common-ui.php	.25

UI Function Test (SimpleTest)¶

TestCases	% Complete	Owner	Notes
Site Tests	100%	Mark
Basic Tests	100%	Mark
Users Tests	100%	Mark
Email Notification	100%	Mark
Verify Tests	50%	Mark	There is only 1 package test. More is needed.</br> There are only a few nomos tests, could be many more.</br> There are no bucket tests.</br>

Ref: :fossology_test_suite

Alpha testing feedback¶

Feedback from alpha tester Cat:

<cat-xeger> Suffice it to say that setting default user/password combinations is at best considered bad practice
<cat-xeger> (from a security standpoint)
<maryl> ok, so... you can click on the Admin tab to create new users
<maryl> true, we would expect the admin to change that
<cat-xeger> Yes -- having logged in now, I can see that... but having a default user and password with admin rights is a Bad Thing.
<cat-xeger> ... at the very least, the install process should let you set a different password
<cat-xeger> (yes, I know it's plaintext, but all the same)
<maryl> yes, understood...
 maryl makes note to file enhancement 
<cat-xeger> Heh.
<cat-xeger> (I seem to be suffering from the wrong end of the "I know where I live, doesn't everybody?" problem)
<maryl> lol
<maryl> not really, we know better, too. just not enough of us to address all the issues
<maryl> it's a matter of priority :-(
<cat-xeger> Heh.  Well... you also know what a running system looks like...
<maryl> we have a wonderful quickstart page on the wiki.... for 1.4.0, http://www.fossology.org/projects/fossology/wiki/Quickstart
<cat-xeger> Heh.
<maryl> it needs to be updated for 2.0
<cat-xeger> ... as does the INSTALL
<maryl> actually, everything needs updating
<cat-xeger> Heh.
<cat-xeger> It also looks like somebody indulged in the canonical sin of renaming a bunch of things, and moving the web root around without a corresponding update to the docs :(
<maryl> hm, I think the intention is to add a webroot for fossology (not move the default)
<maryl> bobg: ^ do I have that right?
<maryl> cat-xeger: this is all excellent feedback, btw  
<cat-xeger> The INSTALL points to /usr/local/share/fossology/www
<cat-xeger> ... it should be /usr/local/share/fossology/www/ui
<bobg> mary, yes.  It would be a big boo boo to move the users default.
<cat-xeger> maryl - it's still unstructured :)  <<< what did she mean here?
<maryl> ok
<cat-xeger> (I'd like to know what gives with:  /srv/fossology  ... though)
 maryl captures notes...
<cat-xeger> Heh.
<cat-xeger> Going through my notes in some sort of order...
<cat-xeger> The README points to INSTALL
<cat-xeger> "Please see the file INSTALL within the FOSSology source distribution for instructions." 
<cat-xeger> ... I looked for it in ./src ... and then ran a find to find it -- was surprised that the INSTALL wasn't in the top level directory, which it is for most apps.
<cat-xeger> It would have been very good to know what the dependencies were in advance, and without reading the code -- an extra 39 packages, and things being, for some reason, set to 'manually installed' is messy at best.
<maryl> cat-xeger: INSTALL should be just above src
<cat-xeger> maryl - it isn't.
<maryl> oops
<cat-xeger> Heh.
<cat-xeger> I elected to go through the short version of the install directions, since that's what I'd do normally.
<cat-xeger> My note is that I skipped changing the kernel.shmmax setting, but thinking about it, it's actually a meta problem, which is needing to know what the hardware requirements are for supporting fossology
<cat-xeger> (and blindly changing kernel.shmmax to a set value isn't the best of ideas, given that you really shouldn't be setting it to be larger than what you've got)
<maryl> there is a script in utils to install dependencies  
<maryl> sudo utils/fo-installdeps
<cat-xeger> I know :)  I used it.
<maryl> was it successful?
<cat-xeger> I'd have been much happier if I knew what it was going to do to my system before it went and did it.
<maryl> Mm.
<cat-xeger> It did things like:
<cat-xeger> libxml2-dev is already the newest version.
<cat-xeger> libxml2-dev set to manually installed.
<cat-xeger> ... the manually installed is going to give me grief later.
<cat-xeger> (along with hardware support, a note that it's Linux only, and anything else is at your own risk would also be nice)
<maryl> we've heard comments to the contrary... user's want it easy & automatic.  
<maryl> perhaps we need an option
<cat-xeger> The automatic is nice -- I want to know what it's going to try to install, re: deps though.
<maryl> -p : promt for ...
<maryl> prompt
<cat-xeger> Just having the dependency list in the docs would help.
<maryl> ok
<cat-xeger> ... means that I can judge what the impact on a system might be
<cat-xeger> Having it do it is great -- not knowing what it's going to do first is not so great.
…
<cat-xeger> Under 2.1.4 - preparing postgresql ... a diff doesn't tell me what the default config is, nor why things have been changed.
<cat-xeger> Actually the postgresql config comment ends up with a meta of "Tell me why I'm doing these changes" -- presumably it's about scaling or app behaviour.
cat-xeger> Ah, there's the dependencies, split up all over (looking further down through the INSTALL ... weird breakup)
cat-xeger> I can see why I didn't see the dependencies -- but adding (described in detail in section <blah>) to the short installation instructions should redirect appropriately.
<cat-xeger> ... no note that postgresql does/doesn't need a restart to pick up config changes (I presumed that it did)
<cat-xeger> re: apache config, the alias points to the wrong place, and has trailing slashes, but shouldn't... http://wiki.apache.org/httpd/FAQ#Why_does_accessing_directories_only_work_when_I_include_the_trailing_.22.2BAC8.22_.28e.g..2C_http:.2BAC8ALw-foo.domain.com.2BAC8Afg-user.2BAC8.29_but_not_when_I_omit_it_.28e.g..2C_http:.2BAC8ALw-foo.domain.com.2BAC8Afg-user.29.3F
<cat-xeger> The make and make install ran cleanly, and quickly enough to make me nervous.
<maryl> any output?
<cat-xeger> The post install, OTOH, left me wanting to rip my hair out.
<cat-xeger> Tons of it, none of it particularly interesting.
<maryl> yup
<cat-xeger> I really, Really, REALLY _HATE_ software that creates system users/groups on me
<cat-xeger> ... never mind creating directories for said user in places that I never put directories for users
<cat-xeger> ... and then adding system users to that group.
<maryl> i c
cat-xeger> I don't want to have things randomly littered around my systems, and may also have standards about what goes where
<maryl> we do that to solve a chicken and egg problem... and to make it easier on the installer
<cat-xeger> ... and if I'm using something other than local auth, it could cause problems.
<cat-xeger> I'd really prefer to see something along the lines of "About to create user fossy, proceed (Y/n):" 
<cat-xeger> ... and ditto for the group, with an exit option that boils down to "You must create the following before continuing with the install" 
<cat-xeger> _especially_ for a user that's been created with an active shell.
<cat-xeger> Do you know if there's a reason for fossy to have an active shell btb?
...
<cat-xeger> Also:
<cat-xeger> mysql:x:112:120:MySQL Server,,,:/var/lib/mysql:/bin/false
<cat-xeger> postgres:x:113:121:PostgreSQL administrator,,,:/var/lib/postgresql:/bin/bash
<cat-xeger> (I installed mysql for something else -- and you can see that it's set to /bin/false, unlike postgresql)
<maryl> when fossology is setup as a cluster, the scheduler needs to be able to start jobs on the clients, as fossy, with ssh (using keys that the user must setup)
<maryl> I *think* that's a postgres-ism
<cat-xeger> I'd call that out somewhere, then.
<maryl> not us
<cat-xeger> Probably :)  I don't think it's something you did... just happened to notice it.
<maryl> yup
<cat-xeger> # /usr/local/lib/fossology/fossology-scheduler -t
<cat-xeger> ... doesn't exist anymore, got renamed to:  /usr/local/share/fossology/scheduler/agent/fo_scheduler
<cat-xeger> ... but even with the renamed one, I haven't the foggiest clue if the tests ran, since it seemed to just go off into space when I ran fo_scheduler -t
<maryl> yes, you're using an outdated INSTALL with old paths... :-(
<cat-xeger> (breaking out w/ ctl-c, and running it again seemed to work)
<maryl> It *should* have written something to the fo log file /var/log/fossology/fossology.log
<cat-xeger> ... so it'd be good to have some idea if it's running tests or doing something...
<cat-xeger> Nope, it didn't write anything there (rando pointed me there)
...
<cat-xeger> I think I only had two other things, both of which are arguably bugs
<cat-xeger>  /etc/init.d/fossology start returns success when run as a non-root user, which is wrong
<cat-xeger> (since it doesn't start)
<bobg> k, I'll file that bug
<cat-xeger> ... and all of the files under /usr/local/share/fossology/www are owned by root:staff, and the ui directory is setgid
<bobg> k, that's wrong.  I'll file that too
<cat-xeger> ... and now that I know how to login (rant already glossed over here), I'll try some uploads yada this evening.
<bobg> thanks cat.  We appreciate it a lot.
 cat-xeger *grins*
<cat-xeger> Glad to help :)
<bobg> btw, I'm not seeing the /etc/init.d/fossology problem.  It returns 0 on success and 1 on failure for my initial tests (start, and bad options).  But I'll do some more.
<cat-xeger> bobg - as yourself, or as root?
<bobg> as myself.  But I am in group fossy
<bobg> http://www.fossology.org/issues/729
<cat-xeger> $ /etc/init.d/fossology start
<cat-xeger> Starting FOSSology job scheduler: scheduler.
<cat-xeger> $ echo $?
<cat-xeger> 0
<bobg> and did the scheduler start?
<cat-xeger> Nope.
<bobg> k
<cat-xeger> The start) doesn't check exit codes.
<bobg> http://www.fossology.org/issues/735
 cat-xeger submits a registration :)

Feedback from alpha tester Will

Response to Will.  (Original email indented below.)

Prior to installing 2.0.0, you should run "sudo utils/fo-cleanold" w/no options.  This will clear out the old 1.4.1 executable files, but leave your database & repository intact.  Re-run "sudo make install".

After "sudo make install", you must also run "sudo install/fo-postinstall" 
Since you installed on a system with 1.4.1 previously installed, the apache Config is still pointing to the 1.4.1 UI.  You must change the Config to point to the new 2.0.0 files. Something like this:

    Alias /repo/ /usr/local/share/fossology/www/ui/
    <Directory "/usr/local/share/fossology/www/ui/">
        AllowOverride None
        Options Indexes FollowSymLinks MultiViews
        Order allow,deny
        Allow from all
        # uncomment to turn on php error reporting
        # php_flag display_errors on
        # php_value error_reporting 2039
    </Directory>

Good feedback to incorporate into our source install documentation.
Thanks Will!
Mary

> -----Original Message-----
> 
> I have followed your download link, untarred in my home directory and 
> run make; sudo make install; sudo utils/fo-installdeps. So far so good 
> as far as I can tell, however http://localhost/repo/ Help->About still 
> shows FOSSology version
> 1.4.1 (code revision 5423).  What did I do wrong?