Testing Basics¶

This is a brief discussion of how to plan and create test cases.

Creating Test Plans and Test Cases¶

Before you can write a test, it's a good idea to have a test case in mind that needs testing. Everyone on this team already does this when they test informally, they just might not think of it that way. For example, I'm going to run X, and I expect Y from the data that it will process. That's a test case. Of course test cases can get much more complex than that.

Test cases are usually written down, either informally or formally. Writing them down in a simple text test plan is usually good enough. Wikis are great places to store test plans. There are several advantages to having your test cases written down. The first is the it provides a record of what you plan to test. Hopefully all the documented test cases will get implemented, but if not there is a record of them so that when time allows they can be implemented. The second advantage is that when a defect occurs after your code passes all the tests, the test plan provides an easy way to examine why the tests didn't catch the defect and provides a place to document new test cases added to fix defects. The third advantage is that a test plan allows one to look at all the test cases and pose the question "what else needs to be tested"? This is hard for most humans to do if there are more than about 3-7 items to consider that have to be stored in our head. I'm sure you can think of other great reasons why it makes sense to have a written test plan. (A test plan in a wiki is a written test plan).

Where do test cases come from?¶

The sources that enable test case creation come from user requirements, external reference documents (man pages, api specs, etc.), internal specifications and defect reports. So, for example, if the man page says the program always creates a log file then one can test if that is true. You can read code to help generate test cases, but that is not very efficient.

This sounds very old school but it works. Write the code, write the man page, write the tests, run tests/add test till all tests pass, user requirements and program metrics are met. Release! Even Xtreme and Agile methods use this, sometimes with a different order.

All methods assume that the user requirements are understood and will be understood before final code. Rapid prototyping for example allows one to learn requirements as one goes but still starts with an initial set of requirements. The point here is that if the requirements are captured in some way, it allows the programmer or tester to use the requirements to generate test cases.

When to Write a Test?¶

• In general, all new code should have at least 1 test (see the section below for details). Specifically
  • all new functions
  • all changes to existing code
    • defect fixes
    • feature modification
    • clean up
  • does not yet meet code quality criteria. For example, percentage of code coverage.

How Many Test Cases?¶

There is no hard and fast rule as to how many test cases one should write. Here are some basic guidelines:

• For any type of testing, if a defect is found by ad hoc testing (i.e. while not running your tests), write a test case for that situation.
• If all tests have been written and run, functionality is still not solid and defects are being found at an unacceptable rate, then re-examine your tests and test plan, something was missed!
• For every defect a test should be checked in along with the code. The fixed code should not be checked in without a test!
• For Functional tests
  • have the user requirements been met?
  • are all options in the external specs tested? Why not? (Maybe some are too hard or too expensive to test).
  • have destructive tests been written (try to do something the application says you cannot do or test a limit, e.g. no more than 10 files for input, so give it 11, then give it 0, what happens?)
  • has the agreed upon Code Coverage metric been met?
  • has the defect per lines of code metric been met? For example, there will be no more than 1 defect per 1000 lines of code.
• For Unit tests
  • have all function parameters been tested and verified?
  • have all function return values been tested and verified?
  • any special cases identified and tested (file system full, db is gone, etc...)
  • has the code coverage metric been reached?
• For System Tests
  • Does the system perform it's intended function?
  • Does the application meet it's reliability metrics?
  • has the agreed upon Code Coverage metric been met?
  • has the defect per lines of code metric been met?

For a given type of test (e.g. Unit test), if the agreed upon team metrics have not been met, then there are more tests to write.