IEEE Standard 610 (1990) defines test case as follows:
“(1) A set of test inputs, execution conditions, and expected results developed for a
particular objective, such as to exercise a particular program path or to verify compliance
with a specific requirement.
“(2) (IEEE Std 829-1983) Documentation specifying inputs, predicted results, and a set
of execution conditions for a test item.”
According to Ron Patton (2001, p. 65),
“Test cases are the specific inputs that you’ll try and the procedures that you’ll follow
when you test the software.”
Boris Beizer (1995, p. 3) defines a test as
“A sequence of one or more subtests executed as a sequence because the outcome and/or
final state of one subtest is the input and/or initial state of the next. The word ‘test’ is
used to include subtests, tests proper, and test suites.
Bob Binder (1999, p. 47) defines test case:
“A test case specifies the pretest state of the IUT and its environment, the test inputs or
conditions, and the expected result. The expected result specifies what the IUT should
produce from the test inputs. This specification includes messages generated by the IUT,
exceptions, returned values, and resultant state of the IUT and its environment. Test cases
may also specify initial and resulting conditions for other objects that constitute the IUT
and its environment.”
In practice, many things are referred to as test cases even though they are far from being fully
documented.
Brian Marick uses a related term to describe the lightly documented test case, the test idea:
“A test idea is a brief statement of something that should be tested. For example, if you're
testing a square root function, one idea for a test would be ‘test a number less than zero’.
The idea is to check if the code handles an error case.”
In my view, a test case is a question that you ask of the program. The point of running the test is
to gain information, for example whether the program will pass or fail the test.
It may or may not be specified in great procedural detail, as long as it is clear what is the idea of
the test and how to apply that idea to some specific aspect (feature, for example) of the product.
If the documentation is an essential aspect of a test case, in your vocabulary, please substitute the term “test idea” for “test case” in everything that follows.
An important implication of defining a test case as a question is that a test case must be
reasonably capable of revealing information.
- Under this definition, the scope of test cases changes as the program gets more
stable. Early in testing, when anything in the program can be broken, trying the
largest “legal” value in a numeric input field is a sensible test. But weeks later, after
the program has passed this test several times over several builds, a standalone test
of this one field is no longer a test case because there is only a miniscule probability
of failure. A more appropriate test case at this point might combine boundaries of ten
different variables at the same time or place the boundary in the context of a longsequence
test or a scenario.
- Also, under this definition, the metrics that report the number of test cases are
meaningless. What do you do with a set of 20 single-variable tests that were
interesting a few weeks ago but now should be retired or merged into a combination?
Suppose you create a combination test that includes the 20 tests. Should the metric
report this one test, 20 tests, or 21? What about the tests that you run only once?
What about the tests that you design and implement but never run because the
program design changes in ways that make these tests uninteresting?
The goal is information. Very often, the information sought involves defects, but not always. (I
owe this insight to Marick, 1997.) To assess the value of a test, we should ask how well it provides the information we’re looking for.
