While everyone knows the theoretical limitations of testing, in practice we devote considerable effort to this task, and would consider it foolish or downright dangerous to skip testing.
Other verification techniques, such as static analysis, model checking and proofs, have great potential, but it is unlikely they will ever fully remove the need for testing. In the meantime we need to understand the scope and limitations of tests, and perform them right.
The seven principles that follow emerged from experience of studying testing and developing automated testing tools (AutoTest, CDD) over the past few years.
Principle 1: Definition of software testing
To test a program is to try to make it fail.
Principle 2: Tests versus specs
Tests are no substitute for specifications. One can derive tests from specifications, not the other way around.
Principle 3: Regression testing
Any failed execution must yield a test case, to remain permanently part of the project’s test suite.
Principle 4: Applying oracles
Determining success or failure of tests must be an automatic process.
Principle 4 (variant): Contracts as oracles
Oracles should be part of the program text, in the form of contracts. Determining success or failure of tests should be an automatic process consisting of monitoring contract satisfaction during execution.
Principle 5: Manual and automatic
An effective testing process must include both manually and automatically produced test cases.
Principle 6: Empirical assessment of testing strategies
Evaluate any testing strategy, however attractive it may appear in principle, through objective assessment based on explicit criteria and a reproducible testing process.
Principle 7: Assessment criteria
The most important property of a testing strategy is the number of faults (or, if faults are not directly measurable, failures) it uncovers as a function of testing time.
Reference: Bertrand Meyer
Quote:
Any fool can write code that a computer can understand. Good programmers write code that humans can understand. Martin Fowler (Refactoring: Improving the Design of Existing Code)
