Testing Code

Untested software is bound to fail. Testing is a central construction technique.

Terminology - An error is something that you do wrong. It is a specific human action that results in software containing a fault. For example, forgetting to check a condition is an error. A fault is the consequence of an error. A failure is the departure of the program's operation from its requirements and may be the consequence of a fault. A bug is often used as a synonym for a fault.

Testing is a methodical process of providing the existence or lack thereof of faults in the software. Debugging is the act of tracking down the cause of this faulty behavior.

Good programming leads to more time spent testing than debugging.

QA: quality assurance. It is a name given to both a tribe of software factory inhabitants and a development practice. Testing is a small part of QA. QA is about prevention. QA ensures that the processes and development practices result in high-quality software.

An important thing to realize is that tests can only discover the presence of faults. It cannot prove the absence of faults.

Developers should not send code to QA that has not been tested. You must test every line of code that you write. Do not expect anyone else to do it for you.

Test code is the code we write to test the code. Code should be tested as it is written to catch errors at the first opportunity. This is also the time when the errors are easiest to fix and affect the fewest people. Testing early and thoroughly is the most effective way to ensure software quality.

Test-driven development advocates that you write the test code before the code being tested. This is a vital point to absorb into your programming routine. For each piece of code you write, do not move on until it is tested and you know that it works. Thinking about testing code at the same time as you think about writing some code will shape the way you design that code. Any fault that manages to slip past the existing tests must have a new test added to prove that the bug fix is correct. Write a test for every fault that you find.

Tests should be run as often as humanly possible and possibly even more often than that. This is embodied in a continuous integration strategy. Run your tests as often as you can.

The test harness must demonstrate these two things:

Read all code cynically. Do not accidentally read what you meant to write rather than what you wrote.

No matter how hard you test, you will miss something. Studies show that most carefully tested software contains 0.5 to 3 errors per 1000 LoC. Testing rarely proves that software is bulletproof. Only that it is adequate.

Unit tests are tests of atomic units: each class or function. Things are tested in strict isolation. Any untrusted external code with which the unit interfaces is replaced with a stub or simulator.

Component testing is a step up from unit testing. This validates the combination of one or more units into a full component. Often this is what people mean by unit tests.

Integration tests test the combination of components as they are brought together in the system.

Regression testing is retesting after fixes or modifications.

Load testing ensures that the code can handle the expected volume of data being thrown at it.

Stress testing throws a huge amount of data at the code within a short time to see what it does. Whereas load testing checks that code can meet expected demands, stress testing is about what will happen when it receives a real battering. It has to fail gracefully and recover well.

Soak testing involves a high load for a prolonged period of time. It may reveal faults that otherwise would go undetected.

Usability testing ensures that the software can be used easily by a shortsighted gerbil.

Black box testing, also known as functional testing, compares actual functionality against intended functionality.

White box testing is also known as structural testing. It is a code-coverage-based approach where each line of code is scrutinized to ensure correctness. Sometimes this is called glass box testing. There are static and dynamic methods of white box testing.

Remember that boundary cases are a rich source of error. Identify the highest and lowest inputs that are valid. Test behavior at the boundary, values above, and values below.

It can be a good idea to test randomly generate input data.

Always test for the zero case when the input is numeric.

The quality of unit tests you can write is largely determined by the quality of the interface you have to test. Testing is easier when the code is written thoughtfully and specifically designed to accommodate inspection and verification. You must design for tests up front.

Some design rules that lead to highly testable code: each section of code should be self-contained and without undocumented and tenuous dependencies on external things. Don't rely on global variables. Limit the complexity. Make the code observable.

The golden rule of testing is automate. Automate testing as much as possible.

Good programmers write tests for all their code. They test at the micro level so that macro level testing is not hindered by stupid mistakes. They care about product quality and take responsibility for it.

Bad programmers don't consider testing to be important. They release untested code. They make their lives more complicated by discovering problems too late.