Software Engineering
Home Planning Requirements Writing Hazard Analysis Requirement Analysis Config Control Software Design Software Testing Software Standards Basic Logic

Software Testing - Philosophy

Product Based Testing vs. Documentation Based Testing

In Basic Logic we’ve proven that we cannot prove anything to be either correct or incorrect! Even if we test all possible conditions (which would exceed the life expectancy of the solar system), we still cannot be certain that our test environment was completely correct. So why do we pay people to test?

Since we cannot prove anything to be either correct or incorrect, we must rely on empirical evidence to support our conclusion that the system is correct within a given context (such as the system's intended use). Empirical science is not based on the validity of proofs (proof is what you did in Geometry class). Instead, it is based on the usefulness of accepting the evidence as being sufficient to accept the Research Hypothesis as true, also accepting the possibility of error. To help ensure that our evidence is sufficient, we need a test philosophy.

What is Quality?

If a tree falls in a forest and hits my brother, will my brother make a noise? And will anyone care?

If a system contains a defect, and the customer is never affected by the defect, then does the defect exist? Does the system have “quality?”

Choosing a Test Philosophy

Product based testing takes the view of minimizing the risk to the customer. In contrast, Documentation based testing takes the view of minimizing the defects in project documents. Both approaches use documentation; only the focus changes. The philosophy chosen must be based on the particular project's needs. To choose a philosophy, consider these points:

  • Type I errors result in systems being shipped with more defects.
  • Type II errors result in test documents with more defects.
  • From Basic Logic - reducing one type of defect will increase the other type of defect.
  • Amount of documentation needed (e.g. regulated industries, medical devices).

Therefore:

  • Product-based testing must focus on reducing actual defects (Type I errors) and accept the increase in documentation flaws (Type II errors). Product based testing produces actual quality.
     

  • Documentation based testing must focus on reducing documentation flaws (Type II errors) and accept the increase in actual product defects (Type I errors). Documentation based testing produces the appearance of quality.

Product based testing focuses on eliminating the Type I errors, even though it produces more Type II errors. The Type I error is the condition that will most adversely affect the customer, since bad output may be mistaken as good output, or the system may become unusable. This also makes product based testing essential for any safety or mission critical systems. If our approach works, then we should see an increase in anomalies that are actually documentation defects (more Type II errors).

Documentation based testing focuses on eliminating Type II errors, even though it produces more Type I errors. If our approach works, then we should see an increase in customer complaints as they find the defects that we missed (more Type I errors). However, some regulated environments may not permit documentation anomalies. I have seen some organizations use product based testing, retrospectively update the documents to remove the documentation related defects, then repeat the testing. However, since time is a limited resource for any project, this approach led to testers simply reducing the test coverage to make up for the lost time, which in turn increased the number of Type I errors. The only solution is to throw more time into the schedule - PAD, PAD, PAD that schedule!

True Horror Story

At one company, a QA team wrote a validation procedure for a series of switchboxes that would be installed at the customer's site. The switches allowed the customer to select between various computers, each holding a different product that was under evaluation.

The validation documents were beautiful! The formatting, the detailed description (descriptive testing) - these documents were works of art that anyone would be proud to display!

They also failed to test one simple condition - what if Box 1 is set to use Product A but Box 2 is set to use Product B? As it turned out, this made it possible to get incorrect results without ever knowing it! This was a serious problem!

Fortunately, once our software group saw the documents, we could see that QA was doing documentation based testing. We generated our own test protocol, which was product-based. That was how we found the problem, enabling us to develop a solution.