Software Engineering
Home Planning Requirements Writing Hazard Analysis Requirement Analysis Config Control Software Design Software Testing Software Standards Basic Logic

Basic Logic - Summary

When is the Evidence Enough?

Consider the original proposition behind the parts of an argument:

IF the premises are true AND the argument is valid THEN the conclusion is true.

Suppose that we already know the conclusion to be true. Does that prove that either the premises were true or that the argument was valid? No – that is the formal fallacy of affirming the consequent.

Also remember that the concept of hypothesis testing only relates to the validity of the argument. In this case, the argument was that IF the system works correctly THEN we will have the expected results. For the conclusion to be true, the premises must also be true. In other words, the system must actually work correctly.

Every time we see the expected results, it lends weight to the research hypothesis. Eventually, the weight of evidence passes our “test of truth” threshold, which is our own value judgment as to how much evidence is needed to demonstrate that we have a good test environment.

We can never prove that the system works correctly – that is an invalid argument. However, by failing to prove that we can break it (rejecting the null hypothesis) we can produce evidence to accept the research hypothesis that the system works correctly. But when is it enough?

Methods of ensuring a good test of truth include:

  1. Requirements Analysis to ensure good test coverage.
  2. Peer review of test objectives to gain consensus on test coverage.
  3. Design tests that have the highest probability of revealing defects.
  4. Ensure that the test conditions (the premises) are appropriate and reproducible. Remember, hypothesis testing only ensures the validity of the argument. The premises must still be true for the conclusion to be true. (Garbage In – Garbage Out.)

 Points to Remember:
Failure to test a given condition constitutes an assumption that B is True for that condition, regardless of the value of A. This allows Type I errors to occur more frequently.

Testing is sampling. The population is the total number of times that the system will ever be run by all users throughout the entire product life.

We can only have a given level of confidence that the system is actually correct. We can never be 100% confident that a Type I error didn’t occur. Even if we test every possible condition, we can never be certain that the test environment itself is 100% correct (per Godel's Incompleteness Theorem). Basically, provability is a weaker notion than Truth.


No amount of observations of white swans is sufficient to verify the proposition "All swans are white." However, a single observance of a black swan is sufficient to disprove it. (John Stuart Mill)

From Ruckner, Infinity and the Mind:

The proof of Gödel's Incompleteness Theorem is so simple, and so sneaky, that it is almost embarassing to relate. His basic procedure is as follows:

  1. Someone introduces Gödel to a UTM, a machine that is supposed to be a Universal Truth Machine, capable of correctly answering any question at all.
  2. Gödel asks for the program and the circuit design of the UTM. The program may be complicated, but it can only be finitely long. Call the program P(UTM) for Program of the Universal Truth Machine.
  3. Smiling a little, Gödel writes out the following sentence: "The machine constructed on the basis of the program P(UTM) will never say that this sentence is true." Call this sentence G for Gödel. Note that G is equivalent to: "UTM will never say G is true."
  4. Now Gödel laughs his high laugh and asks UTM whether G is true or not.
  5. If UTM says G is true, then "UTM will never say G is true" is false. If "UTM will never say G is true" is false, then G is false (since G = "UTM will never say G is true"). So if UTM says G is true, then G is in fact false, and UTM has made a false statement. So UTM will never say that G is true, since UTM makes only true statements.
  6. We have established that UTM will never say G is true. So "UTM will never say G is true" is in fact a true statement. So G is true (since G = "UTM will never say G is true").
  7. "I know a truth that UTM can never utter," Gödel says. "I know that G is true. UTM is not truly universal."

Think about it - it grows on you ...

With his great mathematical and logical genius, Gödel was able to find a way (for any given P(UTM)) actually to write down a complicated polynomial equation that has a solution if and only if G is true. So G is not at all some vague or non-mathematical sentence. G is a specific mathematical problem that we know the answer to, even though UTM does not! So UTM does not, and cannot, embody a best and final theory of mathematics ...

Although this theorem can be stated and proved in a rigorously mathematical way, what it seems to say is that rational thought can never penetrate to the final ultimate truth ... But, paradoxically, to understand Gödel's proof is to find a sort of liberation. For many logic students, the final breakthrough to full understanding of the Incompleteness Theorem is practically a conversion experience. This is partly a by-product of the potent mystique Gödel's name carries. But, more profoundly, to understand the essentially labyrinthine nature of the castle is, somehow, to be free of it.