Basic Logic - Reducing Type I and Type II Errors
Reducing Type I Errors
Prescriptive testing is used to increase the level of confidence, which in turn reduces Type I errors. The chances of making a Type I error are reduced by increasing the level of confidence that the event A and measurement B are within our control and are not being caused by chance or some other external events. This results in more stringent criteria for rejecting the null hypothesis (such as specific pass/fail criteria), thereby resulting in more times where we fail to reject H0 – with a resulting increase in the number of Type II errors (failing to reject H0 when it was really false and should have been rejected)!
Reducing Type II Errors
Descriptive testing is used to better describe the test condition and acceptance criteria, which in turn reduces Type II errors. This increases the number of times we reject the Null hypothesis – with a resulting increase in the number of Type I errors (rejecting H0 when it was really true and should not have been rejected).
Therefore, reducing one type of error comes at the expense of increasing the other type of error! THE SAME MEANS CANNOT REDUCE BOTH TYPES OF ERRORS SIMULTANEOUSLY!
Ergo: If we never find anomalies during testing (and therefore no Type II errors), then we probably have lots of Type I errors. (e.g. a descriptive test process can eliminate Type II errors at the cost of allowing Type I errors.)
Questions to ask when designing your test methodology:
- Which would you rather have: a) a sensitive fire alarm that goes off at times when there is no fire (Type II error – false alarms), or b) an in-sensitive fire alarm that may fail to go off in a fire (Type I error – missed alarms)?
- Which type of error is easier to live with in system testing: Type I (software defect that was missed) or Type II (anomaly in testing; however, there was no defect)?
|
