Software Testing - Software Validation

• What is "Validation?"
• Does my software need validation?
• What constitutes validation?
• How do I validate third party software?

What is "Validation?"

In Basic Logic, we learn that no amount of testing can prove that the software works correctly. It can only prove that it doesn't. However, who would buy software from a sales pitch like: "Buy our software! We failed to break it!" Or how about the FDA auditor (U.S. Food & Drug Administration) who wants to know if our software works correctly? How many auditors would accept the line:"We cannot prove that it's bad!"

Validation is the presentation of objective evidence that the software is suitable for its intended use. This is not the same as proving that the software works correctly. In fact, the software may be proven to have several defects, yet still be suitable for its intended use.

However, in the world of software testing, validation serves no purpose. That's because:

Testing tries to show that the software is broken.
Whereas, validation tries to show that the software is suitable for use.

Therefore, if we test software with an eye towards "validation," we can be sure to miss critical problems. That is because by trying to "validate," we try to prove that it works - when we should be trying to prove that it doesn't! Only by failing to break the software can we consider the possibility that it might work correctly.

So what is "validation," really? Validation is the demonstration that the software (or system) is suitable to perform a specific real-world task. Validation is purely a black box concept, and relates not to how the software is made, or even if it works correctly - it relates to the software's usefulness in performing the business task.

Back to Top

Does My Software Need Validation?

If you live in the medical industry - it's simpler to accept that the answer is always "Yes." Some people would argue that their software does not need validation; but, consider the viewpoint of the auditors who will be asking about your validations. What always happens when we hear the "No" word? We immediately erect barriers. We begin forming counter-arguments. We just "know" that the other person is wrong! Sometimes, we can even feel how wrong they are! So, regardless of how sound your reasoning might be, there are only two answers worth considering:

1. This software (or system) is not part of the Quality System and does not impact the product (like financial software), OR...
2. Yes, and validation consists of ... (the rest is negotiable). Note that by saying "yes," you will automatically be recognizing the auditor's position on validation. And remember, the auditor doesn't have to perform the validation, and so remains undaunted by the magnitude of the task. All the auditor has to do is say "I think it has to be validated." So it's better to assume that "No" is always the wrong answer. Say "yes," and then define the scope of validation for that particular software (or system).

Back to Top

What Consitutes Validation?

Since "validation" is not "proof" and it is not "testing," just what is it?

Imagine that you've just given me a brick and have asked me to validate it. Ok, it's a nice brick, red, heavy, and rectangular. I can weight it, measure it, throw it through a few windows to make sure it works, then send it back to you validated! Ecstatic to have your validated brick, you then try to use it to measure the air pressure in your tires. Good brick - but not suited to that intended use.

Another way of saying this is that we made the thing correctly, we just didn't make the correct thing. In testing terms, showing that we made a thing correctly is "verification," whereas showing that we made the correct thing is "validation."

To validate anything, you must have:

1. An intended use for the thing that is being validated. The "intended use" is the real-world task that the thing will perform. In the software world, these would be the business needs that the software fills. This is NOT the software requirements - the requirements only describe a "brick," not the job that the "brick" was meant to perform. Requirements are "verified" (showing that we made the software correctly), not "validated." Instead, software requirements carry the embedded assumption that a system built to those requirements will satisfy the business need.
2. Objective evidence that our thing is suitable to fill its intended use. If, despite an army of known defects, our thing is still shown to be suitable for our intended use, then it is still validated. Note that our thing is only validated for a specific intended use. If the intended use changes, we need to evaluate our thing to see if it needs re-validation; it may also need changes to be suitable for a different intended use.

As you can see, validation cannot be separated from the intended use. This makes the statement of the intended use the most important part of any validation. Without a statement of the intended use, validation is not even possible. Any attempt to validate a system without first stating its intended use constitutes simple acceptance of whatever software the developers were kind enough to deliver. Even if the software is perfect, it can only be considered verified, not validated.

Back to Top

How Do I Validate Third Party Software?

I have seen many companies go berserk over the question of validating third party software. However, the task is really very simple.

1. Articulate (in writing) what real-world task the software is there to solve. This may include a list of features that the software must provide (such as requiring user login).
2. For medical device companies, consult the FDA Guidance documents to determine the level of concern, and to see if you need to include any risk mitigations. Consider also whether the software is itself a mitigator for a risk, rather than a generator of risk. For example, a spreadsheet mitigates the risk of errors in manual calculations.
3. Demonstrate, through objective evidence, that the software correctly performs the real-world task. This may involve testing and/or review of the manufacturers documentation.

Where most companies go wrong when validating third party software:

• BAD - Trying to test every feature that the software has. This is a classic mistake. Most third party software is written to apply to a wide range of customers, and will therefore have more features and possible configurations than you can shake a stick at. Do NOT try to test them all! When validating third party software, the point of the validation is to show that the software, when installed and configured in this manner, is suitable for performing the task. (Many companies never even try to validate third party software, for fear that they have to test every feature, which would take them months or years. As a result, they "brain freeze" and never start.)
  
• BAD - Thinking that you need the source code, or some validation statement from the vendor. Unless you can afford to buy the intellectual property rights, forget getting the source code. Remember, in validating third party software, the most likely errors (and the hardest to fix) are: a) we bought the wrong thing, or b) we configured it wrong. It is generally safe to assume that the software vendor wrote it correctly - so focus on whether its right for the task at hand.
  
• BAD - Separating software from hardware for validation. Many software systems control hardware, and many hardare systems contain embedded software. Do NOT try and separate them. After all, you bought a system - and validation is supposed to demonstrate that the system, as a whole, performs its intended use. Again - focus on the task being solved by the system. If it contains a defect that impacts the intended use, you'll find out. If it contains a defect that does not impact intended use - it doesn't matter.

Back to Top