Regression testing validates that no new defects have been introduced to the system as a result of modifications. Regression testing focuses on the key functionality of the software.
In order to resolve some common misunderstandings, I will start with a brief description of the testing terms debugging and regression testing.
Testing and debugging
Testing is done by the tester and detects defects. The developer does debugging – he identifies the cause of the bug/defect in the code, and makes corrections, in order to fix the problem. After the fix, the changes in the software needs to be retested, to confirm that the problems have been removed. This activity is called retesting or confirmation testing.
What is regression testing?
Every time a change is made to software that change invalidates all the previous testing done before. Why? Because the impact (if such exists) of that change is unknown.
When doing regression testing we do not test all functionality. Instead, we focus on testing the important functionality and the high-risk functionality, while ignoring the more complex functionality. If the application under test is MS Word, we focus on functionality like opening documents, writing documents and saving documents. This is critical functionality, without which MS Word could not function at all. If the application under test was mission-critical one could justify testing everything including complex functionality in every regression testing test cycle, but when dealing with “normal” software this strategy is to expensive.
It´s crucial to implement the right testing strategy. A testing strategy, where you skip regression testing, increases the risk of not detecting hidden defects. I have two examples on how to deal with changes to the system – one without regression testing, and one with regression testing. Let´s start with the example with no regression testing.
No regression testing – hidden defects
In Figure 1, we have a scenario with three test cases and three test cycles. We only test the fixes – meaning we do not run every test case in every test cycle.
In test cycle 1 test cases A and C passes, but test case B fails. After test cycle 1 has ended, the developer is trying to fix the defect, that test case B detected. When trying to fix the defect in test case B he introduces a new bug in the functionality which test case C covers. Test cycle 2 begins. We only test fixes, meaning we do not run test cases A and C (they passed in test cycle 1). In test cycle 2 we realize that test case B still fails. We are only testing the fixes, and therefore do not run test C once again. By skipping test case C, we do not detect that a new problem has been introduced in the functionality covered by test case C. We ran test case C in test cycle 1 – but the defect in test case C was created after test cycle 1.
In test cycle 3 we are still only testing fixes. We do not run test case A and C. We test the fix in test case B. The fix worked and test case B passes. We conclude that the software is ready for release. In production, the users detect the hidden bug – the defect in test case C
In this example, we only ran all the test cases in test cycle 1. Let´s go further into detail with what happens, when we run all test cases in every test cycle.
Regression testing – no hidden defects
In Figure 2 we once again have a scenario with three test cases and three test cycles. In each test cycle, we run all the test cases.
Test cycle 1: test case B fails, but test case A and C Passes. We are doing regression testing, meaning in test cycle 2 test case A is tested again. We also run test case B again. The assumption was that the defect was fixed, but our test reveals that´s not the case. We also run test case C and detect a defect. We failed to fix the defect for test case B. Because we run all the test cases in test cycle 2 we are able to detect the new defect in test case C. In test cycle 3 we validate that both defects are fixed.
Because of regression testing we now detect both defects in test cycle 2. This example illustrates the importance of running every test case in every test cycle to make sure that new problems do not go undetected. It´s a precondition that the defect in test case C is introduced in the systems key functionality. Another precondition is that test case C is testing exactly the functionality in which the developer introduced a defect, during his attempt to fix defect in test case B. In real world, maybe this is tested in a test case not included in the regression test. This is where the testers experience and knowledge about the system comes into play. One thing though is sure: we only detect the defect introduced in test case C, if we are doing regression testing.