A guide to Regression Testing
What is the purpose of regression testing?
Regression testing is considered to be a Black-Box technique, meaning that testing is performed with no exposure to the software code, i.e. similar to what the end user will see. The goal of regression testing is to ensure that any changes made to the code have not adversely affected the existing behaviour of the software. Regression testing provides several benefits. It can help confirm stability in the software, it can help mitigate risks, it can identify regression defects.
When should regression testing be performed?
Regression tests are typically performed on a new build that contains changes of some kind. These changes can be new functionality that was introduced, or perhaps defect fixes. Regression tests are often performed at the end of a cycle or sprint. However, they can be used whenever deemed necessary, even as often as daily if there is a risk of new code breaking existing functionality.
How are regression tests designed?
Regression tests are typically derived from a selection of existing functional tests. They are selected to ensure sufficient coverage of the software product is obtained. To achieve the desired level of coverage, additional test cases can be added. For example, additional usability, compatibility, accessibility tests could be added based on stakeholder feedback.
Normally, a set of regression tests is significantly less than the total amount of functional test cases. As the goal is to ensure that nothing is broken, careful selection of test cases to form the regression test can achieve the desired result. However, sometimes a situation may arise where a “full regression” of a software product is needed, and this may require re-running the entire set of functional test cases rather than a sub-set, this is commonly known as a “re-test”.
Are regression tests manual or automated?
Regression tests can be either manual or automated. While automating regression tests can save a lot of time during test case execution, it does carry an overhead of maintaining them. Consider an app that is of medium complexity. It may have several hundred regression tests, even the slightest change in code can impact many such test cases. Therefore, automated regression tests require constant maintenance in order for them to be effective and above all, usable.
Time is often a factor with regression testing, as it can be a bottleneck in the developmental process.While running regression tests manually will take longer to execute than via automation, with careful resource planning, this risk can be mitigated by allocating additional testers to the regression testing task.
Which is best? manual or automated?
Sometimes, fewer actual regression tests can be used for manual testing instead of automated testing. This is because the test cases can be written at a higher-level allowing the individual tester to choose their own path through the test itself. For example, a regression test to check the login flow could simply be written as “Test login flow”. Now, each tester may focus more on different areas such as positive / negative flows, field validation, boundary value analysis etc. The result is that there are fewer manual tests than automated tests, but the great thing about this approach is that each time the test cases is ran, it is ran slightly differently depending on the individual testers approach. This results is greater test coverage and increased likelihood of finding an issue.
The key benefit of automated testing is time saved while executing the test cases. However, additional time must be spent automating the test cases and maintaining them. While manually running regression tests does take longer, with the right approach, the test cases don’t become stale because each test case can be designed to allow some openness to interpretation for each tester to utilize. Automated tests if not properly maintained can become stale and will eventually run the risk of missing defects.