What is Regression Testing? Tools, Definitions & Examples

March 15th, 2021

The importance of testing, and specifically regression testing, is hard to over-state. We'll talk about regression testing, what it is, what is isn't, and offer a few recommendations for how to best implement it in your organization.

We’ve all had the experience where we’ve used an app, hit a bug, and wondered “How did nobody notice this before?". As they say, the best defense is a good offense, and when it comes to catching bugs that’s no different. In this article we’ll talk about regression testing, your best defense for catching bugs in your applications before users find them.

What is a regression test?

Regression tests validate that the existing functionality in your application still works as expected. Unlike something like security testing or performance testing where it’s clear when a defect is found, with regression testing there’s sometimes room for interpretation on whether or not something is a bug. This contributes to the difficulty in employing regression testing effectively.

What is regression testing used for?

Regression testing really is a subset of functional testing, which verifies that the underlying functionality of the application meets a set of requirements that you’ve defined.

Let’s say you’re building an e-commerce store. Some of your requirements might be that a user can register, search for a product, and place an order if the user has entered a valid credit card. Now let’s say you’re just about to release a Wish List feature that lets your online shoppers save items for later. This sounds like an isolated feature that wouldn’t affect any other functionality, and so you may think that just testing the new behavior should be sufficient.

But let’s think about how this seemingly isolated feature could affect existing features.

Imagine that this Wish List happens to share the same look-and-feel as the Shopping Cart page, but with a few tweaks here and there. Let’s also imagine that to implement the code powering the Wish List, changes had to be made to the existing users subsystem and orders subsystem so that wish list items could be copied over to a user’s shopping cart.

If you knew about these dependencies, you’d probably also test the Shopping Cart to make sure that none of the changes to the shared UI resulted in a visual regression on the cart page. You’d also probably test pages that rely on the users and orders subsystems, like registering a new user or placing an order, to make sure that they’re still working too.

Regression testing techniques

So now that we have an idea of what regression testing is, how do we go about doing it?

Manual Testing

Manual testing is pretty self-explanatory; to run through a regression test, you or someone else goes through each step one-by-one and validates that the behavior you see matches the behavior you expect. This is time-consuming, but usually pretty effective as long as the people testing the application have a clear idea of what the expected behavior actually is.

Automated Testing

With automated testing, instead of having a person run through the steps manually, you’d instead automate the process of running through each of those tests. Historically this has been done via code-based tools, but Reflect provides a way to create and maintain these automated tests without writing code!

Full Regression Testing

When considering which tests to run, the most straightforward and comprehensive approach is to, you guessed it, run all your tests! This is also known as a “test all” strategy.

The advantage here is obvious, it gives you the most complete view of what’s going on your system. The drawback here is that it takes the most time compared to other approaches.

But there’s another important point to keep in mind here, and that’s that executing all of your tests is not the same as testing all of your existing behavior. Outside of formal verification processes that function as a mathematical proof, it’s basically impossible to know whether your regressions tests are testing all of your existing functionality or not.

We recommend running a full regression test on at least one internal environment (e.g. QA, Staging, UAT) before pushing changes to production.

Partial Regression Testing

With partial regression testing, you’re only executing a subset of your regression tests for a given change. The benefit here is that your tests take less time to execute, but the trade-off is that you’re knowingly leaving a subset of functionality untested.

Which begs the question, what functionality do you test, and what do you ignore?

If you have the infrastructure in place to support it, we recommend that you execute a partial regression test on every Pull Request. Another application of partial regression testing to consider is running a small subset of tests on your production environment directly. This is known as synthetic testing, which straddles the line between testing and monitoring. Reflect has a built-in scheduler that you can use to run your tests, and with Reflect you can use overrides to run a test in a different environment instead of maintaining separate tests for each environment.

Regression test selection

As we’ve established, knowing what to test is a difficult problem. If you’re looking to implement partial regression testing, how would you choose which subset of regression tests to run on a given change?

One common strategy when it comes to test case prioritization is to test only the most important workflows to your business. In an e-commerce store, this would be key workflows like adding to cart, placing an order, viewing a product, searching for a product, and logging in. This approach is also easy to implement, because: a) most companies will have a pretty good idea of what workflows are the most used or are the most critical to their business, and b) because you just need to choose a single subset of test cases once and add and remove to it as required.

A more sophisticated approach to knowing what to test is to use a tool like Nx to determine what functionality can be affected on a given code change. You could then group your tests in Reflect to tags that coorespond to these areas of the code, and have them be automatically execute when a code change could affect that behavior.

How does it fit with other types of testing?

Depending on your requirements, regression testing may go hand-in-hand with other types of testing:

Localization testing

This involves modifying the application based on the locale and culture of the user. Often this involves translating text to the user’s native language, but things like timezone differences and currency rules are common factors.

A common scenario for regression testing in the context of localization is to ensure that changes in text copy are properly translated into the other supported languages.

Testing role-based access controls (RBAC)

A lot of applications support the notion of a user role, such as an ‘admin’ role or a ‘read-only’ role. Role-based access controls are the rules that determine what functionality a user with a given role can or cannot access.

Role-based access controls are a good candidate for regression testing since they tend to be pervasive throughout the application, and thus often have the potential to be affected by a given change.

Regression testing FAQs

Here are answers to some of the most common regression testing questions:

When should regression testing be performed?

Regression tests provide value, but at the cost of the time it takes to create and maintain the tests. With Reflect, regression tests become a lot easier to build and maintain, making the cost of running your regression tests a lot lower.

We recommend running a full regression test suite before deploying to production, and running a partial regression test suite before merging changes into the mainline branch that serves as a smoke test for each Pull Request.

This gives you regression test coverage for your most important workflows at the earliest possible point, and gives you comprehensive coverage before you release new code to end-users.

What is the difference between retesting and regression testing?

Retesting is the type of test performed to verify that test cases that failed a previous execution has been fixed. Regression testing verifies that code changes or bug fixes have not affected previously functional parts of the software. While retesting is directed towards specific bug fixes, regression testing addresses the possible bugs that may come up from the bug fix.

How do you automate regression testing?

The repetitive nature of regression testing makes automation a better option than manual testing in most cases. Before you can automate regression testing, you need to select your test cases based on predefined criteria and choose a regression testing tool that will serve as your environment. Because of its no-code approach, tools like Reflect make regression testing automation easier to implement.

What is an example of load testing?

Load testing tests the resilience of the infrastructure used for hosting an application or software. During this test, the infrastructure is subjected to its peak user load to know how it will behave if multiple users hit it. A common example of load testing is executing a high volume of requests on a web server to simulate production traffic.

Is regression testing part of user acceptance testing (UAT)?

Technically, no. Although, the environment that you’re running tests may be confusingly named “UAT”. User acceptance testing just means that you’re testing that functionality matches the user’s requirements. That could be an internal user, an external user, or a new type of user who hasn’t even used the application before. It differs from regression testing, which is done during the software development phase or whenever there’s a modification or update to the program.

Conclusion

With the right strategy and scheduling, teams can save a lot of time and money on regression testing and release updates faster. However, it’s vital to understand that automation is at the heart of any effective regression strategy. Once you can identify the best testing tool for your program, your job is more than halfway done.

Start automating your regression testing

Automating the regression process can be a great way to improve the performance of your apps and prevent future bugs.

You can get started using Reflect today. Sign up for a free account and create your first test in minutes.

Copyright © 2021 Reflect Software Inc. All Rights Reserved.