The General Idea, Pros and Cons
Testing in production refers to the execution of software tests in a live environment. It is a valuable approach to software testing because it mirrors the real world. A test run in a non-production environment may yield misleading results due to varying configurations, controlled conditions and the absence of other realistic factors such as real data and traffic.
The goal of pre-production testing is to rid the software of problems that may occur in the presence of certain events and conditions. Although test environments are built to be similar to the production environment, there are limits to what can be duplicated and the endeavor can be expensive to follow through.
Perfectly replicating a production environment requires satisfying server requirements, using the same databases, installing the same dependencies, and simulating user traffic etc. Testing in production takes away these associated costs (hardware, licensing, personnel) because the test code lives in the same environment as the production code serving user traffic.
As companies develop their software at a faster pace, there becomes a need for early feedback to prevent or mitigate full-blown disasters in the future. However, this wonderland of running tests against real user behavior comes with several pitfalls.
Some Challenges of Testing in Production
Testing in a production environment is scary for many developer teams, and this is not an irrational fear. Here are some of its cons:
Breaking production
Just like regular tests, a test executed in a production environment will either succeed or fail. A test that fails or doesn’t go according to plan can impact the production environment in unanticipated ways, including breaking production and causing downtime.
Difficulty in handling security measures
Captcha is a type of challenge-response system that can differentiate whether or not the user is human. Removing or disabling the Captcha on web forms for the sake of testing in production may give room to email flooding. This is just one example of the numerous security challenges that may be encountered when testing in a production environment.
Processing payment
when testing a service that allows the purchase of various items, it can be tricky to implement a test that requires the response of a business-critical operation such as processing real debit cards.
Cleaning up mock data
Non-development environments run tests against mock data. It is discouraged to use such data during production testing because it will impact the integrity of production data, and its removal can be a hassle.
Decline in system performance
When a high demand test, such as load testing, is run in a production environment, real users may experience a decline in performance, and this can damage the reputation of the application or organization.
Test window may be narrowed impractically
A test that must run over an extended period, such as soak testing, will be interrupted, and its progress will be distorted by system maintenance activities like defragmentation and backups. These interruptions can lead to false failing and flaky tests.
Integrity of production data may be impacted
It is harmless to generate thousands of records when running non-production tests. A tester may create several accounts and record numerous purchases to test a hypothetical condition. It is an entirely different story with production testing, the commingling of mock data with production data can lead to the overall corruption of production data. Hence, testing in production can pose a business risk.
For safety reasons, less testing tools can be used in such environment
For security and performance reasons, production testing may prevent developers from activating their favorite third-party tools, and this can result in a more difficult process of diagnosing problems and reaching desired results.
Loss of valuable data
In a situation where junk data commingles with production data during a production test, it can be difficult to remove the junk data afterward. An attempt to automate this removal using tools and scripts can be risky and lead to the loss of valuable production data.
A Different Approach
Applications with complex infrastructures can only run at full capacity in a production environment. Any tests executed in a scaled-down environment (with contrived conditions) will produce misleading results. There are tools that can capture real-time scenarios in a production environment and replay them as API tests in a test (QA or pre-production) environment.
Using such tools and services combines the best of both worlds in the following ways:
Puts the software through the variables of a production environment.
Allows for the testing of the entire end-to-end technology stack.
Create tests from real user sessions.
Doesn’t include the risks and drawbacks of testing in production.
Conclusion
As teams push new code to an existing software application, it is essential to test that the application’s functionalities are (still) working correctly. Non-production tests are great for catching bugs, but some bugs may make it to production due to certain configuration and resource disparities in both environments.
Production testing provides a realistic view of how things work when end-users interact with a software product. In this article, we look at some benefits of production testing, its challenges, and we present an alternative approach towards achieving desired results but minimizing disadvantages.