Choosing the scenarios for a load test
What to test?
Occasionally, we encounter customers who have only a single scenario to test. For example, one client developed an application in which the only scenario of interest involved a user registering him/herself with the system and scheduling an appointment. When this happens, the tester may devote their efforts to the accurate simulation, testing and analysis of this single scenario. You are unlikely to be so lucky.
In any moderately complex system, there are dozens or even hundreds of scenarios that are candidates for load testing. You are not likely to have the time or resources to test them all. As a result, you must make some tough decisions about which scenarios will be included in the test plans.
There are a number of key factors that will raise or lower the importance of each scenario:
- Criticality – How critical is the scenario to the overall operation of the application?
- Frequency – How often is the scenario expected to be performed?
- Difficulty – How difficult is the scenario to simulate?
- Verifiability – Can correct operation of the scenario be easily verified?
The first two factors, criticality and frequency, should be the keys in deciding IF the scenario is tested. Scenarios that are both critical and frequent must be load tested. Those that are neither will typically be left out of the testing process entirely. The remaining scenarios, those that are somewhat critical or somewhat frequent or a little of both should be tested as the schedule allows.
We use the second two, difficulty and verifiability, to determine WHEN the scenario will be tested. This is particularly true at the beginning of the testing effort when the tester is not yet experienced with the application, tools and/or testing process. The first scenarios to be tested should be those that are easy to simulate and easy to verify. Unfortunately, at this point the inexperienced tester is not likely to know which scenarios will be difficult to simulate. You will have to substitute complexity as a measure of simulation difficulty until that knowledge is gained. As an example, in a CRM system, a scenario that consists of logging into the system and sending an e-mail is both relatively simple and easy to verify.
You may be asking “Why does the ease of testing or verification matter? Shouldn’t we always be testing the most critical and frequent testcases first?” The answer is a resounding “No!” If we lived in a world with no schedules, I might be inclined to agree with you. But this is the real world. Besides the realities of scheduling, there are two other important facts to take into account. First, bugs cost less to fix earlier in the development/testing cycle. Second, a large percentage of performance problems are related to system configuration, framework limitations or other system-wide factors. In these cases, exercising nearly any scenario will uncover the problem.
To put that in plain english: testing 5 less-critical or less-frequent scenarios this week is better than testing 2 critical and frequent scenarios next month. If your organization does not have significant experience with load testing, this effect is amplified, since the testing schedule will be further extended as a result of the learning curve.
Our Preferred Procedure
Summarizing from above, we generally follow the procedure:
1. Pick one or two simple, easily verifiable scenarios and test them. We use this phase to ensure that the testing infrastructure is in place and operating properly.
2. Add scenarios that are easy to simulate and either frequent or critical to the test suite.
3. Add the most critical and most frequent scenarios, regardless of difficulty, to the test suite.
I hope this helps you plan your load testing more effectively. In the next post, I’ll share some thoughts on preparing a practical load testing environment.
Test early, test often!
Chris, Chief Engineer at Web Performance
When his dad brought home a Commodore PET computer, Chris was drawn into computers. 7 years later, after finishing his degree in Computer and Electrical Engineering at Purdue University, he found himself writing software for industrial control systems. His first foray into testing software resulted in an innovative control system for testing lubricants in automotive engines. The Internet grabbed his attention and he became one of the first Sun Certified Java Developers. His focus then locked on performance testing of websites. As Chief Engineer for Web Performance since 2001, Chris now spends his time turning real-world testing challenges into new features for the Load Tester product.
1 Comment
30 August 2011 Scenarios for Load Testing
[…] Author: Christopher Merrill, Web Performance Originally published in September 2010 on Web Performance Blog […]