Design a Load Test that Measures What You Really Want to Know
When the first few testcases are ready, the next step is to put them together into a load test. How they are combined and executed will have a big impact on the accuracy of the test.
Testcase ratios – Make sure that each testcase is exercised in the correct ratio relative to the other testcases. For example, it is likely that many more people will be searching for products than buying them – the load configuration should reflect this ratio.
Page ratios – Re-visit the hit rate that will be exerted on each page based on the ratio of testcases and the frequency of each page in the testcase. Be sure that this is still close to the expected usage.
Database size – Make sure you have an sufficient quantity of data in your system to reflect real-world usage. If your system is expected to have 100,000 rows in the “products” table and your test system only has 100, then the results will be questionable.
Number of users – There can be much confusion about the number of total users on a system and the number of simultaneous users, especially among the less-technical team members. Be sure that you are testing realistic load levels. A system with 100,000 registered users is likely to only have a fraction of that (typically less than 10,000) using the site at any one time. Testing beyond the expected load can be beneficial – just make sure everyone understands it is happening.
Load ramping – If a system is expected to support 1000 users during peak times, then a test that goes from zero to one thousand users in 30 seconds and then maintains that level for the entire duration of the test is most likely not measuring a realistic scenario. Most systems should ramp gradually to the peak load. There are exceptions, of course, but we frequently find that testers over-estimate the need for testing traffic spikes. In addition to being more accurate, a stepped ramp test configuration can generate much more useful data as well.
3rd-Party Systems – If your system includes resources from 3rd parties, carefully consider the impact of including them in the test. In some cases, these systems are crucial and must be included. But in many cases they are not – if not, leave them out. If these systems gather data, such as click-tracking systems, make sure the data from the testing period is purged or otherwise ignored by those who will analyze that data. If these systems have a financial impact, such as 3rd-party ad providers, make sure that you will not be incurring charges for your company (or worse – a third-party who is not directly associated with your organization). Be sure the 3rd-parties have granted permission to test their system and are aware of the testing schedule.
Simulate the available client-side bandwidth accurately – How will your users be accessing your site? From their home via cable modem? From a clogged corporate T-3 line? The better load testing tools make it easy to simulate different client-side bandwidth profiles – be sure you are using it!
The common theme between all the above topics is that when they are done wrong, they can lead a serious misinterpretation of the test data. Make sure that what you are measuring is what you really want to know!
Chris Merrill, Chief Engineer
When his dad brought home a Commodore PET computer, Chris was drawn into computers. 7 years later, after finishing his degree in Computer and Electrical Engineering at Purdue University, he found himself writing software for industrial control systems. His first foray into testing software resulted in an innovative control system for testing lubricants in automotive engines. The Internet grabbed his attention and he became one of the first Sun Certified Java Developers. His focus then locked on performance testing of websites. As Chief Engineer for Web Performance since 2001, Chris now spends his time turning real-world testing challenges into new features for the Load Tester product.