Load Testing Basics: How Many Concurrent Users is Enough?

• 4 Comments
• 3 February 2011
• Posted in Load Testing

The most common question regarding a website’s performance is not how fast the website is or how it scales, but something more fundamental:

What should the performance goal be in terms of concurrent users?

Should it be one hundred concurrent users? A thousand? Ten thousand? Does it require one server or a hundred to handle the load?

There’s a good reason for how many times this question comes up: it’s tricky and there’s no definitive answer. If it is a new website it’s anyone’s guess what the demand will be. Even if it is an existing website, web analytics software typically give marketing reports on how many customers were acquired per month or per week, and even a per-day measurement but doesn’t give the numbers required. The reason is the difference between average and instantaneous peak. It doesn’t matter if only 5% of a site’s capacity is used all year if the site was unusable for a critical period. The only thing that matters in terms of protecting your site against crashes and slowdowns is instantaneous peak.

So while marketing speaks in terms of transactions a day or even unique customers on the site, the only measurement useful for capacity planning is how many people were on the website at the same time, the instantaneous peak. By “on the website”, I mean looking at their browsers at a page on the website, regardless of whether that customer is reading or clicking. It’s tempting to use hits/sec or transactions/sec, which are fine metrics, but what happens when you redesign the site for higher performance and the number of hits/sec or transactions/sec drops as a result? The capacity of the site may have gone up, but the number of hits/sec would drop. In the end you need a metric that is implementation independent.

The question of figuring out the maximum number of concurrent users is made even more difficult because a surprising number of people tasked with speccing a website don’t understand the business, and that is critical to increasing the accuracy of the estimate. In the end, it is still a guessing game, not only because of the unknowns, but because it involves the future. Even if you have an existing website with accurate web logs, it’s impossible to tell the future and guarantee that there won’t be a huge traffic spike someday.

So, it is impossible to be really accurate and we’re reduced to being close enough. What are the penalties for getting it wrong? Guess too low, and a site may experience a slowdown or crash during a critical period. Will that merely annoy your customers, or will the company lose money? If it’s the later, then it pays to reduce the chances of a crash happening. But what happens if you over spec a site and there’s too much excess capacity? The downside is added cost and complexity, but that’s the price that needs to be paid for reducing the risk; in this way, building in excess load capacity is like insurance.

Let’s take a look at what the typical systems engineer has in terms of information. If the site exists at least there’s some data, although it almost never has the information that’s actually needed. The information that comes from the site tends to be marketing statistics. That’s nice, but how in the world can you possibly figure out the simultaneous peak user load from that?

Obviously the more information you have about a business the more accurate the guesses, but let’s start with a common situation: the person setting up the website has no idea about the business and no way of getting any information. It sounds strange, but that happens more often that you would believe.

For example, let’s say marketing reports that the site either has or expects to have an average of 150,000 visitors per month. First calculation is easy: figure out the number of visitors per day. If the site is used equally 7 days a week, then divide by 30. If, however, the site is used mostly during the week you’d divide by the number of non-weekend days in a month, around 21.7.

For the sake of simplicity let’s assume the site is used equally throughout the week for a per-day visitor number of 5,000. The next step is to figure out how many hours of the day the site is most active, which will differ from business to business. If you actually know the nature of the business and can identify the peak hours, then by all means, use those, but if you don’t, then figure on using 12 hours as a rule of thumb.

Average visits per hour = (5000 average visitors/day) / 12 hours = 417

To figure out the average concurrent users you need to know the average length a user stays on the site. This can be obtained either from the web metrics, or, if that isn’t available, using a stopwatch and going through a couple of user scenarios yourself on the site, reading all of the material and filling out forms just like a typical user.

Once you know the average visit length, you can then calculate average concurrent users. The key to understanding this calculation is the concept of levels of concurrency. If a user spends on average 10 minutes on a site, then one level of concurrency, i.e. one user simulator could simulate 6 visits in an hour, 60 minutes divided by ten minutes per visit.

Average Concurrent Users = Visits per hour / (60 min/hour / average visit)

Using the 10 minute figure above, with an hourly visit rate of 417 gives an average concurrent user level of just 69, a far cry from the 150,000 per month figure we started with. The math isn’t hard, but it’s faster to type into the online form we use every day at Web Performance, Inc..

Check out this great easy-to-use online calculator:
https://www.webperformance.com/library/tutorials/CalculateNumberOfLoadtestUsers/

Now we know the average number of users on the site at one time, which is useful, but is still short of the number we need to know: the instantaneous peak number of concurrent users. Based on absolutely no science whatsoever, in the load testing business we use a multiplier range from 3 to 6 depending on the accuracy of the average concurrent user number.

The more important it is to your business to not have a slowdown, the higher the multiple. In the above example that would put your “safe” number of concurrent users at 414, the target that would need to be verified by a load test.

As an engineer it bothers me having to guess at these numbers, but then again, guessing is part of engineering. As more websites switch to the automatically scalable architectures enabled by cloud-computing this type of guesswork will be less important. Until then, run these calculations on your own website and let me know how close or how far they come to reality.

Michael Czeiszperger
Founder, Web Performance, Inc.

Michael Czeiszperger

Founder of Web Performance, Inc
B.S. Electrical & Computer Engineering
The Ohio State University
LinkedIn Profile

Resources

• Web Performance Center
• Reports
• Case Studies
• Tutorials
• Calculators
• Books
• Links
• Dictionary
• Research