The French Social Security Administration
Switches To Web
Performance Load Tester
From Open Source 
Customer Situation
The Caisse Nationale d'Asurance Vieillesse (CNAV) is the French-equivalent of the Social Security Administration in the United States, responsible for managing millions of pensions. After becoming disillusioned with the length of time it took to do a performance evaluation using the open source program OpenSTA, they evaluated the top seven commercial load testing solutions and chose Web Performance Load Tester because its "no scripting" approach cut their testing cycle to 1/4 to 1/5 of its previous time.
CNAV uses their public website to enable both employers and citizens to file forms and manage their pensions, so the web servers receive a very high number of users, especially when tax forms are due. CNAV started systematic performance testing early in their move to web-based systems according to Project Manager Jean-Luc Orts, who told us "Not running performance and capacity tests could result in very expensive emergencies. There are only a few days for the employers to file returns without encountering issues."
At first they used consultants to do the testing, but Orts found that "though efficient, the cost and the inflexibility quickly encouraged us to become self-sufficient in the area of load testing." They next tried OpenSTA, an open source load testing package for Windows, but found "...the product remains very basic. It can do plenty of things, but only with a programming language, which has an expensive development cost. For example, it was particularly tricky to write scripts to correctly handle the Oracle SSO (Single Sign-On) which is used on our portal."
There are many commercial load testing packages on the market; CNAV tried the top seven, and eventually settled on Web Performance Load Tester. "Compared to the companies we evaluated (far from being exhaustive with seven candidate companies) Web Performance Load Tester stood out with the best quality/price ratio. The differences are obvious in terms or productivity, design, and presentation of the results. In using Web Performance, the development time for test scripts has been cut to 1/4 to 1/5 of how long it used to take."
But that wasn't the only reason. "I also want to underline that its important to have careful and reactive support. Its an assessment that I have made since the beginning and which confirms my choice. With Kapitec [the representative of Web Performance in France] and Web Performance I notice a responsiveness that allows me to quickly progress with answers in less than one day, sometimes even half a day."
The Solution: Web Performance Load Tester
The Web Performance Load Tester™ is comprehensive and easy to use Web performance testing software, which includes an analysis module (Analyzer) and a load testing module (Load Tester). It is the only performance testing software capable of handling most popular web-design technologies such as AJAX, Web forms, J2EE, ASP, .NET/C#, PHP, Ruby, ColdFusion, Java, etc, without having to write complex code in a programming language.
With a programming-based testing system, a programmer has to reverse-engineer and write the test case in a programming language, which is not only difficult, but tedious and repetitive as well. Web Performance instead detects and automatically configures the test cases for most situations. The result is anyone with a knowledge of the business requirements for a website can record and playback test cases in a fraction of the time it would take with a traditional product.
The testing procedure is straightforward: a browser is used to access the website in the same way as a customer. Web Performance Load Tester™ analyzes all the HTTP/S transactions, and automatically configures the difficult and complicated state and session variables, as well as page validation. The built-in wizards make it easy to complete the configuration for such things as usernames and passwords and other data for the simulation. For example, for a test case for performing an online purchase, each virtual user could login with a separate username and password purchase a different item.
Between the ability to watch the test cases play back in a browser and the automatic page validation there is no question as to wether the test is working correctly. While the product is designed for a non-programmer to use, that doesn’t mean it lacks features. All of the hard-to-find load testing features are included such as support for client certificates, IP spoofing, dynamic IP addresses, AJAX and XML support, etc.
The most important thing in performance testing is interpreting and presenting the results, and this is where Web Performance Load Tester™ excels. The reports are designed to translate raw numbers and find the answers important business questions such as “How many users can your website handle?”, “Which web pages are slow”, as well as pinpoint the location of bottlenecks.
Interview - Jean-Luc ORTS - CNAV (Translated from French)
About six months after the implementation of the Web Load Performance Tester™ tool within the CNAV, we interviewed Mr. Jean-Luc ORTS, the Project Manager in charge of the load testing tool and the follow-up of the Internet applications hosted by the Computing Site in Tours.
Why the load testing is essential for the CNAV? And which kind of applications do you test?
J.L. ORTS - The load testing is now part of our web applications lifecycle process. Today it is an almost compulsory step, which is decided by the Web production team. All the web applications that we host (static and dynamic web, updates, Internet…) are tested, including when we make technical and/or architecture improvements. The purpose of these tests is to make a smooth transition into production and to make sure we can absorb the inherent load peaks in our applications (there are only a few days for the employers to file returns, and the effects of new announcements relating to the legislation changes) without encountering issues. These tests fit into a total quality control of the CNAV.
Not running performance and capacity tests could result in very expensive emergencies such as needing to purchase servers in a hurry, whereas tuning the existing architecture is very often sufficient. The default settings are often far away from a computer’s real capacity.
Why have you chosen to acquire a load testing tool instead of calling upon a specialized consulting company?
J.L. ORTS - We engaged a consulting services company to perform a load testing campaign about three or four years ago. Though efficient, the cost and the inflexibility quickly encouraged us to become self-sufficient in the area of load testing.
How and by who are used the load testing results? And for which purposes?
J.L. ORTS - The reports are delivered to the service manager and developers, etc., but they are primarily used by the web production team, which uses the results to make broad architecture improvements. The final result is then used for comparison purposes from one test campaign to another.
You used to use OpenSTA to perform your load tests, then why have you wanted to stop using it and to use a commercial software tool?
J.L. ORTS - We used OpenSTA for about 3 years. Several reasons led us to reconsider our choice and to migrate towards a commercial solution. First of all the OpenSTA community looked like it wasn’t progressing: no new version, no bug fixes, and a stagnant forum. And the product remains very basic. It can do plenty of things, but only by using the programming language, and for an expensive “development” cost, mainly for the scripting including multiple pages, which is the situation of our sites. For example, the Oracle SSO handling which is now our basis for the portals is possible, but it remains particularly tricky. Moreover, no support assistance is offered.
How do you position Web Performance Load Tester versus the competition in terms of technical features and pricing?
J.L. ORTS - Compared to the companies which we evaluated (far from being exhaustive with seven candidate companies) Web Performance Load Tester™ stood out with the best quality/price ratio.
How is Web Performance Load Tester™ integrated in your development and testing methodology?
J.L. ORTS - The load tests with Web Performance are performed during the integration step, starting with doing comparisons of technical settings for performance tuning before the development step.
Is Web Performance of a good quality-price ratio?
J.L. ORTS - Yes, because the load tests performed with Web Performance Load Tester™ during the integration step allow us to make performance improvements that otherwise would be left to make in hurry during production peaks.
How do you compare Web Performance Load Tester™ versus the other solutions you have evaluated?
J.L. ORTS - The only serious comparison that I can make, it is with OpenSTA. In terms of rough technical results the two tools are about at the same level, which is normal since they have the same purpose. But the differences are obvious in terms of productivity, design, and presentation of the results. In using Web Performance the design time is split by 4 or 5, even more if they are complex scenarios. To create the uncommonly professional reports we again split the time required by 4 or 5.
Is the scripting language absence in Web performance Load Tester™ a problem?
J.L. ORTS - At a pinch… if we should find a workaround for a feature not developed yet or for a bug.
Does the SOAP feature, as it has been implemented in the version 3.1 upon the CNAV request, answers your requirements?
J.L. ORTS - I have read in the manual that you could replace a field of a HTTP request by the content of a file field which would allow me to perform what I need for the SOAP applications I should validate. But I haven’t experiment this feature yet, as our 2006 applications aren’t working anymore and the 2007 applications are still being developed.
In conclusion what do you think of Web Performance Load Tester™, and which improvements would you think which could be useful?
J.L. ORTS - I am very happy with Web Performance Load Tester™, and even more with the 3.0 version which is more intuitive/easy-to-use than the previous version.
I also want to underline that it’s important to have careful and reactive support. It’s an assessment that I have made since the beginning and which confirms my choice. With Kapitec Software and Web Performance, I notice a responsiveness that allows me to quickly progress with answers in less than one day, even in a half day.
Among the wished improvements, and this list isn’t exhaustive, I think that it could be useful to free itself from the automatic load balancing, i.e. to have the ability to set the number of virtual users assigned to each remote engine. Another improvement would be to have an option to drilldown to the URL level instead of the web page when collecting statistics (and then for the reports). For instance OpenSTA does it automatically, but as a result with very large files which are very hard to handle as soon if the test is a little long.
Web Performance Load Tester is a registered trademark of Web Performance, Inc. All the other trademarks are trademarks or registered trademarks of their respective owners. All rights reserved - February 2007