Impact of Zend Optimizer on PHP Performance
Load Testing SugarCRM and the Zend Optimizer
Christopher L Merrill
©2007 Web Performance, Inc.
v1.1
Summary
The performance of our reference PHP application under load (a default SugarCRM installation) showed no measurable improvement after installing the Zend Optimizer.Overview
This article measures the performance impact of the Zend Optimizer on a real-world processor-bound PHP application (SugarCRM) under load. Our measure of performance is user capacity. We define that as the number of simultaneous users that the system can support while meeting the specified performance criteria. The performance critera for this test require that all pages load within 6 seconds and no errors are encountered in the application.The Zend Optimizer FAQ answers the question "Why use the Zend Optimizer?" with this statement: "The standard Zend run-time compiler used by PHP is indeed extremely fast, generating code that is usually 2 to 10 times faster. But an application that uses the Zend Optimizer typically executes another 40% to 100% faster."
We test this assertion using SugarCRM as our reference application. According to the SugarCRM website, SugarCRM is "the market-leading comercial open source CRM application". SugarCRM is written in PHP. Based on previous testing, SugarCRM becomes quickly processor-bound on a typical installation. Measurement showed that the majority of the CPU load (~75%) was consumed in the Apache httpd processes, which makes it a good choice for testing the impact of a PHP optimizer.
After our initial results were analyzed, we contacted Zend for assistance in verifying that our configuration was correct. They suggested that we explicitly set the values for some Optimizer settings in the php.ini file:
zend_optimizer.enable_loader=0 zend_optimizer.disable_licensing=1 zend_optimizer.obfuscation_level_support=0After making these changes we retested and found no change in performance.
We have invited Zend to suggest a readily-available application that can demonstrate the claimed performance increase but, at the time of this writing, had not received a response.
Caveats
This report is concerned with the performance of the system under load. It does not make any claims about the performance of an idle system. The measurements may be quite different when a single user accesses the system.The purpose of the report is not to estimate the capacity of a SugarCRM system or make any predictions about expected performance. Every SugarCRM deployment will be different. The purpose is to measure the performance before and after installing the Zend Optimizer to determine performance benefits of the optimizer on a real-world processor-bound PHP application.
Methodology
The analysis is based on the results of a series of load tests performed in our test lab.To skip the methodology, you can jump straight to the analysis.
Software
The Zend Optimizer version 3.3.0a was installed using the procedure on the Zend website (ZendOptimizer-3.3.0a-linux-glibc21-i386.tar.gz)Both tests were run using PHP 5.1.6.
The server is running version 4.5.1e of SugarCRM installed via the standard installer (SugarOS-4.5.1e.zip).
The output from phpinfo() was captured for the baseline and optimized test configurations.
The load-testing software is Web Performance Load Tester 3.5.5447. The Web Performance Server Agent (same version) was installed on the server.
Hardware and OS
The SugarCRM server is running on a Dell Poweredge SC1420 with 2 x 2.8 GHz Xeon processors and 2GB of RAM. It has an 800MHz bus and 1MB of L2 Cache. All factory BIOS settings were restored prior to testing. This includes the hyperthreading support which is enabled. The floppy drive support in the BIOS disabled (no floppy drive is installed). The server contains a single Western Digital 80G (WD800JD) SATA disk drive and uses the on-board 1G network adapter. The operating system is the standard server install of CentOS 5.The load-generating engines (3) are each running on a Dell Poweredge 300 with 2 x 800MHz Pentium III processors with 1G of RAM. Each engine is running the Web Performance Load Engine 3.5 Boot Disk version 3.5.5275. For these tests, one load engine would have been sufficient.
The engines and server are networked via a Dell 2324 Powerconnect switch. The server is connected to a 1Gb port and the load engines are each connected to 100Mb ports.
The load test controller (the Load Tester GUI) is running on Windows XP SP2 on an IBM Thinkpad Z61m. It is connected via a second switch on a 100Mb port. This has no impact on the results, since the controller bandwidth requirements are small, but is mentioned here for completeness.
Testcases
The testcases selected for this load test mirror those selected for the SugarCRM Load Testing tutorial.Please note that the user mix, think times etc. may or may not reflect your SugarCRM implementation. The rated user capacity of the system is not likely to reflect any specific real-world deployment. To a large extent, these choices are irrelevant to this test, since the goal is not to evaluate the performance of SugarCRM, but rather determine the impact of the optimizer on before-and-after results.
The test data respository file is available for examination - the demo version of Load Tester can view the testcases, load configurations and detailed raw metrics. It cannot view the test reports, which is why they are linked from this page (see references).
Load Configuration
The Load Configuration was created with these parameters:- Four testcases, as described above
- VUs (virtual users) are distributed among the testcases as descibed in the SugarCRM Load Testing tutorial.
- VUs use the recorded think times, which are typically ~2-6 seconds per page. Think times are larger for some pages pages, such as adding a contact or a note, or reading a note.
- Each VU is simulating a 5Mbps network connection (e.g. cable/DSL connection)
- The test starts with 20 users and adds 1 user every minute. VUs start at random intervals within each minute.
- 10 second sample period
Database Management
The database was restored to an identical state prior to each test using the procedures outlined in the SugarCRM Load Testing tutorial.Test Procedure
Each test run followed these steps:- Restore database to pre-configured state with 100 users and 250 accounts, 1000 contacts and 3000 notes.
- Restart the server
- Start the server monitoring agent
- Run warm-up test (5 users running the View Note testcase for 3 minutes)
- Run the load test
Analysis
User Capacity
Our analysis shows the capacity of the system in baseline and optimized configurations to be 30. This analysis, along with a variety of charts is produced automatically in Load Tester's Load Test Report. Consult the User Capacity section of the test reports for more details on this analysis (see references). The charts shown below are extracted from that report section. The red line on the chart depicts the maximum page duration at each user level measured during the test. The location where that line crosses the duration threshold indicates the point where performance no longer met the criteria. Had any errors occurred during the test, the yellow line would depict the error rate and indicate when it crossed the error threshold. Note that the optimizer test encountered some page durations that were higher than the baseline test, resulting in a different scale on the two charts.Baseline User Capacity analysis
Zend Optimizer User Capacity analysis
Page Duration
To confirm the findings from the User Capacity analysis, we next look to the Page Duration charts. These show the minimum, maximum and average page durations during the test. These charts are from the Summary section of the load test reports. Note that the optimizer test encountered some page durations that were higher than the baseline test, resulting in a different scale on the two charts.Note that capacity limit (30 users, which occurred at ~12 minutes), both systems showed similar page durations - maximums of 5-6 seconds and averages ~2 seconds. As the system was pushed past the rated capacity, you can see that the optimized system actually showed higher maximum and average page durations. This was consistent across 3 the test runs performed on each configuration.
Baseline page durations
Zend Optimizer page durations
Server processor utilization
As expected, the limiting factor in the application performance is processor utilization. The charts below (extracted from the Servers section of the load test reports) show several processor-related measurements that were collected during the test using the Advanced Server Analysis module. The light-blue plot depicts the CPU utilization (%). In both tests, the CPU is nearing 100% utilization when the page duration threholds fail.Baseline processor measurements
Zend Optimizer processor measurements
Conclusion
Under the conditions tested, the reference PHP application (SugarCRM) showed no measureable performance gain from the use of the Zend Optimizer.Reference Materials
- phpinfo() output for baseline test
- complete load test report for baseline test
- phpinfo() output for Zend Optimizer
- complete load test report for the Zend Optimizer test
- Load Tester repository file (to be used with Web Performance Load Tester for further data analysis)
Feedback & Comments
Comments about this report may be posted at the company blog post.
Version History
v1.0 - 1st public release (5 Nov 07)
v1.1 - email cleanup (23 Jan 09)