Impact of APC Module on PHP Performance
Load Testing SugarCRM and the APC Module
Christopher L Merrill
©2007 Web Performance, Inc.
v1.1
Summary
The performance of our reference PHP application under load (a default SugarCRM installation) showed a 180% increase, measured by total system capacity, after installation of the APC module.Overview
This article measures the performance impact of the APC module (a PHP code cache) on a real-world processor-bound PHP application (SugarCRM) under load.While researching our previous PHP Optimization report, we were referred to the APC module for PHP. On the module web site, it is described this way: "APC is a free, open, and robust framework for caching and optimizing PHP intermediate code."
We tested the impact of the APC module using SugarCRM as our reference application. According to the SugarCRM web site, SugarCRM is "the market-leading commercial open source CRM application". SugarCRM is written in PHP. Based on previous testing, SugarCRM becomes quickly processor-bound on a typical installation. Measurement showed that the majority of the CPU load (~75%) was consumed in the Apache httpd processes, which makes it a perfect choice for testing the impact of a PHP cache/optimizer.
Our primary measure of performance is user capacity. We define that as the number of simultaneous users that the system can support while meeting the specified performance criteria. The performance criteria for this test require that all pages load within 6 seconds and no errors are encountered in the application.
Caveats
This report is concerned with the performance of the system under load. It does not make any statements about the performance of an idle system. The measurements may be quite different when a single user accesses the system.The purpose of the report is not to estimate the capacity of a SugarCRM system or make any predictions about expected performance. Each SugarCRM deployment will be different.
Methodology
The analysis is based on the results of a series of load tests performed in our test lab.To skip the methodology, you can jump straight to the analysis.
Software
For the optimized test, the APC module version 3.0.14 was installed vi PECL. The default options were used. The only change to the php.ini file was the addition of "extension=apc.so" to activate the module.Both tests were run using PHP 5.1.6.
The server is running version 4.5.1e of SugarCRM installed via the standard installer (SugarOS-4.5.1e.zip).
The output from phpinfo() was captured for the baseline and optimized test configurations.
The load-testing software is Web Performance Load Tester 3.5.5447. The Web Performance Server Agent (same version) was installed on the server.
Hardware and OS
The SugarCRM server is running on a Dell Poweredge SC1420 with 2 x 2.8 GHz Xeon processors and 2GB of RAM. It has an 800MHz bus and 1MB of L2 Cache. All factory BIOS settings were restored prior to testing. This includes the hyperthreading support which is enabled. The floppy drive support in the BIOS disabled (no floppy drive is installed). The server contains a single Western Digital 80G (WD800JD) SATA disk drive and uses the on-board 1G network adapter. The operating system is the standard server install of CentOS 5 with unneeded services deactivated.The load-generating engines (3) are each running on a Dell Poweredge 300 with 2 x 800MHz Pentium III processors with 1G of RAM. Each engine is running the Web Performance Load Engine 3.5 Boot Disk version 3.5.5275. For these tests, one load engine would have been sufficient.
The engines and server are networked via a Dell 2324 Power connect switch. The server is connected to a 1Gb port and the load engines are each connected to 100Mb ports.
The load test controller (the Load Tester GUI) is running on Windows XP SP2 on an IBM Thinkpad Z61m. It is connected via a second switch on a 100Mb port. This has no impact on the results, since the controller bandwidth requirements are small, but is mentioned here for completeness.
Test cases
The test cases selected for this load test mirror those selected for the SugarCRM Load Testing tutorial.Please note that the user mix, think times, etc may or may not reflect your SugarCRM implementation, so the rated user capacity of the system is not likely to reflect any specific real-world deployment. To a large extent, these choices are irrelevant to this test, since the goal is not to evaluate the performance of SugarCRM, but rather determine the impact of the optimizer on before-and-after results.
The test data repository file is available for examination - the demo version of Load Tester can view the test cases, load configurations and detailed raw metrics. It cannot view the test reports, which is why they are linked from this page (see references).
Load Configuration
The Load Configuration was created with these parameters:- Four test cases, as described above
- VUs (virtual users) are distributed among the test cases as described in the SugarCRM Load Testing tutorial.
- VUs use the recorded think times, which are typically ~2-6 seconds per page. Think times are larger for some pages pages, such as adding a contact or a note, or reading a note.
- Each VU is simulating a 5Mbps network connection (e.g. cable/DSL connection)
- The baseline tests start with 20 users and add 1 user every minute. VUs start at random intervals within each minute. The optimized tests start with 70 users and add 1 user every minute.
- 10 second sample period
Database Management
The database was restored to an identical state prior to each test using the procedures outlined in the SugarCRM Load Testing tutorial.Test Procedure
Each test run followed these steps:- Restore database to pre-configured state with 100 users and 250 accounts, 1000 contacts and 3000 notes.
- Restart the server
- Start the server monitoring agent
- Run warm-up test (5 users running the View Note test case for 3 minutes)
- Run the load test
Analysis
User Capacity
Our analysis shows the capacity of the baseline system was 30 users. After installing the APC module, the capacity increased to 86 - an increase of 180% over the baseline capacity. This analysis, along with a variety of charts is produced automatically in Load Tester's Load Test Report. Consult the User Capacity section of the test reports for more details on this analysis (see references). The charts shown below are extracted from that report section. The red line on the chart depicts the maximum page duration at each user level measured during the test. The location where that line crosses the duration threshold indicates the point where performance no longer met the criteria. Had any errors occurred during the test, the yellow line would depict the error rate and indicate when it crossed the error threshold. Note that the baseline test encountered some page durations that were higher than the APC test, resulting in a different scale on the two charts.Baseline User Capacity analysis
APC User Capacity analysis
Page Duration
To confirm the findings from the User Capacity analysis, we next look to the Page Duration charts. These show the minimum, maximum and average page durations during the test. These charts are from the Summary section of the load test reports. Note that the baseline test encountered some page durations that significantly higher than the optimized test, resulting in a different scale on the two charts.Notice that on the baseline test, the average page durations had roughly doubled (from 1 second to 2) when the system had reached its maximum capacity. When the maximum capacity was reached running with APC, the average page durations had not degraded substantially.
Baseline page durations
APC page durations
Server processor utilization
As expected, the limiting factor in the application performance is processor utilization. The charts below (extracted from the Servers section of the load test reports) show several processor-related measurements that were collected during the test using the Advanced Server Analysis module. The light-blue plot depicts the CPU utilization (%). In both tests, the CPU is nearing 100% utilization when the page duration thresholds fail.Baseline processor measurements
APC processor measurements
Conclusion
Under the conditions tested, the reference PHP application (SugarCRM) showed a 180% increase in total user capacity when the APC module was used. In other words, the APC optimized system had 2.8 times the capacity of the baseline (unoptimized) system.Reference Materials
- phpinfo() output for baseline test
- complete load test report for baseline test
- phpinfo() output for APC configuration
- complete load test report for the APC test
- Load Tester repository file (to be used with Web Performance Load Tester for further data analysis)
Feedback & Comments
Comments about this report may be posted at the company blog post.
Version History
v1.0 - 1st public release (2 Jan 2008)
v1.1 - email cleanup (23 Jan 2009)