Beta Browser Shootout
Safari 4 Beta, IE 8 Beta, and Chrome Beta 1 are compared to Firefox 3.0.
Matt Drew
©2008 Web Performance Inc, March, 2009; v1.0
Introduction
The purpose of this test was to compare the page loading performance of the newest versions of three major browsers: Internet Explorer 8 (IE8), Chrome 1, and Safari 4. Firefox 3.0 was included as a baseline. When we conducted a similar real-world browser performance test in 2007 we looked at Safari 3 beta, Firefox 2, and Internet Explorer 7. In that test Safari 3 beta showed a significant advantage in page loading speed. Do Apple’s claims for Safari 4 uphold this tradition? We’ll see.
Which Web Pages?
Which web pages to test is the first question to answer. We chose fifteen of the top sixteen rated english-language sites from Alexa.com (as of February 3, 2009). This approach gave us a good cross-section of sites ranging from very light, small sites to sites loaded with javascript, images, and dynamic content.
We also wanted to test representative content for each site, as well as the sites’ front pages. For example, on search sites, we tested a sample search page using the search term “test”. For Facebook, we used its own Facebook page. We also used a test Ebay auction and sample articles from various sites. We had trouble getting consistent data from Blogger, so we used MySpace (#16) instead. The following is a list of the URLs we tested from March 3 through March 5, 2009:
- http://www.google.com/search?hl=en&q=test
- http://search.yahoo.com/search?p=test
- http://www.youtube.com/watch?v=iBBcibQByNs
- http://www.facebook.com/facebook
- http://search.live.com/results.aspx?q=test&go=&form=QBLH&qs=n
- http://articles.moneycentral.msn.com/taxes/home.aspx
- http://cgi.ebay.com/Test-Item-Please-do-not-bid-0749-2-19 05_W0QQitemZ6156056805QQcmdZViewItemQQptZLH_DefaultDomain_0?hash=item6156056805
- http://en.wikipedia.org/wiki/Tim_Duncan
- http://raleigh.craigslist.org
- http://www.aol.com
- http://www.amazon.com/Spore-Mac/dp/B000FKBCX4
- http://sports.espn.go.com/mlb/news/story?id=3886444
- http://www.cnn.com/2009/POLITICS/02/04/voter.anger/index.html
- http:///www.microsoft.com/security/default.mspx
- http://celebrity.myspace.com/index.cfm?fuseaction=celebrity.risingstar
Testing Considerations
Static vs. Real-World Testing
Static benchmark tests are useful tools for checking web browser performance. They measure browser performance in a highly controlled way with few variables. Unfortunately, when you or I sit down to browse the news or check out the latest Facebook postings, we will not have a fixed set of carefully structured data files. Real world testing gives a much better approximation of the day-to-day user’s experience. Site quirks and flaws are more likely to show up under these conditions.
Network Effects
Bandwidth and latency are key challenges in testing a browser’s performance directly. In our 2007 test, we established that test results were consistent between local network copies of sites and the public sites. Consequently, for this test, we did not repeat the local portion of the test and instead focused on improving the quality of the live data.
Advertising
Advertising is part of the web browsing experience, and we wanted to include it in our test. However, since ads varied widely in size, speed, and latency, we had to block traffic to most advertising sites to achieve consistent results. We emphatically make no claims about the superiority of this approach over any other - it has clear limitations and should be taken as only one of many data points in the overall browser performance picture.
Caching Effects
We knew that there would be a difference between loading a page for the first time and loading it again when the data had already been downloaded, so we wanted to test both. First we would load each page into the browser to trigger any server-side caching and ensure that the server was performing as fast as it reasonably could. We would then purge the cache using the tools available in each browser. Once the cache was purged, we would load the page again, and measure the total time it took from the beginning of the first HTTP request until the last element was downloaded to the browser. The cached page loads were done a few seconds after the uncached full page loads using the refresh button in each browser. Each browser was tested several times across different times and different days to help eliminate network effects. The scores shown in the graphs below represent the mean of the best data points. In some cases, we were unable to get five data points for uncached page loads in the correct sequence, and portions of the tests had to be completed at later times.
Background Processes
All of the browsers have background processes which could (and did) interfere with the test results. These are activities such as updating rss feeds, checking for updated phishing information, or pre-loading search results. Since these processes are part of the browsing experience, we chose to tolerate their effects on the testing process.
Test Setup
The operating system used was Windows XP Professional SP 3, patched to current as of February 3rd, 2009. Browsers tested were Firefox 3.0.6, Internet Explorer 8.0.6001.18372, Chrome 1.0.154.48 and Safari 4 public beta (528.16). All browser configurations were the default as installed on Windows XP, with the exception of the proxy configuration to point the browsers at our recording proxy. All tests were performed remotely via the Microsoft terminal services client.
The test platform was a Dell PowerEdge SC1420, with a single hyper-threaded 2.8Gz Xeon and 4GB of RAM (PAE enabled).
Measurement of web page performance was done using our own Web Performance Load Tester™ version 3.5.6673, which analyzes HTTP traffic. Our software uses a Java HTTP proxy server to record all HTTP traffic between the client browser and the site. The reason this measurement technique is reasonably accurate is that the browsers try to pre-render the pages as much as possible while simultaneously pipelining HTTP image requests on multiple threads. By the time the last byte of the last element arrives, the web page is entirely rendered except for that last element, for which the render time is negligible. The software groups HTTP traffic that happens after the initial load separately, so in general we avoid including later AJAX requests in the initial page load timings. One thing we had to watch out for were pages (such as the AOL main page) that included timers that automatically loaded new content.
Testing Issues
Ideally, we would like to show test results that were unbiased by unanticipated errors in the operation of the browsers. However, the real world is rarely ideal, and we did find a few bugs:
- Internet Explorer 8 had an odd result that we only noted near the end of the test. When clearing the cache for the second full page load, IE8 worked the same as the other browsers, with one exception - the microsoft.com site. For whatever reason, this site was still loading from cached data even though the cache had been cleared. We worked around this by opening the site , then closing and reopening the browser before starting recording. This meant that we had to re-record the IE8 data for microsoft.com later in the testing period than the rest of the IE8 samples.
- Chrome sometimes would not load the first URL that was cut and pasted into its address bar.
- Cutting and pasting URLs into the Safari 4 address bar would sometimes load the previous URL instead.
Test Data
The three new browsers had uncached load time averages of approximately two seconds; Firefox 3 was slightly slower at just under three seconds. Cached load times showed slightly more variation but were still within half a second of each other.
Conclusions
All three of the newer browsers were so close that there was no clear winner. Internet Explorer 8 had a small advantage, but not one that would exceed the margin of error for the test.
All three of the browsers tested were significantly faster at retrieving uncached data than the baseline Firefox browser. They were roughly equivalent in retrieving cached data.
Safari did not maintain its advantage, despite Apple’s claims that Safari 4 loads pages almost three times faster than Firefox 3. Their claims are more than likely valid in static benchmark tests, but in real-world applications, Safari 4 has less than one second advantage over Firefox 3.0.6 on uncached live page loads, and none at all over browsers of its own generation.
Version
History
v1.0 - 1st public release (March 18, 2009)