How HTTP Authentication works and why load testers should care
The most commonly used authentication method for websites is a login form on a web page. We’ve all seen them – enter your username and password into fields on the web page and press the Submit or Login button. From the standpoint of the underlying technology, this is no different than submitting any other form – only the names of the fields distinguish them as login or password fields and the security mechanism is implemented within the web application.
Web Performance Consulting
The next most common form is authentication via credentials passed in the HTTP message headers. There are a number of schemes, but we can consider them all together as HTTP Authentication. This form of authentication is built into the web browser and the web server based on RFCs related to the HTTP specification.
From a performance testing perspective, the various HTTP Authentication schemes work very similarly. However, it can be useful to understand the context in which they are used, especially when the web admin or developers might throw around their specific names – so I break down the HTTP Authentication methods into these three groups:
- Basic and Digest – Basic is defined in the HTTP 1.0 specification and digest is defined in RFC 2069. These are most commonly used on Apache and other open-source web servers. Basic sends the username and password insecurely (Base64-encoded text). Digest encrypts the password using an MD5 hash.
- NTLM and Negotiate (RFC-4559) is most commonly used on Microsoft IIS web servers. NTLM can be used exclusively, but modern servers typically use Negotiate to select which scheme will be used. The result is usually NTLM, but could also be Kerberos (or other yet-to-be-defined schemes).
- Proxy Authentication is defined in the HTTP 1.1 specification. This method is used by proxy servers when access to the proxy requires authentication. It may use any of the above schemes and operates in the same manner. However, it uses different headers to indicate to the browser that a proxy is requesting authentication, rather than the target server.
How they work
In all of these methods, the process begins when the browser requests a URL. If the URL requires authentication, the server returns a 401 response to the browser (or a 407 for a proxy server requesting authentication):
Example of a 401 response from a server requesting authentication via the Basic method
In the above example, the server has requested authentication using the WWW-Authenticate header. In that header, it indicates the scheme to be used to send the authentication credentials (Basic). It further specifies a realm that the credentials should apply to. You can think of the realm as a sub-site within the website for which the credentials will apply. This allows the browser to cache the credentials and automatically resend them whenever it is challenged for authentication with the same realm.
After prompting the user for credentials (or fetching them from the cache) the browser will respond with the same request, but with an additional Authentication header that includes the scheme and the credentials. For the Basic scheme, the credentials are a Base64-encoded version of the username and password. The Base64 encoding is required to comply with HTTP protocol specifications – it was never intended to provide security.
Example of a request with the basic authentication header
The Digest scheme works the same way, except that a series of MD5 hashes of the username, password and other information is returned to the server – making it considerably more secure. The NTLM and Negotiate schemes work similarly, but are a little more complex.
Before looking at an NTLM example, though, lets stop to consider the performance implications of the above example:
- There is an additional transaction required to get the web page. That implies the user will wait longer to get the web page and the server will encounter a higher request rate under load.
- Remember that the HTTP protocol is not stateful – each request is independent from every other. As a result, this process could happen for every request to the web server. For some of the methods, every javascript, CSS and image request could need two full transactions to satisfy each request.
- If the browser is smart enough to remember that a URL is part of a particular realm, then it will not need to make the second request – it can send the credentials with the first request. It might even try to guess what realm a URL belongs to based on, for example, the path of the URL.
For any system you are testing, it would be wise to inspect the recording closely to determine how many additional transactions are being performed due to the authentication scheme. If it becomes apparent that this is a performance bottleneck, it may be necessary to switch to a better performing scheme.
NTLM and Negotiate
Note: NTLM is now referred to as Windows Integrated Authentication in most documentation and the Windows UI, but the underlying protocols still use NTLM so we will use that terminology here.
An IIS server will typically respond to an unauthenticated request for protected resources as below. Note that it has returned two WWW-Authenticate headers, giving the browser a choice of which scheme to use, Negotiate or NTLM. The order of the schemes implies the schemes the server prefers.
Example of the first response from an IIS server
Modern versions of IE will elect to use the Negotiate scheme – though when the server is IIS and Kerberos is not configured on the workstation, it appears that IE masquerades NTLM as the Negotiate scheme. The resulting header will indicate the scheme the browser has chosen as well as some other encoded information. The details are beyond the scope of this article, but it does not yet include the credentials (username & password). The browser and server are still negotiating how the credentials will be sent. The second request from the browser will look like this:
Example of the second request in a Negotiate sequence
The server will respond with another 401 status and a WWW-Authenticate header that includes some additional encoded information that the browser can use to finally send the authentication credentials:
Example of the second response in a Negotiate sequence
Next the final request is made. This time the Authorization header includes the authentication credentials. You can’t tell by looking at it, but it is an encrypted conglomeration of the username, workgroup/domain, password hash and a few other things to improve security.
Example of the third (and final) request in a Negotiate sequence
After verifying the authentication credentials, the server will return the requested content with a status code 200 (ok) response..
The most obvious difference between this and the Basic HTTP Authentication example we saw earlier is that there are three transactions executed to fetch a single URL using NTLM instead of two using Basic.
At first glance, NTLM would have a significant performance disadvantage because of the additional transaction – and for a single URL this is true. However, NTLM and Negotiate are intended for use on modern servers that support persistent connections (indeed that can not work otherwise). Under these implementations, the authentication is not for a single request, as was the case for Basic authentication, but rather for the life of the connection. Assuming that many requests will be made on a single connection, then the overhead of the Connection Authentication Negotiation (CAN) is constant – two transactions per connection. If the connection is re-used for 100 transactions, the overhead is a mere 2% of the total. In reality it is even less than that, since the 401 responses from the server have little or no content and are therefore much smaller than the average response size.
There is one other performance-related detail to note: If the first request on a connection is a POST with very large content (e.g. a file upload or a very large Viewstate), then the cost of that POST is incurred twice, since it must be re-sent along with the final request.
CAN Transactions
A final note for users of Load Tester: we refer to the preliminary transactions as CAN transactions, or Connection Authentication Negotiation transactions. It is tempting to simply call them authentication transactions. But since the authentication actually happens in the final transaction, we felt that could be misleading. In the CAN transactions, the browser and server are negotiating what protocol will be used to transmit authentication credentials for the remainder of the connection.
Chris Merrill, Chief Engineer
When his dad brought home a Commodore PET computer, Chris was drawn into computers. 7 years later, after finishing his degree in Computer and Electrical Engineering at Purdue University, he found himself writing software for industrial control systems. His first foray into testing software resulted in an innovative control system for testing lubricants in automotive engines. The Internet grabbed his attention and he became one of the first Sun Certified Java Developers. His focus then locked on performance testing of websites. As Chief Engineer for Web Performance since 2001, Chris now spends his time turning real-world testing challenges into new features for the Load Tester product.