Smart Start Hotel Executive Report: The Last Mile and Beyond
By Roger Lopez, Benchmark Analyst, Gomez
Mr. Roger Lopez
If you have read my previous articles, or seen the Gomez: Hotel Room Search Benchmark, you will be familiar with the three metrics we use to measure website performance: Availability (Success Rate), Response Time and Consistency. Together these three metrics give us an idea of how a website is performing. These measurements are taken from Gomez’s global network of Internet Backbone nodes, featuring Tier 1 data centers with big servers and thick pipes out to the Internet.
This approach provides a great baseline of web performance, but also limited one. It doesn’t reveal all the real world ‘noise’ of the Internet, as experienced by website visitors at the ‘Last Mile’ i.e. at their desktops around the world. Factors such as geography, Internet Service Provider and connection speeds all impact the website experience at the desktop.
A customer in, say, Iowa may have a dramatically different website experience from a customer accessing your website from India. So how can you be sure that your website loads equally fast or transactions are consistently accurate and available for all your customers – wherever they are located and however they connect to the Internet?
According to a study by Nielsen Research, almost 90% of Internet users in the US have high speed (high broadband) connections, which leaves a little over 10% of users still on dial up (Figure 1).
But, before you write off testing your website’s performance from the Last Mile because you are sure that you have no dial up users at all, consider that most broadband wireless cards, which are highly popular with business travelers because they allow you to access the Internet from anywhere, will sometimes operate at dial up speeds depending on their location.
If we assume for a minute that the user population visiting a hotel website is distributed in the same way as the general Internet user population, this means that a website that is not dial up friendly could be jeopardizing 10% of business. Add in the factor of advertised maximum connection speeds, compared to the actual speed at which you are able to cruise the Internet on these connections, and a user who is paying for a DSL connection may only be able to download the site at dial up speeds. This is particularly true with cable Internet where the bandwidth is shared with other users on the same street, and so download speeds decrease when multiple people on the street use the Internet at the same time.
With this in mind, we put eight of the tests we use for the Gomez: Hotel Room Search Benchmark into the Last Mile environment for a month to find out how some of the hotels on the benchmark perform. Testing on the Last Mile is broken up into three categories: dial up, low broadband and high broadband. Using an analogy to differentiate the three we could say that dial up is like walking, low broadband is like driving and high broadband is like flying. We selected a sample of eight sites from the benchmark: Radisson, Best Western, Marriott, Starwood, Kayak, LasVegas.com, Hotels.com and InterContinental Hotels. We selected these brands because they represent the spectrum of performance within the three metrics we use to measure performance. We looked at the three core measurements and compared the results with tests performed on our Backbone network.
The fastest download speeds a home user would see are high broadband so we would expect the results for response time and success rate to be similar to Backbone performance. Dial up will have significantly different results since it’s the slowest connection.
First up we’ll look at Success Rate. In the Backbone environment, which as I mentioned before is like a lab environment, all of the sites we chose for this exercise were above 99%. The behavior we have observed is that Success Rates decrease as connection speeds decease. Starwood performed best in the Last Mile tests, but even its performance for high and low broadband were lower than success rates in Backbone tests. Starwoods’s dial up success rate was 93%. In comparison, Kayak and Best Western were the sites most affected when measured from a dial up connection with 36% and 42% availability respectively.
A large page size will have a role to play in success rates. This isn’t really visible in the Backbone tests since high speed connections can handle larger pages without much of an effect on success rates, but as connections get slower there is a higher possibility of data being dropped and an increase in errors due to timeouts. Looking at Figure 3, it is interesting to note that the two sites with the worst success rate for dial up, low and high broadband are also two of the heaviest in terms of number of bytes on their web pages.
Examining response times tells a similar story: the faster the connection, the faster the response time. The thing to notice here is the gap between high broadband and low broadband, and then the really big gap between low broadband and dial up (Figure 4). To see how radically different the response times can be, let’s take a look at Radisson. Testing via our Backbone nodes Radisson averages 5.01 seconds response time. But for Last Mile tests, on high broadband the same transaction takes 9.30 seconds, almost double the response time. Moving on to low broadband the average response time increases to 24.84 seconds. Dial up average response time comes in at 79.51 seconds. Other sites don’t fare as well as Radisson which has the fastest dial up average of the sites in the study.
Of course these results should be put into perspective and assessed alongside the expectations of the users on the different types of connections. A user on dial up might be willing to take a whole minute to go through the transaction of booking a room. But I’d be surprised if even a user on dial up would be willing to wait two and a half minutes for trying to book a room on InterContinental Hotels or wait even longer for Kayak.
All this leads us to what I consider to be perhaps the most important metric for the Last Mile, consistency. If the user is able to get to the website, they will most likely go through with the process of booking a room. If their experience with the site is always slow they will know this and expect to have to wait x amount of time to book a room.
But if the user gets to the site and can book a room in 10 seconds one day but it takes 60 seconds the next time, there is a greater chance that that user won’t return to this site if a competitor’s site can deliver the predictable, consistent experience all the time. This is where consistency, a measure of site response variation, comes in.
Consistency for Radisson was the best across all Last Mile categories, dial up, low and high broadband and on our Backbone tests. Keeping with the Radisson example we can see that even then the variation in response time for low broadband was 18.65 seconds and 35.44 seconds for dial up. This means that every time someone tries to book a room on Radisson using a dial up connection, their transaction could take anywhere from 44.07 seconds, to as high as 114.54 seconds. That variation only gets worse with the rest of the sites in the study.
In conclusion, testing website performance from the Last Mile provides a much richer picture of how your customers are experiencing and interacting with your website. Whether they are doing price comparisons, research or booking rooms, the web experience you deliver impacts your brand and your revenue and, as we’ve seen, can vary dramatically at the Last Mile. My mantra is ‘know your customers.’ If you think you don’t have many dial up users going to your site, you should think about why that is? It could just be that they got tired of waiting for it to download.
Gomez Hotel Performance Index
May Results (5/1/08 – 6/1/08)
The Gomez Hotel Performance Index helps hoteliers keep an eye on the competition, measure customer experiences, and accurately understand the connection between web application performance, customer behavior and business results. Covering the top online hoteliers and aggregators, it measures the responsiveness; availability and consistency of the benchmark participants along these metrics (see full methodology):
Success Rate Summary
The majority of the hoteliers had a success rate above the benchmark average of 98.88%. Gomez considers a rate above 98% as excellent performance. In the month of May, 17 of the 24 hoteliers performed above the benchmark average. Those performing above the average include Marriott, Best Western, Radisson and Expedia. Notables that performed below the benchmark average were Cheap Tickets and Hyatt rounding out the bottom of the list with 95.32% availability.
Response Time Summary
The benchmark average of 12.87 seconds is a little slower than it has been in previous months. Radisson, Hotels.com and Best Western lead the pack with Radisson averaging sub 6 second response times. Orbitz is at the bottom of the chart with 26.25 seconds average response time.
Consistency Summary
Consistency is a measure of the disparity of the response time from visit to visit for completed transactions. The average consistency for the benchmark in May was 8.40 seconds. Radisson tops the chart with a consistency of 4.36 seconds. 15 of the 24 hoteliers performed above the benchmark average. Notables below the benchmark average include Hyatt and InterContinental Hotels.
Roger Lopez is a Consultant in Gomez’s Professional Services Group. His role includes working directly with customers to help them understand their site’s performance and identify possible areas of improvement. Prior to joining Gomez just over two years ago, Roger held positions as a technology support executive at TTG, Inc., a business territory design software company and in search engine optimization for BizX, a bartering service for online marketplaces. Roger has a degree in Systems Engineering from the Universidad de las Americas. Roger can be reached at rlopez@gomez.com
