Graceful Degradation and Disabling Heavy Features

One contingency is to disable some of the site’s heavier features. Building in the ability to turn certain features on or off can help capacity and operations respond significantly, even in the absence of some massive traffic event. Having a quick, one-line configuration parameter in the application with values of on or off can be of enormous value, particularly when that feature is either the cause of a problem or contributing to unacceptable performance. For example, you can have the web servers perform geographic (country) lookups based on client IP addresses for logged-out users in an effort to deduce their language preferences. It’s an operation that enhances the user experience, but it is yet another function the application must handle. Back at Flickr, after the launch of the localized version of the service in seven different languages, the aforementioned feature was turned on with the launch. It almost immediately placed too much load on the mechanisms that carried out the country lookups. The problem turned out to be an artificial throttle placed on the rate of requests the geo server could handle, which was tuned too conservatively. The issue was isolated and fixed by lifting the throttle to a more acceptable level and then turned the feature (which is mostly transparent) back on. Had a quick on/off switch not been implemented for that feature—in other words, if it had been hardcoded within the application—it would have taken more time to troubleshoot, disable, and fix. During this time, the site would have been in a degraded state, or possibly even down.

Ideally, you should work with the various products, development, design, and operations groups to identify an agreed upon set of features to which you can apply on/off switches. When faced with a choice between having the full site go down and operating it with a reduced set of features, it’s sometimes easy to compromise. This is particularly important in domains such as ecommerce for which there’s immediate impact to the bottom line.

Here’s an anecdotal example: a large news organization was serving web pages with the tallied results of a United States presidential election. Its web servers were running close to capacity all day. On the night of the election, traffic overwhelmed the organization. It had no spare servers to add quickly, and the website began crumbling, serving broken images, and some pages with images but no other content. The decision was quickly made to stop logging. Now remember, this was before any large-scale traffic counting services were available, and all traffic metrics for ad serving were taken from the logs written by the origin servers themselves, audited alongside any ad-serving logs. By disabling logging, the site could continue and buy some time to assemble more equipment to handle the load. For hours, the site went without any concrete way to measure how much traffic it received, on what was certainly its busiest traffic day up until that point. The decision to stop logging all traffic was the correct one. The relief on the disk systems was enough to allow the servers to recover and serve the rest of the traffic spike, which lasted into the early hours of the next day.

Likewise, at Netflix, at times, personalization of recommendations was foregone and generic recommendations were served in the interest of better response time. Today, most websites use a wide variety of third-party services for, say, advertising, analytics, social media presence, and so on. In times of stress, you can turn off one or more of these third-party services, given that they are not core to the user experience.

Baked Static Pages and Beyond

Another technique frequently employed by sites that encounter heavy and erratic traffic is to convert a dynamic page to a static HTML page. This can be either terribly difficult or very easy, depending on how dynamic the page is originally—increasingly, with today’s websites, a large percentage of client time is spent executing JavaScript code. You can gain some safety by building static pages for only the most popular and least dynamic pages.

Converting pages from dynamic to static is called baking a web page. An example of how this can work well is using a news page showing recent photos that one updates every two or three hours. Under normal conditions, the obvious design is to create a dynamic page that reads in the photos of the hour from a database or other content management system. Under duress, you could hardcode the image URLs into the page and change them manually as needed.

Baking a page into static HTML clearly breaks a lot of functionality found in today’s more dynamic websites, but static pages come with some operational advantages:

• They don’t initiate database lookups.

• They can be served very fast. Static content can display up to 10 times faster than dynamic pages that must wait for other backend services.

• They are easy to cache. If you need even more speed, you can cache static pages quite easily via reverse-proxy caching. This, of course, can introduce an entire new layer of complexity, but if you are already using caching for other parts of a site, it can be easily implemented.

The disadvantages of baking static HTML pages under duress are also worth noting:

• You need a framework in which to bake and rebake those pages quickly and easily. Ideally, you should have a single command or button on a web page that will replace the original dynamic page with the static HTML replacement, and also reverse the operation. This takes some time and effort to develop.

• You need to track what is where so that when changes do happen, you can propagate them. The generation of the static content should be synchronized with the content origin (usually a database). If changes are made to the database, those changes must be reflected (rebaked) to all of the static pages that include the content.

Beyond static pages, you can employ many other approaches to minimize the impact on user experience during times of high traffic. For instance, you can substitute high-resolution images with low-resolution images. In a similar vein, you can substitute 8 K UHD/4 K UHD video with HD video.

Cache but Serve Stale

Caching is used in many different components within backend infrastructures. Caching frequently requested objects from clients (or other backend server layers) can have a significantly positive effect on performance and scalability but also requires careful deployment and increases the cost of management. Normally, caching done this way accelerates content delivery, and the freshness of each cached object is controlled and monitored by headers that can indicate the age of an object and how long it’s desirable to serve the cached version of it.

As an extension to baking pages, and to take more advantage of caching, you can relax the content’s freshness requirements. This is usually a lot easier than building static pages where there were none before, but it involves more complexities. In the context of images and videos, t he use of Content Delivery Networks (CDNs)—which are typically distributed around the globe—has become a norm. This reduces response time by obviating the need to pull content from the origin server. CDNs also purge (remove and update) content constantly so that fresh content is delivered. At Netflix, the use of a CDN was key to minimizing buffering when a subscriber streamed a video. Likewise, at Twitter, the use of a CDN was key to ensuring fast delivery of photos and videos in tweets.