Quality and Image Byte Size

There is a general understanding that adjusting the quality level in lossy formats (JPEG, JPEG 2000, JPEG XP, WebP) results in a commensurate reduction in bytes. The more compression applied, the fewer bytes there will be. There is also a point of diminishing returns. Setting compression to 100 doesn’t result in a pristine lossless image, just a large image.

Figure 13-4 shows the quality graph for the different image formats compared to the relative byte savings. This does not compare the relative sizes between formats but rather the change in bytes within the format. Changing between formats will yield additional relative byte savings.

This quality graph is based on a sample set of 1,000 product detail images and is fairly representative of a typical quality scale. This also highlights the variances between different libraries. It also emphasizes that quality does not mean percentage. It is tempting to conflate quality index to image quality or even file size.

Figure 13-4. Quality graph comparing image formats to byte savings

As you can see, regardless of format, each encoding library can impact the byte size of an image differently. Specifically, there is a rapid reduction in byte size until we hit an index of around 35. Also, we can see the distortion of the highest index values on the scale. If we reset the scale and focus on an index of 90 through 35, we adjust our expectations (see Figure 13-5).

Figure 13-5. Quality graph comparing image formats to byte savings with index between 90 and 35

Nearly universally we can see that reducing the quality index can quickly reduce file sizes. Most follow a similar curve shape, but there are still noticeable differences. What we can conclude from this is that we should expect an additional 20% byte savings by moving from quality 90 to quality 80, and another 20% by moving to quality 70. From there the gains become smaller but are still impressive.

Quality Index and SSIM

But does the quality index of one encoder equal the quality index of another? Can we just assume that all quality indexes are the same? Using Structural Similarity (SSIM) calculations, we can compare the different encoding libraries and their effects. Can we assume that selecting index 80 in one library is the same as quality 80 in another? Or across formats?

Using the same dataset, we can compare the SSIM values at each index value. Using the 90th percentile value (conservative) we arrive at the curves shown in Figure 13-6. It’s important to emphasize that this is a conservative view, and an individual image could well get a lower SSIM value when run through the different quality indexes. The purpose of this illustration is to provide general guidance and conclusions, so a 90th percentile was selected.

Figure 13-6. Quality graph comparing image formats to SSIM values

Clearly quality is not a consistent metric across different libraries. Each encoding library impacts the visual perception differently at the same quality index. If you set libjpeg-turbo to quality 80, you would expect the same SSIM of a MozJPEG set at quality 65.

Just as before, the top and bottom indexes heavily skew the graph. Zooming in on index 90 through 40 yields the charts shown in Figures 13-7 and 13-8.

Figure 13-7. Quality graph comparing image formats to SSIM values with index between 90 and 40

Figure 13-8. Quality index graph comparing JPEGs to other formats

One thing this does not take into account is DPR. There is anecdotal evidence that suggests that the perception of higher SSIM values goes down based on pixel density as well as form factor; that is, humans can accept a higher SSIM value when it is on a smartphone versus on a desktop. Research is early and inconclusive on the impact of visual perception based on display form factor.

How do we select a quality index and apply it across the different encoders and expect the same results? Fortunately for you, I have run the regressions and derived the charts in Figures 13-9 and 13-10 to help with our conversions.

Figure 13-9. Quality index: JPEG (libjpegturbo) versus other formats

Figure 13-10. Quality index: JPEG (libjpegturbo) versus other formats

Is this a chapter on image quality or image delivery? Well, both. They are tightly linked. In order for us to select the best image quality to reduce bytes, we should also keep in mind the effective equivalent in the other formats.

So, we’ve reached two major conclusions about how to better deliver images:

• Focus on the desired quality index for your images to maximize SSIM and reduce file sizes.

• Layer image format and responsive images after the quality index adjustment.

Selecting SSIM and Quality Use Cases

You can take this one step further and create use cases for quality:

High: 0.01 SSIM
Medium: 0.03 SSIM
Low:  0.05 SSIM

In this way you could intentionally distort the image to maximize the user experience. For example, you could use the network Client Hints to inform the quality use case. Alternatively, you could look at the HTTP socket performance (packet RTT) or instrument latency detection with service workers. By doing so, you could adjust the user experience based on the hostility of the network conditions.

This is very similar to what was suggested in Chapter 12. In fact, both can be done at the same time for maximal benefit: adjust both the image dimensions and then adjust the quality. There are many possibilities.

Of course, there is always a point of diminishing returns. Applying these use cases to a 800-byte image has little value to the user experience. However, if the image is 100 KB, then of course you would want to apply this algorithm. Remember: every packet counts—especially on poor network situations.

The bottom line is this: if you can gain a full packet in savings it is probably worth adjusting image quality. Accordingly, we might augment the previous chart as follows:

High: 0.01 SSIM
Medium: 0.03 SSIM and >4,500 Byte savings (~3 packets)
Low:  0.05 SSIM and >12,000 Byte savings (~8 packets)

Creating Consensus on Quality Index

One final word about quality. As I mentioned, this topic is often very emotive—especially among those in your organization who are the custodians of brand (i.e., your marketing teams). Their job is to ensure that the public’s opinion of your brand is positive. You, in contrast, are responsible for ensuring that the site or app works for the highest number of people. These are two sides of the same coin.

In order to bring marketing and creative teams onboard with adjusting the quality index of your images, it is useful to show instead of explain. For example, you can gain consensus by selecting a set of images and running them through the different quality indexes, as shown in Figure 13-11.

Figure 13-11. Building consensus on image quality

Make sure you are consistent: if you are using MozJPEG as your JPEG engine, then use this to initiate the conversation—don’t use libjpeg-turbo.

Quality Index Conclusion

When applying changes to the quality index, follow these best practices (see Figure 13-12):

• Reduce the quality index based on SSIM values instead of a fixed setting.

• Apply the equivalent quality index to other formats.

• Add network awareness to select a lower-quality index.

• Use the Client Hint Save-Data: on to select a lower-quality index.

Figure 13-12. Workflow for selecting the right image quality

Size on Disk

Most modern filesystems, from EXT4 to NTFS, use a block size of 4 KB. This ensures that the block size lines up with the physical attributes of the disk. Alignment to physical disk matters more with a spinning disk than it does for solid state. There is always inefficiency in filling every block. The assumption is that there will be more completely filled blocks than partially filled blocks.

In the previous example, rounding to the nearest block size adds an extra 25% to the total storage; that is, the 9 GB actually uses 12 GB of storage. Fortunately, as file sizes increase, the impact of size on disk decreases.

How Do I Avoid Cache Busting and Redownloading?

We have two objectives when using sharding: maximize the browser cache, and avoid downloading the same resource twice. However we implement domain sharding, we must ensure that i-love-broccoli.jpg doesn’t show up using img1.example.com on the first page but img2.example.com on the second. This would effectively void the browser cache and force redownloading the content.

To avoid this, you should partition your images into groups of content. However, avoid using a counter to switch between shards. Also, it is tempting to use one shard for CSS and another for JPEG. You should avoid this temptation because you don’t want all the critical resources to be bunched up on a single request queue. Instead, use a hash or an index to equally distribute filenames between available shards.

How Many Shards Should I Use?

Selecting the right number of shards is not as clear cut as you would expect. Early research suggested two to four shards per page, but this was a best practice from 2007 when browsers only made two connections per hostname. Steve Souders has provided the most recent guidance, suggesting ~20 resources per domain to provide a good balance of sharding for performance.

This remains the best general guidance. However, there are other questions, such as: what is the impact on congestion control and TCP scaling? If each socket is attempting to maximize the congestion window but competing with itself, this could result in packet loss and thus decrease overall performance. The size of the resources also impacts the effectiveness of sharding. Sharding works because many small resources don’t use more than a few packets to send/receive. (We discussed this more in Chapter 10.) However, this value can diminish with larger content, many more resources in parallel, or low bandwidth.

What Should I Do for HTTP/2?

Is domain sharding an anti-pattern for HTTP/2? The short answer is: no. The longer answer is: it could be, if you don’t consider HTTP/2 in your implementation.

HTTP/2 has many advantages, one of which is the ability to have multiple parallel requests on a single socket connection. In this way we can avoid the HTTP/1.1 head-of-line blocking problem. Domain sharding is not necessary to saturate the network connection. By using a single socket you can also scale the congestion window more quickly and avoid packet loss and retransmission.

However, there are a number of barriers to HTTP/2 adoption. Aside from the consideration of adopting TLS (because there aren’t any implementations that can do HTTP/2 without TLS), there is also the factor of the user adoption curve. HTTP/2 requires recent versions of modern browsers. For native apps it also requires modern OSes (or client libraries) that can likewise use HTTP/2. Beyond user adoption there is also the challenge of corporate (and home) content filters that intentionally decrypt and resign TLS encryption. In some situations, Akamai has observed the interception of TLS requests to be as high as 17% in a region or demographic. There are many causes, most likely from web filters or local antivirus software. The problem is the same: these content filter proxies likely do not use HTTP/2, even if the browser behind the proxy supports HTTP/2.

As with many other web technologies, we should expect the organic adoption of HTTP/2 to take many years. Consider that between 1998 and 2016, only 10% of users were on IPv6 reachable networks. Likewise, SNI (Server Name Indicator) support in TLS has been a standard since 2003, but it wasn’t until 2016 that 95% of TLS web traffic supported SNI (mostly as a result of the end-of-life of Windows XP). As of 2016, HTTP/2 adoption is between 50% and 75% depending on the demographic or segmentation. We should expect the long tail of adoption of HTTP/2 to take three to five years before we come close to 100%.

So what should we do in this interim?

Option 1 is to dynamically generate the domain sharding. If the user is connected on a HTTP/2 connection, disable domain sharding. If the user has HTTP/1.1, then utilize multiple sharding as before. This approach, of course, requires that your local caching infrastructure and your CDN be aware of the different rendered outputs and add the HTTP/2 connection as part of the cache key. Unfortunately, there isn’t a corresponding Vary: header that can properly describe a variation based on the protocol. The best solution is to use Vary: User-Agent to communicate the variation (as you would with a RESS design).

Option 2 is to simply ignore the problem. Fortunately, most (if not all) HTTP/2 implementations have an optimization to address domain sharding. Specifically, if the shared domain resolves by DNS to the same IP and the shard is on the same certificate (the hostname is in the Subject-Alternate-Name list), then the HTTP/2 connection will consolidate the sockets. In this way, multiple hosts will use the same HTTP/2 connection and therefore avoid any penalty of sharding. It still remains a single connection. The only penalty is the DNS request. See Figure 13-20.

Figure 13-20. HTTP/2 with two resource domains

This is the easiest option because it simply means less work. It also allows you to continue to use sharding for the laggard adopters. And it is these laggard adopters that likely need any performance bump you can give them.

Best Practices

You should continue to use multiple domain shards for your website to maximize the connection throughput. This also helps you avoid images from blocking more critical resources like CSS and JS.

In preparation for HTTP/2 make sure that you:

• Use the same DNS for all of your shards and primary hostname.

• Use the same certificates—add them to the SAN fields of your TLS certificate.

Secure Transport of Images

Up until recently the majority of the Web has been delivered unencrypted. As we have all experienced, there are many locations where content can be hijacked in an unencrypted flow. Public WiFi does this intentionally to force you through a captive portal before granting you access to the Internet. ISPs, with good intentions, have notoriously applied higher compression, distorting the visual quality of your brand. Using Cache-Control: no-transform works for some, but not all, well-behaved image transformations (see Figure 13-21). But there are also not-so-well-intentioned transparent proxies that hijack image requests and replace the content with different advertisements or placeholders.

Figure 13-21. Use Cache-Control: no-transform to prevent degraded quality by ISP proxies

Securing the transport for images is straightforward. Using TLS you can ensure that the communication from user to server is trusted and that there aren’t any middle boxes interfering with or mutating your content. Moving to HTTP/2 also requires the use of TLS.

Secure Transformation of Images

Securing image delivery is about more than just the transport layer. We should also be concerned about the attack surface of our transformation engines. Whether you are using an on-premise image transformation engine or an off-premise one, there are many possible vulnerabilities. Third-party and open source libraries are extremely useful but also can introduce risk to the enterprise if not properly isolated.

An index of Common Vulnerabilities and Exposures (CVE) is maintained by Mitre (see Figure 13-22). It is critical to keep up-to-date with the latest known exploits on the libraries and tools used in your image transformation workflow. Isolating and patching should be part of your regular team practices.

Figure 13-22. CVEs reported for ImageMagick and common image libraries

The main concern for image transformation engines is if a contaminated image enters for processing and, through the decode or mutation process, exploits a vulnerability. This could leverage a byte alteration from the logic edge case, checksum collision, or remote code execution. Consider that the famous Jailbreakme exploit that allowed jailbreaking on iOS 3 used a flaw in the TIFF decoder in iOS. This single flaw allowed rooting of the entire operating system. Imagine the potential impact on your images. This vulnerability could impact subsequent images, possibly tagging them with brand-damaging messages. Just because the bytes of the image have left the processor doesn’t mean that there isn’t residual code running on the thread. The last thing you want is all of your product images graffitied with “EAT BROCCOLI” without your realizing it (see Figure 13-23).

Figure 13-23. We want to avoid one image affecting other images on the platform

How could a contaminated image enter your workflow?

• User-generated images, compromised at source

• Vendor-supplied product images, compromised at source

• In-house photography, compromised by malware on the artist’s laptop

It is easy to imagine how a compromised image could enter your workflow. So how can you ensure that a compromised image doesn’t impact your ecosystem? How do you isolate the impact to just that compromised image? How can you minimize risk and exposure to your image transformation service?

Secure Transformation: Architecture

Whether your image transformation is on premise or with a cloud-based SaaS provider, you should evaluate the architectural security of the transformation engine. Ideally, there should be isolation at every level of processing. You want to ensure that no single compromised image can affect other parallel threads/processes/systems that are also transforming other images. You also need to ensure that there isn’t any residual code that may impact the next image processed by this specific thread.

A well-secured transformation architecture should consider three major areas for isolation (see Figure 13-24):

• TCP connection pools (retrieving and storing)

• Transformation engine (e.g., ImageMagick)

• Encoding and decoding shared objects

Figure 13-24. A model for secure image transformation architecture

For example:

• We need to ensure that there is no way that images being sent or received via TCP (or disk) can impact another thread or process. The initiating worker should only have access to the stream of bytes for this job.

• The transformation engine, such as ImageMagick, must not be able to store, execute, or preserve any state between image processing. The worker threads must be each isolated to exclusive scratch areas and restricted to access only certain system libraries. For example, the transformation engine should not be able to open up new TCP sockets or leave temporary files or memory state between jobs.

• The various encoding and decoding shared objects (e.g., libjpeg-turbo) also need to be isolated. Memory state should not be allowed to persist or be accessed by parallel threads or other jobs.

This is not an exhaustive list of ways to isolate and segment the architecture. Your local security team should be able to help you ensure that there is no way that a maliciously tampered image can have ecological impact on the rest of your valuable assets. If you are using a cloud solution, you should also ensure that the same level of scrutiny can be applied.