There’s no doubt that Responsive Web Design (RWD) has had a huge impact on the way we build websites nowadays. If you are operating a website, chances are that it’s either responsive already, or it will be soon enough. There is no other way to serve websites while coping with the diversity of devices your users are using to access your site, and providing these users a pleasant experience.
But it’s not all sunshine and rainbows. The issue of responsive images has been a thorn in RWD’s side for a long while and a huge source of pain for developers trying to implement performant responsive websites.
Early on, when RWD was coined in Ethan Marcotte’s seminal article, the approach to images was fairly simple, if not to say naive: just send the browser the largest possible image and let it resize it on the client side to match the responsive layout.
While that approach works when testing the simpler use cases over the local network or even on a high-speed broadband network, it fails miserably when we take into account the reason RWD was needed in the first place: mobile devices over cellular networks. With the earlier approach we’re sending unnecessarily large images to mobile devices, often over poor connectivity.
The immediate result of that approach was that RWD got a reputation for being slow, and it became obvious to many that a responsive website meant a bloated one.
The web developer community realized that RWD is the only scalable future for building websites that can address the myriad devices out there. At the same time, it also realized that users cannot afford to download 85 MB (!!!) of data on their potentially limited data plans when looking up sunglasses. As a result, the community decided to take action.
And, as often happens on the Web, the first course of action was to hack around the problem.
There were multiple attempts to resolve the responsive images issue using JavaScript or server-side logic. These attempts included:
Serving images limited to the viewport dimensions based on the user agent string (e.g., sencha.io)
Serving images limited to the viewport dimensions based on a cookie set by the website (e.g., adaptive images)
Adding the actual src
attribute to images via script
Overriding the page’s <base href>
via script
Rewriting the entire page’s HTML via script after modifying the image URLs (e.g., mobify.js)
Serving oversized yet highly compressed images, to avoid retina-related blurriness (aka compressive images)
Heroic and fearless as some of these attempts were, it was obvious pretty early on that all of them were lacking either in accuracy or performance.
The server-side approach didn’t handle cases where the browser viewport was not identical to the device’s dimensions (desktop and some tablets) or didn’t work on first load. At the same time, the client-side methods were adding a non-negligible latency to the browser’s resource loading process, by adding the images fairly late, and preventing the browser’s preloader (which we discussed in Chapter 7) from loading them earlier on.
So, the Responsive Image Community Group (RICG) was formed to find native in-browser solutions to this pressing issue, and after a long struggle, managed to do so.
But before we dive into the solutions, let’s take a look at the various use cases that needed addressing.
The first step of solving the responsive images problem was to define the various use cases that developers face when using images on the Web today. The result was a document that covered many different aspects of the problem. We’ll cover the major ones.
In order to frame this use case, think of a layout that resembles Figures 11-1 and 11-2.
As you may have noticed, the image in these layouts remains in the same dimensions regardless of the layout changes that result from the responsive design, the same as images in nonresponsive designs. So why would we consider this image “responsive”?
Well, the problem starts when you’re browsing that site over a high-resolution screen, and notice significant blur.
Retina screens “need” images that are twice (or more) the resolution of traditional resolution screens. If the image we provide the browser in that case is of lower dimensions, the browser will upscale the image, resulting in blurriness.
So, how do we resolve it? The first reaction from web developers was to upgrade their images, and serve larger images to their entire audience. The problem with this was that they were now serving larger images to all devices, including the ones that have absolutely no need for images that are twice as large.
For these devices, the result of the larger images was, besides the bandwidth costs and delay, increased CPU costs and higher memory consumption, as larger images had to be decoded and then stored in memory. See Chapter 9 for more details.
The variable-dimensions image use case is slightly different from the previous one, since it’s tightly related to responsive websites. Consider the layouts shown in Figures 11-3 and 11-4.
In this case, larger viewports need larger images; otherwise, the images will be blurry. But, similiarly to the previous use case, higher-resolution screens also need larger images. Again, the initial developer response was to send the largest possible image, but that’s hardly scalable. In a world with 28-inch high-resolution screens, the largest possible image can be pretty big. That’s not something that you want to send down a mobile connection to your average user. That case is sometimes referred to as “download and shrink”: you’re downloading bytes that users don’t necessarily need, burdening their mobile data plan and slowing down their experience.
What happens when your images are materially different in the various responsive breakpoints? When you want to adapt the images to the different breakpoints so that they will be clearer in the context of the different layout?
Well, that’s a different use case from what we’ve seen before. A few examples of this use case are shown in Figures 11-5 and 11-6.
What we see in these examples is that serving the intended images is essential in order for the user to fully understand the page and for the page to be properly laid out. The different proportions of the images mean that if we were to serve images that don’t fit the layout, the layout would break.
In a way, this use case is less about “performance optimization” and more about “content optimization.” The problem here is not so much about the image’s quality as it is about getting the image’s message accross to users, regardless of viewport restrictions. With that said, when large parts of the image are being cropped away when served to mobile devices, that certainly doesn’t help the site’s performance.
The fixed and variable dimensions use cases are often referred to together as resolution switching. The main difference between resolution switching and art direction is an issue of control. In the resolution switching cases, the issue at hand is quality. We want the user to get the best experience, where experience is a combination of visual quality and speed (and one might claim that the eventual bandwidth costs and battery life are also part of the experience). These aspects of the user’s experience are not something that the web developer knows in advance, and any attempts to predict the user’s “context” are bound to fail.
Therefore, for the resolution switching case, we want to give the browser the final word. Our solutions enable the developer to declare multiple resources and enable the browser to pick the one that best fits the user’s current situation.
On the other hand, for the art direction case, the issue is one of fidelity. If the browser shows a different image than the one the developer intended, users may get a distorted image or a broken layout, hurting their experience as well as their ability to use the site properly. In this case, the browser doesn’t know more about the user’s context than the site’s developer did when creating the site. So, we want the browser to be “bound” to obey the developer’s instructions and download the specific image that the developer had in mind for particular viewport dimensions or other environmental constraints.
We need two distinct mechanisms to handle each one of these cases, one where the control is in the browser’s hands, with the guidance of developers, and another where the control is in the developer’s hands.
Another use case, which isn’t directly related to responsive images but is very relevant to content images in general, is that of serving different image formats according to browser support. Traditionally, the answer to that has been content negotiation: have the browser advertise its capabilities using the Accept
request header, and then the server can dynamically serve it the right image.
While that approach certainly works (as we will discuss in Chapter 13), it doesn’t work for everyone. There are scenarios where the images are served from a static host (e.g., gh-pages or S3), where you have no control on the server-side logic and cannot dynamically adapt the image to the headers the browser advertises.
The “download and hide” scenario often happens when desktop sites are retrofitted to be responsive and some parts of the page are just not needed on mobile.
A common mistake in such a scenario is to hide the unnecessary parts with CSS and hope for the best. The problem with that approach (as you probably guessed from the scenario’s name) is that even if the end user does not see these parts of the page, the resources they require—and content images in particular—are downloaded nonetheless.
You could think of this scenario as a form of art direction, where in some viewport sizes, the required image is a zero-sized one. We’ll later see how to address this use case properly.
There can be cases where a certain image does not strictly fall into a single use case, but combines a little bit of both art direction and resolution switching, depending on the breakpoint we see it in. For example, consider Figure 11-7.
We could also mix the image formats use case here to further optimize the delivery when content negotiation is not an option.
So when we’re looking into making a certain image responsive, the question we should ask ourselves is not “what is the use case this image fits into?” but “what are the use cases?”
We’ve detailed the various use cases that need addressing, but how do we address them in practice? That is exactly what we’ll explore next.
We have seen that the use cases are split into two major cases: art direction and resolution switching. Because of the fundemental difference between these two cases, we also need two different syntax machanisms in order to tackle them.
Since these are new additions to HTML that have made some noise recently, you may have heard about them: the picture
element and the srcset
attribute. In general the picture
element is designed to handle the art direction use case, and srcset
is designed to handle resolution switching.
With that background in place, we’re ready to dive into the details of each syntax part.
So, you have an image of a cat that you want to display on your site, where said image would have the same dimensions regardless of responsive breakpoints. So, you start out writing something like:
<img
src=
"cat.jpg"
alt=
"A funky cat"
>
But, when viewing that work from a retina screen, you notice a blur. Each image pixel is displayed over 4 physical pixels, and it just doesn’t look sharp. You want to provide the browser with a 2x image, twice the width and twice the height, which would get rid of the blurriness, but without sending that over to browsers that don’t need it. The following syntax will do just that:
<img
src=
"cat.jpg"
srcset=
"cat_2x.jpg 2x"
alt=
"A funky cat"
>
That is not extremely different than what we’ve seen earlier. All we added is the srcset
attribute, providing the browser an alternative resource to fetch for the same image. That attribute enables the browser to create a list of available resources, so it can pick which one to download and display.
As you probably noticed, that syntax enables us to tackle the fixed-dimensions use case we discussed earlier.
But what if we wanted to provide multiple alternative resources? Well, srcset
is actually a comma-delimited list, so you can provide as many
resources as you want!
<img
src=
"cat.jpg"
srcset=
"cat_2x.jpg 2x, cat_2.8x.jpg 2.8x"
alt=
"A funky cat"
>
Simple, right? The value of srcset
in the preceding example is a list of the resources for the browser to choose from. Each resource has a descriptor attached to it, which tells the browser something about this resource to make its job of picking the right one easier. In this case, the descriptor in question is the x descriptor, which describes the resource’s density. That gives the browser the knowledge to pick the resource that best fits the user’s screen.
Image density is the ratio between the image pixels that you provide the browser and the area (in CSS pixels) that the image is displayed on. Let’s say you have a 400×400 CSS pixels space in which to fit an image and you provide the browser a 600×600 pixel image to fit that space. That image would be displayed with a density of 1.5, and would look perfectly sharp on screens with up to similar density, but not necessarily on screens with higher density.
Now, different browsers may do different things when picking the best resource, and they are entitled to do that. The specification is purposely vague about the selection process inside srcset
, in order to enable browsers to innovate in that space.
Therefore, browsers can take not only the screen density into account, but also the user’s cache state, network conditions, user
preference, and more.
Already today there are differences between browsers regarding which image they fetch when the screen is not an exact match to any of the resources, or differences when some of the resources in the list are already in the browser’s cache. These differences are likely to increase over time as browsers get smarter about resource downloads, so you should not rely on the browser picking one specific resource over another.
Now the fixed-dimensions case is great when this is what you need, but in responsive designs the variable-dimensions case is often more common. Your image changes its size as the viewport changes, either due to the fluid layout that contains it or due to a breakpoint change that impacted it.
The syntax to achieve that would be:
<img
src=
"cat.jpg"
srcset=
"cat_200w.jpg 200w, cat_400w.jpg 400w"
sizes=
"100vw"
alt=
"A funky cat"
>
That’s very similar to what we’ve seen before, but with different descriptors. The w descriptor is there to describe the width of the image, in pixels.
And what’s that sizes
attribute that I sneaked in there, you ask? Nice catch—I hoped I could get that by you. It’s there to tell the browser what the image’s display width would be. The browser needs that in order to figure out which resource it should download, and it really doesn’t know that at the time it starts downloading images.
In order to know the dimensions in which images will be displayed, the browser needs to perform layout, and in order to do that it needs to download all the CSS in your page, process it, and calculate which rules apply. Only then can it calculate the layout of all the elements in the page, and it’s pretty late in the game. To make things worse, the downloaded image resources can also impact the layout, as the image’s intrinsic dimensions are used to lay it out in case neither HTML nor CSS knows better. So, waiting for layout information to decide which image to download is just not an option.
This is the reason we need the sizes
attribute to tell the browser what the image dimensions would be. In the preceding case we told the browser that the images will be displayed at 100vw
or 100% of the viewport width (see Figure 11-8).
100vw
is also the image width that the browser would assume when calculating density if we didn’t include a sizes
attribute at all (even though we probably should include it, as our markup would be invalid if we didn’t).
But often in responsive layouts, images take a smaller chunk of the viewport, and assuming they are 100% of the viewport width would mean downloading images that are just too large. For these cases, we can define a different CSS length as the value of sizes
—for example, 70vw
(see Figure 11-9).
See? That wasn’t so hard, was it?
Variable width images often require viewport-dependent CSS lengths (e.g., the vw
units), but if we want to, we could
satisfy the fixed-dimensions case by using w
descriptors and setting sizes
to a non-viewport-dependent length—500px
.
In other types of layout it can get more complicated than that. How can we tackle images that “shrink” inside the page’s responsive breakpoints, but change dimensions entirely between breakpoints, as in Figure 11-10?
We can handle that case by extending the sizes
value beyond the simple CSS length, including the various breakpoints. The syntax to implement that would be:
<img
src=
"cat.jpg"
srcset=
"cat_200w.jpg 200w, cat_400w.jpg 400w"
sizes=
"(max-width: 460px) 50vw, (max-width: 1200px) 25vw, 300px"
alt=
"A funky cat"
>
How does that work? The browser takes the entire value of the sizes
attribute, and breaks it up into pairs of a media condition and a CSS length. A media condition is very similar to a media query, only it doesn’t specify a media type and is optional. The browser goes over the media condition and length pairs, and checks if the media condition matches or is missing. If so, the CSS length is picked to be the element’s source size. The browser uses the source size in order to translate all the w
descriptors into internal density values, and then applies the same algorithm that it applies on x
descriptors, so again, it takes into account the screen density and other factors when picking the appropriate resources from srcset
.
It’s important to emphasize that sizes
is an optimization and even if you define a rough sizes
value (e.g., 100vw
), it is still in many cases better than simply sending the same image to tiny devices and retina 28-inch displays. But sizes
enables you to get as close as you’d like to the actual display dimensions, and enables the browser to pick the right image resource accordingly.
The previous srcset
descriptors all assume that all the image resources represent the “same” image and are
interchangeable, only in differing qualities and dimensions. While that’s enough to cover the fixed-dimensions and variable-dimensions use cases, when it comes to art direction, we need more control than that. We need to be able to tell the browser “download this image resource in this particular breakpoint” and be fairly confident that it will. Otherwise, layout may break and our site may become unusable.
So, how do we define the image resources for Figure 11-11?
How does that work? The important bit to understand is that even though <picture>
gets a lot of attention, <img>
is still the element that drives image loading and display. Among other things, this means that if <img>
were missing from the preceding example, nothing would be displayed. So, <img>
gets created by the parser, and then the element checks to see if it has a <picture>
parent before it starts loading an image. If a <picture>
parent is present, the <img>
element walks that parent’s <source>
children until it reaches itself, and picks the first one that matches as the source of image resources. In our case, “matches” means that
the media query inside the media
attribute matches. If no <source>
element matches, <img>
will be its own resource source.
Once a source is picked, its srcset
attribute will be used to pick the right resource, in a similar process to what we’ve seen earlier.
And while we need strict control in order to get art direction right between our responsive breakpoints, we may also need to be able to mix that with the other use cases inside the breakpoints.
That can be achieved with syntax like the following (see Figure 11-12):
<picture>
<source
media=
"(max-width: 600px)"
srcset=
"narrow_viewport_200.jpg 200w,
narrow_viewport_400.jpg 400w"
>
<img
srcset=
"wide_viewport_200.jpg 200w,
wide_viewport_400.jpg 400w"
alt=
"probably a cat"
>
</picture>
Another use case we talked about earlier is the “download and hide” case. We can resolve that using <picture>
by adding a “spacer GIF”
data URI as the source for the images that are not supposed to be there. The reason this is neccessary is that the selection algorithm skips a <source>
without any srcset
attribute. So we need our <source>
to have a valid srcset
, only with a meaningless small image.
So, if we want our image to “disappear” at viewports smaller than 600 pixels, we could do:
<picture>
<source
media=
"(max-width: 600px)"
srcset=
"data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAA
AALAAAAAABAAEAAAIBRAA7"
>
<img
srcset=
"image_only_needed_for_wide_viewports.jpg"
>
</picture>
The main reason the RICG came up with <picture>
was to tackle the art direction use case, but that’s not the only thing it is good for. Serving different image formats to browsers that support them while having a common-ground image format as a fallback is another use case it tackles.
If we look at font, video, or audio files, the web platform supports many different formats and enables client-side fallback for them right from the moment that these media types are added to the Web. Contrary to that, images always lacked such a client-side fallback mechanism. As <img>
stabilized support on the three major file formats before the first browser wars were over, there were no compatibility issues related to image format support, so no one worked on a mechanism to enable them. When new file formats such as WebP or JPEG XR were introduced, the answer to people trying to use them was content negotiation, and modifying the server’s logic so that it would return the newly supported types only to browsers that supported them.
There was one major compatibility issue related to image formats between the big browsers at the end of the first browser wars. It was the matter of PNG alpha channel support, which was lacking in IE6. Since the MIME type for transparent PNGs was no different than nontransparent ones, content negotiation did not help there, and various specific hacks were created to tackle the issue instead, until IE6’s market share became low enough for this to be a nonissue.
That is, until responsive images became a thing. While the RICG was working on new markup solutions to load responsive images, it became clear that the same constructs (with slight additions) could be used to create a client-side fallback, and help introduce new image formats to browsers that supported them, even if you had no control over the server’s logic.
How can we do that, you ask? By using the type
attribute!
<picture>
<source
type=
"image/webp"
srcset=
"rad_wolf.webp"
>
<source
type=
"image/jxr"
srcset=
"rad_wolf.jxr"
>
<img
src=
"rad_wolf.jpg"
alt=
"A rad wolf"
>
</picture>
Here again, the browser will go over the list of <source>
elements and pick the first that matches, only this time, “matches” means
that the type
attribute contains a MIME type the browser supports.
Up until now we have talked about the basic syntax, but there are a few further considerations that you probably want to take into account when addressing responsive images in your real-life project.
The first question that often comes up when discussing these features is: “do we need to polyfill it for legacy browsers?” The answer, like many things in computer science, is: “it depends.”
The features were built with an inherent fallback in mind. As <img>
must always be present in the markup, it is sufficient to add a src
attribute to it with a fallback image, and nonsupporting browsers will have something to display. In many cases, that’s good enough.
Until recently browser support for the entire set of responsive images features was not complete, and therefore if you needed art direction, for example, you had to use a polyfill, whereas if you needed fixed-width resources, you could have gone with a reasonable fallback instead.
But since then, support was added to all major browsers, and nowadays the only reason to use a polyfill is if you need to support older browsers, such as IE, and such support cannot be accomplished with a simple fallback image, since the image is art-directed.
If you do need to use a polyfill, the official and standard-compliant polyfill would be Picturefill.
Every image has intrinsic dimensions that are defined by the image file itself and composed of the image’s width and height in “physical” pixels. The browser then takes these intrinsic dimensions into account when it decides how to lay out the image. If there are no HTML- or CSS-based instructions that tell the browser what the image’s display dimensions should be, it is the intrinsic dimensions that dictate that.
However, when we’re talking about responsive images, we don’t want them to be displayed according to their “physical” intrinsic dimensions, but according to dimensions adjusted to the image’s density. That helps us get properly dimensioned and sharp images, rather than oversized blurry ones—for example, when sending 2x images to the browser.
How does the browser accomplish that? When picking image resources as part of srcset
’s selection algorithm, the browser calculates the image
density. If the image has an x
descriptor, that’s easy. The descriptor’s value is the value of the image’s density. If the image has a w
descriptor, we also need to know which dimensions the image will be displayed in. As we’ve seen earlier, that’s where the sizes
attribute
comes into play. The browser takes the sizes
attribute, figures out the CSS length that applies to the current breakpoint, and uses that
and the resource’s w
descriptor in order to conclude the image’s density. The image density is then used by the browser to compensate and
correct the “physical” dimensions when it calculates intrinsic dimensions.
OK, but why do I think that this long and complicated story is of interest here? Isn’t that the problem of the people working on the browser implementation?
Well, it has real-life implications because if you give the browser the wrong details, the outcome may surprise you. I’ve seen many examples where people
put in approximate w
descriptor values or incorrect sizes
attribute values while relying on the image’s intrinsic dimensions, and
end up surprised that the image is displayed in the wrong dimensions.
The point is, if you feed the browser the wrong data about the images you give it, you’re likely to have a bad time debugging why your images are too big or too small. So don’t.
We already talked about the ways that the browser selects the right <source>
, the right sizes
length, and the image resource in srcset
,
but it’s worth repeating in order to avoid confusion.
<source>
elements are picked using a “first-match” algorithm. The first <source>
element that matches both media
and type
(where a
missing attribute is considered as matched) is the one that gets picked.
For sizes
, it is very similar. The browser goes over the list of media conditions and CSS length pairs, and picks the first length with a
matching media condition, or one with no media condition at all. This is why we often leave the last length value in sizes
as a standalone
value, to be used as a fallback.
srcset
, on the other hand, doesn’t use a “first-match” algorithm, so the order doesn’t matter unless you have resources with the same
density. That means that as long as your descriptors are correct, you don’t need to worry about the order of the resources inside srcset
.
srcset
was designed so that its selection algorithm can evolve over time to make smarter decisions about the tradeoff between image
quality and download speeds. We want srcset
to be able to respond to many things beyond simple screen density: browser cache, bandwidth conditions, user preferences, network costs, and other
user conditions. Therefore, and since srcset
only covers cases that are not related to art direction, the browser has a great deal of
liberty when it comes to picking resources.
That’s the reason you shouldn’t rely on what you think the browser should load in different conditions when using srcset
. That can
and will differ between browsers and between browser versions. As browsers improve and get smarter, your assumptions regarding their
behavior will not remain accurate for long. So make no
assumptions and let the browser do its thing.
Since the responsive images features were recently introduced to the web platform, there is a chance that you’ll need to be able to tell if your user’s current browser supports these features or not. The classic use case for this is when creating a polyfill (which you don’t really need to do, as Picturefill is well maintained and fully supports the syntax), but there could be other occasions where you find yourself trying to figure out browser support for the responsive images features.
For these cases, you can use the presence of HTMLPictureElement
in window
to detect support for <picture>
and use the presence
of sizes
and srcset
in an HTMLImageElement
node to figure out if they are supported.
More concretely, here’s how Picturefill detects that support:
var
image
=
document
.
createElement
(
"img"
);
var
srcsetSupport
=
"srcset"
in
image
;
var
sizes
support
=
"sizes"
in
image
;
var
pictureSupport
=
!!
window
.
HTMLPictureElement
;
One more tool you can use when working with responsive images is the currentSrc
property on HTMLImageElement
, which enables you to see
which resource is currently loaded and displayed on a specific <img>
element.
You can use this if, say, your JS interaction with an art-directed image should differ in case a different resource is picked.
Up until now, we have discussed the various markup solutions for delivering responsive images. But that’s not the only type of solution we have up our metaphorical sleeves. For some cases, it might be significantly easier to modify a server configuration than to modify HTML. For these cases, content negotiation could be a better option than markup.
Content negotiation is an HTTP-based mechanism in which the client (in our case, the browser) sends HTTP headers indicating its support or
preference regarding the content, and the server responds with the desired content. There are multiple examples for that method in the HTTP
protocol, the most prevalent of which is the Accept-Encoding
request header, to which the server couples a Content-Encoding
response header,
indicating if Gzip or any other encoding method was applied to the returned response.
To offer a content negotiation–based responsive images solution, a new set of HTTP request and response headers was created, under the banner of “Client Hints.” We will discuss those further in Chapter 12.
We have definitely made a lot of progress in recent years to tackle the use cases of responsive images, but everything related to software is rarely “done.” In this section we will discuss some potential future improvements.
In the early days of responsive images, the subject of background images wasn’t considered important, and the focus was on content images. After all, background images could be controlled through media queries, and on top of that, WebKit-based browsers (so,
Safari and Chrome) supported the -webkit-image-set
CSS property, which enables the browser to load based on device pixel ratio (DPR). That was considered a
handy shortcut to spelling out separate rules based on the resolution
or -webkit-device-pixel-ratio
media queries.
So we continued to resolve content images, leaving background images as they were. Only now, with the hindsight of the solutions for content images, do we see the deficiencies that still need to be resolved in background images.
So, let’s take a look at how each one of the use cases can be resolved for background images.
The fixed-width images can be resolved with a fairly simple markup (which resembles, and in fact inspired, the markup for srcset
’s x
descriptor):
.fixed-dimensions-image
{
background-image
:
-
webkit
-
image
-
set
(
url(1x.jpg)
1
x
,
url(2x.jpg)
2
x
);
}
The standard form of that is the unprefixed image-set
. Unfortunately, that is not implemented anywhere at the time of this writing.
The art direction use case is easy to solve with media queries:
.art-directed-image
{
background-image
:
url(narrow.jpg)
;
}
@media
screen
and
(
min-width
:
800px
)
{
.art-directed-image
{
background-image
:
url(wide.jpg)
;
}
}
But there’s no way today to define a background image that loads an efficient, variable-width image. One could imagine an extension to
image-set
that includes something like the w
descriptor srcset
has, but that’s not yet specified or implemented.
You may have noticed that when discussing the use cases, we’ve used the term variable dimensions, yet the only resource descriptor we have for
this case is the w
descriptor, describing the resource’s width. Images are two-dimensional! That’s not fair!
While we were working on the responsive images solutions, we noticed the same injustice, yet the major use case to tackle was solving width-constrained layouts. We had significantly fewer examples for height-constrained layouts, so we preferred to wait with that use case until there’s more experience with it “in the wild.”
Nevertheless, the processing algorithms take the future existance of an h
descriptor into account, and ensure that the introduction
of such a descriptor will go over smoothly.
After having the basic set of features out there, with developers using them in production, we now see some demand for height-constrained
layouts, mainly for image galleries. So, hopefully work on that front can continue and h
descriptors will eventually be part of the
srcset
.
When we talk about responsive images, the question “why not solve it using a file format?” often comes up.
While solving the responsive images problem using a file format is certainly feasible (at least for some of the use cases), there are some caveats. The browser would have to download a first chunk of the image in order to know its dimensions and the fitting byte range for the image dimensions and breakpoints it needs. That would require coming up with a loading scheme where a few initial bytes are downloaded from all images in order for the browser to know what ranges need downloading. In HTTP/1, that will most likely result in performance regressions, as there’s a limit to the number of resources that can be fetched in parallel. In HTTP/2 that is less of an issue, but would still be less then ideal for the first images, especially if they don’t end up being of a responsive format.
With that being said, there have been attempts to create formats that may fit the “responsive image format” label. Although none of them is of practical value today, the curious among you may find these attempts interesting.
As we’ve seen earlier in the book, progressive JPEG is, well, progressive. The browser can decode it as it is being downloaded, and the result is a full image, with its details filling in as more chunks of image data are downloaded.
Therefore, we could emulate a lower-resolution image (or a smaller image) by truncating a high-resolution, large image and scaling it appropriately. Assuming we have multiple JPEG scans, we could use something like SSIM to determine the appropriate size and resolution for each one of the scans, and then communicate that information to the browser (e.g., using special JPEG markers at the start of the file), and have it download only the scans that it needs.
Such a concept has been experimented with in the past. It seems like something that might work for the fixed- and variable-dimensions cases, but not for the art direction use case.
Additionally, from experimentation, if the images that you’re trying to serve need to fit both a very small space (e.g., on low-end devices) and a very large one (e.g., retina 28-inch screens), quality will suffer or bytes will be wasted. There’s a limit to the range of quality that can be communicated using progressive JPEG scans.
JPEG 2000 (which we discussed in Chapter 5) is a progressive format by nature, and therefore, at least in theory, could be an ideal candidate for progressive loading.
Unfortunately, previous experiments that were conducted on that front proved it to be less promising in practice.
There have also been attempts (by yours truly) to create a responsive image container—an image-format-agnostic container that would encode the image using different layers, where the first layer is a thumbnail of the image, and each consecutive layer adds more information to the image, enabling it to target higher-resolution screens, larger display dimensions, and even crop-based art direction. The intent behind creating a container rather than a full-fledged new format was to avoid patent and political issues often surrounding file formats, likely increasing its chances for adoption.
More recently, a lossless progressive file format named FLIF was introduced. The name stands for Free Lossless Image Format, and it shows very good results when compared to other lossless formats.
One of its most touted advantages is its progressive nature, which could make it a candidate for a responsive format. However, it’s still very early days for the format, so it’s hard to be certain of the direction in which it will evolve. Its lack of a true lossy mode makes it less applicable to real-life imagery than other formats.