13.2.1 Physical Characterization

A conceptual illustration of an LCD panel is given in Fig. 13.1. The main components are the backlight, diffuser plate spreading the light smoothly on the display area, and a grid of liquid crystals, forming the pixels of the displayed image (Anandan, 2008). In color displays, each pixel consists of separate subpixels for red, green, and blue color channels, respectively. In conventional LCDs, backlight and the diffuser have been designed to provide uniform distribution of light across the display. In display architectures supporting local backlight dimming, backlight is divided into segments that can be controlled independently. Local dimming allows the use of lower illumination levels in regions where bright content is not present. The basic backlight model is introduced below and more details about local dimming will be given in Section 13.3.

Since liquid crystals are essentially voltage-driven light filters, we can model the relative luminance l(i,j) at pixel position (i,j) as the product of the normalized backlight intensity b(i,j) at the pixel position and the normalized transmittance t(i,j) of the liquid crystal:

$\begin{array}{l}\hfill l(i,j)=b(i,j)\cdot t(i,j).\end{array}$

In Eq. (13.1), we assume that all the values are normalized to the interval [0,1], where 0 represents the minimum value (backlight is turned off or the liquid crystal blocks all the light) and 1 represents the maximum value (backlight has its full intensity and the liquid crystal is set to its maximum transparency level, ie, as little light is blocked as possible). The maximum value of the backlight may later be chosen for a specific HDR display.

Unfortunately, light leakage is usually observed in practical liquid crystals, even if the control signal of the liquid crystal is set to zero. This is why intended black pixels in LCDs do not look entirely black, but look slightly grayish or blueish. To take light leakage into account in the pixel luminance model, we can define a leakage factor ε, expressing the proportion of light that will pass through a pixel that is intended to be black. Including ε, we can redefine the relative pixel luminance as

$\begin{array}{l}\hfill l(i,j)=b(i,j)\cdot t(i,j)+\epsilon \cdot b(i,j)\cdot \left(1-t(i,j)\right).\end{array}$

Because of the physical characteristics of liquid crystals, light leakage is dependent on the viewing angle; leakage is less pronounced straight in front of the display, in comparison with a tilted viewing angle (Burini et al., 2013a). In addition, ε may vary slightly at different pixel positions and for different color components. For a fully accurate leakage model, ε should be redefined separately for red, green, and blue subpixels (ε_r, ε_g, and ε_b) as a function of pixel position and viewing angle φ. However, such a model would be highly specific to a certain physical display, and for the sake of simplicity, will not be presented here. Later we discuss the effect of the viewing angle in terms of letting ε be dependent on the viewing angle.

If the backlight is uniform, a constant value for b(i,j) can be used in Eq. (13.2) to derive the observed pixel luminance. If the backlight is at full intensity, b(i,j) = 1. However, practical backlights are not accurately uniform, and in local dimming, displays deviating from uniformity are even desired to increase the contrast (eg, for HDR images). For accurate backlight intensity modeling, we need to know the point spread function (PSF) of each backlight segment. Basically, the PSF is defined separately for each backlight segment, and it expresses the backlight intensity at different positions when the backlight element in question is turned to maximum intensity. Physical measurements of luminance–that is, the luminous intensity of a surface (cd/m²) — are typically required to define the PSF accurately for any real-life display. However, a Gaussian function, for example, may be used for rough approximation of symmetric backlight segments. In the following, we denote the PSF as h_k(i,j), expressing the resulting backlight intensity at position (i,j) when backlight segment k alone is set to full power.

In practical LCDs, backlight segments are usually not optically separated from each other. This is why backlight segments overlap: at each pixel position, the total backlight intensity is contributed by several backlight elements. Each backlight element contributes to the intensity of a large number of pixels, beyond the backlight segment boundaries. A simple additive model can be used to compute the backlight intensities in different positions as a sum of individual contributions from different backlight segments:

$\begin{array}{l}\hfill b(i,j)=\sum _{k=1,\dots ,N}{B}_k{h}_k(i,j),\end{array}$

where B_k denotes the normalized backlight intensity for segment k, and N denotes the total number of backlight segments.

13.2.2 Display Architectures

Dimming of the backlight may be used to increase the dynamic range. This is always the case seen over time, but the effects on the contrast within an image depend on the display architecture.

In small displays, such as those for smartphones or small tablet computers, the backlight typically consists of only one LED, located in one of the corners. A light guide (diffuser) with special characteristics is used to diffuse light uniformly on the screen. Also, large conventional LCDs, especially those in the lower price range without HDR capability, usually have only one backlight segment for the whole display. Adaptive backlight dimming may also be used in displays with only a single backlight segment, referred to as global backlight dimming or 0D dimming. Here dimming will only be effective over time, and there are no local dimming effects.

Displays with several backlight segments can be categorized according to the configuration of the segments. In direct-lit LCDs, LEDs are located behind the diffuser, forming a 2D structure of backlight segments. In displays of this type, local backlight dimming is referred to as 2D dimming. Backlight segments can be allocated in different layouts; for example, SIM2 HDR display uses a hexagonal shape for 2202 controllable backlight segments providing a peak luminance of 4000 cd/m² (SIM2, 2014). Another possibility is to allocate LEDs at edges, forming either horizontally or vertically directed backlight segments for 1D dimming. Edge-lit design is especially popular in flat panel televisions, since edge-lit LCD panels can be built thinner than direct-lit panels. One can arrange edge-lit segments in two columns by allocating separately adjusted LEDs on both sides of the screen. This kind of backlight dimming architecture is called 1.5D dimming as it offers a compromise between 1D and 2D dimming. Different display architectures for backlight dimming are compared in Fig. 13.2. The 2D dimming architecture is recommended for HDR material. In all cases, the PSF will also play a role in the contrast as a function of the distance within the image.

In addition to conventional backlight displays relying on white light sources, it is also possible to use a combination of red, green, and blue LEDs as backlight elements. This kind of architecture allows local backlight dimming to be performed separately for each color channel, termed 3D dimming. The benefit of 3D dimming is apparent only on pictures where one of the color channels is highly dominating, and 3D dimming has not gained wide popularity in practical backlight-dimming LCDs.

13.2.3 Evaluating Distortion

Since the backlight resolution is lower than the display resolution, it is usually not possible to concurrently eliminate light leakage in dark pixels and provide full backlight for bright pixels when they are located close to each other, which leads to nonuniform backlight and halo effects (Chen et al., 2012). To achieve the best possible trade-off between leakage in dark pixels and clipping the luminance of bright pixels, we first need to find a method to evaluate the distortion caused by leakage and clipping. Assuming that the backlight intensities and PSFs for each segment are known, Eq. (13.3) can be used to compute the backlight luminance at each pixel position.

Since the transmittance levels of liquid crystal cells are set by the image signal fed to the liquid crystal array, the intensities of the red, green, and blue components of each pixel can be solved with Eq. (13.2). In practical displays using adaptive backlight dimming, the original image is often processed to compensate the relative loss of brightness in regions with dark backlight. This process is often referred to as brightness preservation, pixel enhancement, or liquid crystal pixel compensation. Uniformity (or absolute consistency) is best preserved if the backlight model is used to assist in brightness preservation. For the original image we assume that it has been mapped to the range of the display. Thus for an HDR image defined outside the range of the (HDR) display, we assume a tone mapping of the source image has been applied to form the original target image we will try to render on the display.

The computations in the model so far, Eqs. (13.1)–(13.3) are based on relative physical luminance values, and the relationship between physical luminance and perceived lightness (luma) is not linear (Cheng and Chao, 2006). Since images in digital signal processing systems are usually represented in a color space, such as sRGB, where each color component follows a perceptually uniform scale, conversion from the perceptual domain to the physical domain is necessary before transmittance values are used in the model. In displays with peak luminance lower than 100 cd/m², a conventional gamma function with γ = 2.2 can be used to approximate the relationship between physical luminance and perceptual luma. However, for HDR displays with higher peak luminance, a brightness adaptive conversion function would be preferred for more accurate results. Approximating the data from Aydın et al. (2008), we can define conversion functions from physical luminance L to perceptual uniform luma P and vice versa defined as (Korhonen et al., 2011)

$\begin{array}{ll}\hfill P(L)& =ln(0.56\cdot L^{0.88}+1)/ln(0.56\cdot L_{max}^{0.88}+1),\hfill \end{array}$

$\begin{array}{ll}\hfill L(P)& ={\left[\left(exp\left(P\cdot ln(0.56\cdot L_{max}^{0.88}+1)\right)-1\right)/0.56\right]}^{1.14}.\hfill \end{array}$

In Eqs. (13.4) and (13.5), perceptual luma P is normalized between 0 and 1, so the maximum physical luminance $L_{max}$ (cd/m²) corresponds with the maximum perceived luma, P = 1.

When the relative physical red, green, and blue luminance values are solved by a model, the values can be converted back to the perceptual luma domain and compared against the original target image to evaluate clipping and leakage distortion. Basically, any conventional full-reference image quality metric can be used for this purpose, including simple mean squared error (MSE) and peak signal to noise ratio (PSNR), as well as more sophisticated models, such as the structural similarity index (Wang et al., 2004) and the HDR visible difference predictor (HDR-VDP) (Mantiuk et al., 2005). A block diagram of the objective quality evaluation process is shown in Fig. 13.3.

13.3.1 Local Dimming of LED Backlight

In Section 13.2, the leakage effect due to the liquid crystal elements was introduced. This deficiency may be attenuated by application of local dimming of the backlight segments depending on the image content. Albrecht et al. (2010) approached this by minimizing the leakage (or power consumption) under the constraint of not introducing clipping (ie, a clipper-free reproduction of the image). From a perceptual point of view, we may seek a better visual result with a higher (local) contrast by additionally attenuating the LED values to further reduce the leakage at the expense of introducing clipping.

A number of algorithms for local dimming have been presented and they may be characterized on the basis of what image statistics they use (eg, simple image statistics as the average value of the backlight segment, either directly or taking the square root). This may be generalized to algorithms collecting one or more image histograms and basing the dimming solely on this information. A more complex class of algorithms is based on calculation and evaluation of the displayed pixel values based on modeling the display process. The clipper-free approach (Albrecht et al., 2010) is one example. We will present a class of optimization-based dimming algorithms using a model of the displayed image.

13.3.2 Liquid Crystal Pixel Compensation

Given a set of LED values defining the modeled backlight, b = b(i,j)(13.3), there is a choice for adjustment of liquid crystal values for all pixels. We will consider simple pixelwise compensation of liquid crystal values with the property of minimizing the square (or absolute) error. When the backlight is dimmed such that the target value, l_y, is not achieved, even when the liquid crystal element is fully opened, the liquid crystal value is saturated at the maximum value. This is called hard clipping. Conversely, when the leakage contribution exceeds that of the target pixel value, the liquid crystal value is set to 0. This may be expressed for target value l_y for a pixel by

$\begin{array}{l}\hfill l=\left\{\begin{array}{ll}\epsilon b,& \text{if}\epsilon b>l_y,\\ b,& \text{if}b<l_y,\\ l_y,& \text{otherwise}.\end{array}\right.\end{array}$

Better visual solutions may be obtained by application of, for example, soft-clipping, but we consider the simple solution as a mathematically tractable approach providing a good initial solution (eg, toward determining the backlight values).

13.3.3 Distortion of Backlight Images and Optimizing the Backlight Values

On the basis of the modeled displayed image, l, we can calculate an overall distortion. Having a distortion measure, we can straightforwardly pose an optimization formulation given by minimizing the overall distortion. Besides potentially improving the contrast, another important reason for dimming the backlight is to reduce energy consumption as also reflected in energy norms and labels for television sets today or to limit power dissipation in HDR displays.

The clipper-free solution precisely ensures no clipping but ignores leakage and as such does not involve a distortion measure. A first extension is to consider the distortion of luminance values directly (Shu et al., 2013),

$\begin{array}{l}\hfill \text{minimize}||l-l_y||,\end{array}$

subject to Eq. (13.3), Eq. (13.6), and 0 ≤ B_k ≤ 1.

As shown in Shu et al. (2013), this can be rewritten as a convex problem. The argument is based on for each pixel our considering each of the cases in Eq. (13.6) (ie, leakage, clipping, and cases with no distortion) and noting that these all provide a linear lower bound on the contribution to distortion of all elements. Thus, for distortion by absolute errors, this optimization problem is linear and for the MSE it is a quadratic problem.

To evaluate and include the power consumption, we note that the variable part of the display power consumed by the backlight may be expressed by a simple sum of LED values. The reason is that the LEDs are time modulated, being turned on in a fraction of the give time slot proportional to the desired value. The LED contribution to the power consumption, p, is thus expressed by

$\begin{array}{l}\hfill p=\frac{1}{N}\sum _{k=1}^N{B}_k.\end{array}$

The basic contribution to the power consumption of the platform may be modeled simply as a constant independent of the image content and will not be evaluated or part of the optimization process. Below we will combine distortion and power consumption in one cost function.

Considering the (visual) distortion in Eq. (13.7) directly based on luminance values has the limitation that, as mentioned, the luminance errors do not accurately reflect the visual impression. For this purpose a nonlinear mapping of the errors may be introduced. We consider a nonlinear mapping, f, of luminance, l, into a luma value ideally forming a perceptual uniform domain (13.4)–(13.5). As the gamma function is often used in relation to displays, we choose this as a simple example of a nonlinear mapping — that is, displayed pixel is x = f(l) = l^1/γ. Likewise, we define the target $y=f(l_y)=l_y^{1/\gamma }$ (Burini et al., 2013a, b). The corresponding full-resolution display and target images are denoted x and y, respectively. The distortion and energy consumption may be combined in one cost function with use of a Lagrangian variable, q, to adjust the relative weighting of the two terms. Furthermore. addition of a weight w provides the flexibility to adjust the importance of each distorted pixel. This gives the general optimization problem:

$\begin{array}{l}\hfill \text{minimize}||(\mathbf{y}-\mathbf{x})\mathbf{w}||+q\times p,\end{array}$

where the weight vector w default is set to unity and the MSE (norm 2) is chosen as image distortion. The power, p, is given by Eq. (13.8).

The displayed image, x, may further include the effects of quantization of LED values (13.3) and liquid crystal values (13.1). This has a marginal effect when one is deciding on LED values, but to calculate representative distortion numbers the quantization effect should be considered.

A suggestion for mapping luminance values into a perceptual uniform presentation was given in Eq. (13.4). We may also incorporate this in the distortion (13.9) either by changing the mapping f or by using the weight function, w, if the absolute values on the display are known. On a similar note, w may also be used to adjust the relative importance of clipping and leakage or more generally prioritize a selected range of values or spatial regions of interest. The weight function may also be used in connection with tone-mapped HDR images.

Even if the focus is on quality, with little or no concern for the energy, it still makes sense to include the energy term, p, but with a small weight, q. The reason is that this will stabilize the solution to Eq. (13.9) by introducing a regularizing term. There may be (parts of) images with little or no highlight and dark areas and thus a large space of solutions for the LED values will provide (close to) optimal solutions. Even a small power term, q, will reduce the solution space significantly and stabilize the algorithm. Increasing q further will reduce LED values and visually the leakage will be further attenuated at the expense of increasing clipping.

Eq. (13.9) applies equally to direct backlight (2D) and edge-lit (1.5 D or even 1D) displays as the backlight is in these cases described by Eq. (13.3). To solve the minimization (13.9) on, for example, full-high-definition-resolution images, a gradient descent approach was used by Burini et al. (2014). For further speedup, histogram statistics were first collected within blocks of a fixed size in the block-based gradient descent approach.

The formulas above are defined for the luminance and luma or black/white images. We may generalize them to color images given by three color components in a number of ways. We may simply take ||y −x|| as the color pixel by pixel distance in the 3D color space of choice. For RGB image representations two alternatives were presented in Burini et al. (2013a, b): a weighted sum of the red, green, and blue contributions or selection of the color component with the minimum value for leakage and the maximum value for clipping for each distorted pixel. Because of the large number of pixels per LED segment, the distortion function used in optimization may be simpler than the ideal distortion function to be used in evaluations. Likewise on the processing side, soft clipping may be applied as part of the pixel compensation, even if hard clipping is applied in the optimization of backlight elements. In Fig. 13.4, the performance for quality versus power is depicted for 48 images from the IEC database (Mantel et al., 2013b). A gradient descent-based version of Eq. (13.9) as described in Burini et al. (2014) was used and compared, among other methods, with the method in Albrecht et al. (2010) as well as with simple methods based on maximum, average, or square root of average pixel values (Mantel et al., 2013b).

Above we focused on global formulations of distortion. This may be extended to more local formulations, truly treating x and y as the whole image representations and not just summing contributions pixel by pixel. This extension may include adaptive tone mapping, local contrast, backlight artifacts as halos, etc. These issues are treated in other chapters in this book. To encompass this in the optimization in Eq. (13.9), the weight vector may be formulated as a function of the target image, w(y).

The approach above may be used image by image for image sequences as in video. However, application of full image-by-image optimization may well prove to be too aggressive, when the temporal dimension is included, and lead to flicker. Thus, extension from images by addition of the temporal dimension is nontrivial. The temporal dimension will be addressed in the next section. Restrictions on how much a given LED segment can change from frame to frame provide one approach to control flicker in local backlight dimming (Burini et al., 2014). For quality display of video material, flicker must be attenuated to an acceptable level. Once this has been solved, other issues over time may be to constrain power and variations in quality. A solution for this was presented in Mantel et al. (2013b) to control the quality-power trade-off by adjustment of the weight in Eq. (13.9).

13.4.1 Liquid Crystal Pixel Compensation

In 2D LCD systems as described, an image is displayed by attenuation of the light emitted by the backlight source with a thin layer of liquid crystal filter array (13.1). For simplicity, here we will use a single variable i to index pixels in the displayed image, and its intensity value is determined by the transmittance τ_i(t) of the corresponding unit of the liquid crystal array and the backlight luminance L_i(t) in the time duration T of a frame. Here we emphasize the time-variant properties of both τ_i(t) and L_i(t). The former is due to transitional behavior of liquid crystal molecules; the latter is required by backlight modulation. To present stereoscopic vision, images for the left eye and the right eye are displayed alternately on a screen, and the liquid crystal shutter glasses (LCG) worn by the viewer select the correct view for each intended eye by rapidly changing their transmittance g(t) accordingly. The (average) luminance of pixel i in stereoscopic display can be modeled as

$\begin{array}{l}\hfill I_i=\frac{1}{T}{\int }_0^T{L}_i(t)\cdot {\tau }_i(t)\cdot g(t)dt.\end{array}$

Without loss of generality, we discuss only one monocular image, the left one, in stereoscopic display in the following discussion. The condition for the image for the other eye can be derived similarly. Without g(t), the model may be used to describe the temporal aspects of backlight dimming for monoview video, extending the time-discrete version in Eq. (13.1).

Design of stereoscopic vision systems usually adopts the following assumptions:

1. The liquid crystal transmittance of pixel i changes to the target level x_i instantly when pixel i is updated, and it stays the same until pixel i receives a new value.

2. The light transmittances of shutter glasses also switch instantly between a constant $\mathcal{G}$ and 0, corresponding to the open and closed states, respectively.

3. The backlight luminance is a positive constant $\mathcal{L}$ when the backlight is on, or 0 when it is off.

In view of the above assumptions, Eq. (13.10) can be approximated as

$\begin{array}{l}\hfill I_i\approx \frac{\Phi ({\tilde{L}}_i\cap \tilde{g})}{T}\cdot \mathcal{L}\cdot x_i\cdot \mathcal{G},\end{array}$

where ${\tilde{L}}_i$ is the time interval of the backlight at pixel i being on, and $\tilde{g}$ is the LCG open interval. Their intersection, ${\tilde{L}}_i\cap \tilde{g}$, is the time interval when pixel i is visible to viewer. The function $\Phi (\tilde{t})$, representing the duration of an interval $\tilde{t}$, is defined as

$\begin{array}{l}\hfill \Phi (\tilde{t})=b-a\text{if}\tilde{t}=\left[a,b\right],\end{array}$

where a and b are the start and end times of $\tilde{t}$, respectively.

In addition, to ensure that pixel i of one monocular image is displayed correctly without any crosstalk, the backlight panel or shutter glasses can both be on only when the pixel has finished the state transition — that is,

$\begin{array}{l}\hfill {\tilde{L}}_i\cap \tilde{g}\subseteq {\tilde{\tau }}_i,\end{array}$

where ${\tilde{\tau }}_i$ is the time interval when pixel i is stabilized at the new target level. In a conventional stereoscopic LCD with a constantly-on backlight panel, the visible time interval of a pixel is entirely decided by the LCG open interval $\tilde{g}$; thus, $\tilde{g}$ must be contained within a short interval when every pixel is stable,

$\begin{array}{l}\hfill \tilde{g}\subseteq \bigcap _{i=1}^N{\tilde{\tau }}_i,\end{array}$

where N is the number of pixels. For some LCDs, however, there is not a moment at which all pixels are stable at the same time; therefore, the LCG open interval $\tilde{g}$ that makes the display crosstalk-free might not even exist.

For LED-lit LCDs, the accuracy of assumption 3 is acceptable, as the alternation of LED backlight is several orders faster than what is noticeable by the human visual system. However, although the other two assumptions greatly simplify the design and analysis of stereoscopic display systems, they are excessively strong and do not accurately reflect the characteristics of actual hardware. The factors ignored in the approximation, such as the slow response of liquid crystals, are the major causes of crosstalk and low brightness in stereoscopic display. For example, in Fig. 13.5, the more realistic liquid crystal response curves of pixels and LCG show little resemblance to square waves as assumptions 1 and 2 suggest (Tourancheau et al., 2012).

Contrary to assumption 1, which presumes liquid crystal pixels are able to switch to a new transmittance instantly, the actual transition time T_T from a pixel receiving a new value to becoming stable is nonnegligible as shown in Fig. 13.6. Besides the considerable transition time, another practical consideration adding to the complexity of the LCD timing scheme is that pixels at various positions start transmittance transitions at very different times. Normally, pixels receive and start changing to new values line by line from top to bottom, and the offset T_P between the starting times of the first and last pixels can be as long as T/3.

Since the transmittance transition time and pixel refreshing time are not insignificant, if the sampling window $\tilde{g}$ is too wide, some pixels are still at the early stage of transition. Thus, the information intended for the other eye may leak into the current view, causing stereoscopic crosstalk. For example, in Fig. 13.5, since the sampling window starts when the bottom pixel is still in transition, crosstalk will be visible at that pixel.

On the other hand, to avoid excessive crosstalk, the sampling window $\tilde{g}$ needs to be narrow enough to ensure that most pixels have reached or are close enough to the target level in the window. By Eq. (13.13), the duration $\Phi (\tilde{g})$ of the sampling window $\tilde{g}$ in Fig. 13.5 must be as short as 2 ms in order to eliminate the appearance of crosstalk in a typical LCD. However, with such a small LCG opening window, only a fraction of the light emitted from the backlight panel could reach to viewer’s eyes, and much of the luminance is traded off for crosstalk reduction. To mathematically measure the loss of luminance caused by backlight control and LCG, we can use the image-independent mean luminance, which is defined as

$\begin{array}{l}\hfill E=\frac{1}{N}\sum _{i=1}^N\frac{I_i}{x_i}.\end{array}$

By substituting for I_i using Eq. (13.11), we can approximate E as

$\begin{array}{l}\hfill E\approx \frac{\mathcal{L}\cdot \mathcal{G}}{N\cdot T}\sum _{i=1}^N\Phi ({\tilde{L}}_i\cap {\tilde{g}}_i),\end{array}$

because ${\tilde{L}}_i\cap \tilde{g}\subseteq {\tilde{\tau }}_i$, E has an upper bound,

$\begin{array}{l}\hfill E\le \frac{\mathcal{L}\cdot \mathcal{G}}{N\cdot T}\sum _{i=1}^N\Phi (\tilde{g})=\frac{\mathcal{L}\cdot \mathcal{G}\cdot \Phi (\tilde{g})}{T}.\end{array}$

For a typical LCD, the duration T of a frame is 1/60 s and the transmittance $\mathcal{G}$ of a fully opened shutter glass is less than 0.5 because of the polarizing filter. If the sampling window $\tilde{g}$ is set to only 2 ms to combat crosstalk as shown in Fig. 13.5, then the image-independent mean luminance E is less than $0.06\mathcal{L}$ as introduced in assumption 3. This means that at most 6% of the light from the backlight panel is visible to the viewer.

13.4.2 Formulation of Optimal Backlight Modulation

As discussed previously, crosstalk reduction by adjustment of the LCG open interval $\tilde{g}$ might not be feasible in a conventional LCD with a uniform backlight panel. Furthermore, this method also greatly reduces the output brightness, causing a worse problem as viewers might prefer good brightness and contrast. Therefore, trading brightness for crosstalk reduction in conventional LCD systems is not always beneficial to the perceptual visual quality.

However, with LED local backlight systems, crosstalk reduction is possible without sacrificing much of the brightness. Turning on an LED backlight segment makes the pixels within the segment visible to the viewer, while at the same time, a block of pixels can be hidden from the viewer by turning off the corresponding backlight segment. Thus, if we only turn on the backlight of a segment when each pixel within the segment is stable, then the transmittance transitions of liquid crystal pixels are invisible to the viewer, hence eliminating the crosstalk. Because compared with LCG, backlight modulation provides finer control of the visibility of pixels, the same idea of adjusting LCG for crosstalk reduction can also be implemented with only backlight modulation.

Use of backlight modulation instead of LCG does not fundamentally change the idea of crosstalk reduction by hiding pixels from the viewer during their transmittance transitions. Inevitably, the reduced visibility, via the turning off of either LCG or backlight segments, affects the output brightness and deteriorates the perceptual visual quality. To find a better balance between crosstalk and brightness, we formulate crosstalk reduction as an optimization problem maximizing brightness while keeping the crosstalk below a given level as follows:

$\begin{array}{l}\hfill \begin{array}{ll}\text{maximize}& E\\ \text{subject to}& \Phi ({\tilde{L}}_i\cap {\tilde{g}}_i)=l,1\le i\le N,\\ & {\tilde{L}}_i\cap \tilde{g}\subseteq {\tilde{\tau }}_i,1\le i\le N.\\ \end{array}\end{array}$

The objective function of this optimization problem, the image-independent mean luminance E of the display system as defined in Eq. (13.15), measures the perceived output brightness. The first constraint is to guarantee the uniformity of the output. It requires the amount of backlight emitted for each pixel to be identical over the entire display. With this constraint, the approximation of E in Eq. (13.16) is given by

$\begin{array}{l}\hfill E\approx \frac{\mathcal{L}\cdot \mathcal{G}}{N\cdot T}\sum _{i=1}^{N}l=\frac{\mathcal{L}\cdot \mathcal{G}}{T}\cdot l,\end{array}$

as the backlight luminance $\mathcal{L}$, LCG transmittance $\mathcal{G}$, and frame duration T are constant in the optimization problem, maximizing the objective function E is equivalent to maximizing l.

The second constraint of the optimization problem requires a pixel to be shown only when it is stable as in Eq. (13.13). Before a liquid crystal pixel finishes the transmittance transition to a new level x_i, its transmittance τ_i(t) is still affected by the previous level y_i intended for the other eye. The amount of crosstalk (ie, distortion caused by the previous level) d_i(t) over time t can be quantified as follows (Tourancheau et al., 2012):

$\begin{array}{l}\hfill d_i(t)=\frac{x_i-{\tau }_i(t)}{x_i-y_i}.\end{array}$

Because of the slow response of an LCD, it takes a long time for a pixel to become completely stable at the new level, as shown in Fig. 13.5, making the absolute crosstalk-free time interval very short. Thus, to maintain a reasonable level of display luminance, the crosstalk constraint should be relaxed to allow a pixel to be shown as long as the crosstalk d_i(t) is below a threshold D. This relaxation can be achieved by our redefining the stable time interval ${\tilde{\tau }}_i$ of a pixel as

$\begin{array}{l}\hfill {\tilde{\tau }}_i=\{t|d_i(t)\le D\}.\end{array}$

Because in a typical LCD as in Fig. 13.5 the crosstalk d_i(t) decreases rapidly to a low level at the beginning of the transmittance transition, only a small crosstalk allowance D is necessary to make the duration of the stable interval ${\tilde{\tau }}_i$ long enough. The LCD response function τ_i(t) can be acquired from the measurement of a real LCD system or from a mathematical model (Adam et al., 2007), but either way it is constant in the optimization problem; hence, the stable interval ${\tilde{\tau }}_i$ is also constant with a given crosstalk threshold D.

For the case of an LED local backlight panel, since pixels within a backlight segment are illuminated by the same LED unit, the time interval ${\tilde{L}}_i$ when the backlight is on is identical for all these pixels. In other words, if pixels i and j are in set S_k of the pixels in backlight segment k, then ${\tilde{L}}_i={\tilde{L}}_j={\tilde{B}}_k$, where ${\tilde{B}}_k$ is the time interval when backlight segment k is on.

Combining all the above points, we finally arrive at a new formulation of the crosstalk reduction problem:

$\begin{array}{l}\hfill \begin{array}{ll}\text{maximize}& l\\ \text{subject to}& \Phi ({\tilde{L}}_i\cap \tilde{g})=l,1\le i\le N,\\ & {\tilde{L}}_i\cap \tilde{g}\subseteq \{t|d_i(t)\le D\},1\le i\le N,\\ & {\tilde{L}}_i={\tilde{B}}_k,1\le i\le N,i\in S_k,\\ \end{array}\end{array}$

where l, $\tilde{g}$, ${\tilde{L}}_i$, and ${\tilde{B}}_k$ are variables.

13.5.1 Necessity of a Display Model for Evaluation of Local Backlight Dimming

There are two ways to model an HDR display: the first one is to characterize the rendering on the display, the way images are reproduced, and the second one is to include it in an image quality assessment chain. Whereas the first indisputably requires a display model, the second has not been thoroughly investigated in the quality assessment literature.

The first way, for example, plays a role when one is displaying HDR images as was done in Mantel et al. (2014) on a SIM2 HDR display, which applies local backlight dimming. For HDR images, tone mapping is generally applied. In our experiment, just a simple scaling was applied when the maximum image value exceeded the peak white value (which was set to 3000 cd/m²). The rendering was done with the full capacity of the 2202 segments of the display and the MSE-optimal backlight computed with the approach described in Section 13.3 (with the PSF provided by the manufacturer) by means of the gradient descent-based version in Burini et al. (2014). Table 13.1 gives the error of rendering in terms of the signal-to-noise ratio (SNR) and the mean relative squared error (MRSE) (Richter, 2009) due to the display limitations (the SNR is calculated in the gamma-corrected domain) evaluated by the display model expressed in Eq. (13.23) to reflect the actual display of the images. The values given here are for perfect-quality input images and therefore the error stems only from the rendering (ie, clipping, leakage, and quantization). As the rendering is spatially dependent on the image content, the error varies significantly from one image to another. It reflects the rendering complexity specifically for each image and shows that HDR inputs can have very different characteristics: some are easy to render, while others “stress” the renderer as also shown by Narwaria et al. (2014) for tone mapping. The SIM2 HDR display used in the experiment can provide locally a peak white value of 4000 cd/m², but for the very high values of peak white the actual display may deviate from the model, and hence we chose to operate the display at the lower peak white value of 3000 cd/m².

The second way addresses the broader question of the use of display modeling in quality assessment. This was addressed, for example, by Huang et al. (2012), who applied a display model, which includes gamma correction and ambient light reflection, before applying objective quality metrics. They show that use of a display model significantly improved the performance of quality metrics on the database tested.

Each step in the signal chain plays a role. When the quality of the displayed image or video is low, the influence of a display model is less crucial as the error the display introduces can be masked by other artifacts present. However, for settings where the input is an HDR image or just of high quality, the perceived level of errors introduced by the display may become higher than that of other defects. When small-amplitude artifacts or errors are not detected visually, it may remain an open question whether they are not perceivable or whether they are hidden by the rendering errors.

In the case of local backlight dimming, for example, display modeling for quality assessment is crucial as the rendering can be the sole processing applied to the target image. Subjective studies done on images (Mantel et al., 2013a) and highly contrasted videos (Mantel et al., 2015) of original quality displayed on a screen with various local backlight dimming algorithms have shown that observers perceive the difference between the different renderings. Thus, modeling of the dimming algorithms and resulting defects is necessary.

Traditional metrics have been designed to evaluate different types of distortions, such as compression noise and transmission artifacts, and only few attempts have been made to develop metrics tailored for specific display technology (eg, including backlight dimming artifacts). Development of such metrics is restricted by the scarcity of reliable subjective assessment results that could be used as ground truth for assessing the performance of objective metrics. Subjective results on backlight dimming artifacts are specific to the physical display used, and accurate modeling of backlight is a challenging task. This limits the possibilities to create generic annotated quality databases on backlight dimming artifacts.

A few subjective quality assessment studies on backlight dimming artifacts have been reported in the literature, but typically their main purpose is to compare the performance of different backlight dimming algorithms rather than to provide ground truth information for objective quality assessment. Some promising results in objective quality assessment, based on subjective results on a limited set of images, have been achieved with HDR-VDP-2 (Mantel et al., 2013a), but more research is required to confirm the findings on larger sets of both test images and display devices for the HDR domain.

13.5.2 Key Points of the Backlight Model

As expressed by Eq. (13.23), besides the PSF measurements necessary to compute the backlight intensity, the key aspects of the display model are the leakage and clipping modeling and the transfer function transforming physical luminance into perceived luminance.

To the best of our knowledge, except for the HDR-VDP-2 metric, all currently published image or video quality metrics take as input the pixel values of the content and not the absolute physically displayed luminance. (The MRSE also takes physical values as input, but uses them only as relative values.) As seen in Section 13.2.3, the mapping function between the two is highly nonlinear and it impacts the objective metrics evaluations. Traditionally, the metrics based on the pixel value assume a gamma relation that is the inverse of that of the display, which often has a value of 2.2 as recommended by the ITU for the electro-optical transfer function (ITU BT.709 or BT.2020). As stated in Section 13.2.3, that might not be accurate for luminance values as high as those HDR displays can provide. Below we briefly touch on the effects of leakage inherent to the combined LCD and backlight technology, which imposes a limitation on the image contrast.

13.5.2.2 Leakage dependence on the viewing angle

The leakage highly depends on the viewing angle, meaning that the bigger the viewing angle, the greater the leakage. Fig. 13.7 represents the angular variation of leakage, as measured on a Bang & Olufsen BV7 display, which is an LCD with edge-lit (1.5D) LED backlight. Even though the leaking light starts to stabilize around 40°, the emitted light also decreases and therefore the leakage continues to increase.

There are two aspects to this dependency: (1) the location of the observer — whether the observer sits in front of the display or whether the observer is watching it at an angle — and (2) the spatial variation of the leakage value — indeed, even when an observer is sitting in front of the display at a distance of three times the height (as recommended for subjective tests by the ITU), the viewing angle for the screen varies from − 16° to + 16° from side to side.

In Mantel et al. (2015), the highly contrasted sequences selected were shown to the observers at 0° and at an angle of 15°. The results show that the increased leakage value at 15° influences the observers’ perception. In this study, two models of the leakage were used as input for the quality calculation: one assuming a constant value over the whole screen and one with values varying spatially. The data show no evidence that inclusion of the spatial angle variation of the leakage over the screen represents an improvement for quality metrics, whereas distinguishing between subjects’ viewing angles of 0° and 15° showed significant improvement.