3.3.1.1 Parametric methods for radiometric calibration

While the CRF can differ from camera to camera, and even for the same camera from scene to scene, it is unlikely that its form is too exotic. On the basis of this observation, several methods assume a specific functional form for the CRF and attempt to estimate it using different strategies.

Farid (2001) assumes the CRF to be a simple gamma function, Z = X^γ, in which case the radiometric calibration process is reduced to estimating γ. He then observes that gamma compressing a signal introduces higher-order harmonics in the spectrum of the image. With that, he estimates the gamma as

$\begin{array}{l}\hfill \underset{\gamma }{arg\; min}\sum _{{\omega }_1,{\omega }_2\in [0,2\mathrm{\pi })}|B({\omega }_1,{\omega }_2)|,\end{array}$

where B is the bicoherence of the Fourier transform of Z, a measure of the correlation of harmonically-related frequencies (Farid, 2001).

In their work on HDR, Mann and Picard (1995) assume the CRF to be of a slightly more general form: Z = α + βX^γ. To estimate α, essentially the black level of the camera, they use a picture captured with the lens cap on, usually referred to as dark frame. Then, assuming the images to be registered, they compute the cross-histogram of the intensity values of a pair of images. For eight-bit images, for example, this is a 256 × 256 two-dimensional histogram where bin (r,c) contains the number of pixels such that Z_i(p) = c and Z_j(p) = r. The parameters (β,γ) can then be found by regression. Mann (2000) later extended this work by considering a number of different analytical forms for the CRF.

Mitsunaga and Nayar (1999) assume a polynomial form of the inverse of the CRF. Specifically,

$\begin{array}{l}\hfill X=f^{-1}(Z)=\sum _{k=0}^K{c}_k{Z}^k,\end{array}$

where K is the order of the polynomial. Given a rough estimate of the exposure time ratio R_i,i+1, the exact ratio and the inverse of the CRF can be found as

$\begin{array}{l}\hfill \underset{\{c_n\},R_{i,i+1}}{arg\; min}\sum _i\sum _p{\left(\sum _k{c}_k{(Z_i(p))}^k-R_{i,i+1}\sum _k{c}_k{(Z_{i+1}(p))}^k\right)}^2.\end{array}$

This is a straightforward least-squares optimization that can be solved iteratively for R_i,i+1 and the polynomial coefficients {c_n}, until convergence.

Grossberg and Nayar (2003b) relax the assumption on the CRF having a specific analytical form. They start by observing that, under the assumption that CRFs are monotonic, the space of the CRFs is convex, and therefore a linear combination of CRFs is still a CRF. After collecting a large number of real CRFs, {f_j}_j=1:J, they compute the first M eigenvectors of the covariance matrix, whose elements are defined as

$\begin{array}{l}\hfill C_{\text{r,c}}=\sum _j(f_j(X_{\text{r}})-\stackrel{-}{f}(X_{\text{r}}))(f_j(X_{\text{c}})-\stackrel{-}{f}(X_{\text{c}})),\end{array}$

where $\stackrel{-}{f}=1/J{\sum }_{j=1}^J{f}_j$, X are the different exposure values, and (r, c) index the bin in the covariance matrix. They show that as few as M = 3 eigenvectors can capture 99.5% of the energy, while use of M = 9 eigenvectors produces curves that are visually indistinguishable from the ground truth. This approach is accurate and extremely efficient, which is why it is used by several methods, as we will see in later sections.

3.3.1.2 Nonparametric methods for radiometric calibration

Making explicit assumptions on the analytical form of the CRF is not necessary. After all, radiometric calibration can be reduced to computing the lookup table that maps digital values to irradiance (or exposure) estimates, while respecting some properties.

In one of the seminal articles that perhaps most popularized modern HDR stack-based imaging, Debevec and Malik (1997) use a least-squares formulation to recover the inverse of the CRF as well as the irradiance values, while imposing smoothness of the recovered response:

$\begin{array}{l}\hfill \underset{E,g}{arg\; min}\sum _{i,p}(E(p)-g(Z_i(p))/t_i)+\lambda \sum _{z=Z_{min}}^{Z_{max}}{g}^{″}(z),\end{array}$

where g = f⁻¹. Essentially, the first term imposes the condition that Eq. (3.3) be satisfied, while the second encourages smoothness of the recovered CRF. Changing λ in Eq. (3.8) has a strong impact on the overall shape of the estimated CRF; for this reason their method can be seen as a means to convert the images in the stack to the same domain, rather than to perform accurate calibration.

Lee et al. (2013) make the acute observation that the estimates for the exposures X_i from the different images in the stack should be linearly dependent. More formally, the matrix formed with the N exposure images represented as column vectors, [X₁,X₂,…,X_N], should have rank 1. With the matrix O = [Z₁,Z₂,…,Z_N], where the columns are the observed images, the inverse of the CRF g can then be found as

$\begin{array}{l}\hfill g=\underset{g}{arg\; min}\text{rank}(g\otimes O),\end{array}$

where the operator ⊗ represents an element-wise application of a function. For numerical considerations, rather than minimizing the rank of the matrix, Lee et al. suggest minimizing the ratio of the first two singular values of g ⊗ O (this is also called the condition number). They propose solving

$\begin{array}{l}\hfill g=\underset{g}{arg\; min}\varphi (g\otimes O)+\lambda \sum _{Z_i}H\left(-{\left.\frac{\partial g}{\partial Z}\right|}_{Z_i}\right),\end{array}$

where ϕ(⋅) is the first condition number of a matrix, and H(⋅) is the Heaviside step function, which is 1 if its argument is nonnegative, and 0 otherwise. The second term encourages monotonicity: it adds a penalty that is proportional to the number of points where g is decreasing.

Another interesting approach is the work of Kim et al. (2012). On the basis of their analysis of a large database of JPEG+RAW⁵ images taken with different cameras and settings, they propose that a single CRF, as traditionally defined and estimated, is not sufficient to explain all of the in-camera processing steps. They observe that cameras perform a gamut mapping that is a function of the scene, or, more specifically, of the “picture style.” Therefore, they propose the following image formation model:

$\begin{array}{l}\hfill Z=f(h(T_{\mathrm{s}}{T}_{\mathrm{w}}X)),\end{array}$

where h(⋅) is the gamut mapping, T_s is the matrix for convection to sRGB, and T_w is the white balance matrix. The key observation is that the gamut mapping, carefully adapted to the scene’s type to improve the visual quality of the JPEG image, can be detrimental to the estimation of the inverse of the CRF; however, they hypothesize that the pixels that are affected the most by gamut mapping are the ones that are highly saturated (ie, in the HSV color space they have a high saturation value), and show that removing them from the computation of the CRF allows a more accurate result to be obtained.

Virtually all the methods described in this section require that the input stack be perfectly registered; in other words, they assume that the underlying irradiance at pixel p is the same across the stack. Grossberg and Nayar (2003a) propose overcoming this constraint by estimating intensity mapping functions (IMFs). The idea is similar to that of comparagrams (Mann and Picard, 1995): IMFs capture how the brightness values change between two images in the stack. However, rather than building the cross-histogram of two images, which implicitly assumes registration, they look at the cumulative histogram of brightness: $\text{C}(\tilde{Z})={\sum }_{Z=0}^{\tilde{Z}}H(Z)$, where H is the histogram of the image. The advantage of the cumulative histogram of an image is that it is robust to small motions in the scene. Given the cumulative histogram of two images C₁ and C₂, it is straightforward to compute the IMF τ_1,2:

$\begin{array}{l}\hfill {\tau }_{1,2}(Z_1)=C_2^{-1}(C_1(Z_1)).\end{array}$

More recently, Badki et al. (2015) proposed an algorithm specifically designed to tackle the problem of radiometric calibration for scenes with significant motion. Their approach builds upon both the work of Grossberg and Nayar (2003a) and the method for radiometric calibration by rank-minimization of Lee et al. (2013). First, inspired by the method of Hu et al. (2012), they extend the method of Grossberg and Nayar to large motions by proposing a new random sample consensus (RANSAC)-based method for computing the IMFs that is robust to such motions. Second, they replace the least-squares optimization for solving for the CRF in Grossberg and Nayar with the rank-minimization scheme of Lee et al. However, the original method of Lee et al. uses artifact-prone, pixel-wise correspondences in the optimization, so Badki et al. reformulated the optimization to replace these correspondences with IMFs. The result is an algorithm that can solve for the CRF even in cases of significant camera and scene motion.

3.3.1.3 Single-image methods for radiometric calibration

The methods we have described thus far assume that multiple images Z_i of the same scene are available, essentially allowing one to measure different segments of the CRF. However, radiometric calibration can also be performed on a single image, although with potentially lower accuracy. Single-image methods can be beneficial in the context of stack-based HDR imaging when the assumption that the CRF is the same across the stack is not valid.

Matsushita and Lin (2007) leverage the fact that the different sources of noise in a camera system can be modeled with symmetric distributions: the cumulative noise distribution should then be symmetric as well, and any deviation from the overall symmetry results only from the nonlinearity of the CRF. Therefore, using existing methods to estimate the noise distribution from a single image, they frame the problem of radiometric calibration as

$\begin{array}{l}\hfill g=\underset{g}{arg\; min}{\xi }_{\eta },\end{array}$

where ξ_η is a measure of the skewness induced by f to the noise distribution η. Eq. (3.13) essentially states that g should restore the symmetry of the noise distribution that is expected before the CRF is applied to the image.

Lin et al. (2004) propose another single-image method. They observe that, owing to the finite size of the pixels, the irradiance of pixels at the boundary between two uniform regions is a linear combination of the values on either side of the edge. Moreover, moving along a direction orthogonal to the edge in image space should correspond to moving along a line in RGB space only if the CRF is linear. However, if a nonlinear CRF is applied to the pixel values, these linear segments become curved. Therefore, they use the method of Grossberg and Nayar (2003b) to parameterize the CRF, and formulated an optimization problem where the solution for g maximizes the linearity in the RGB space of several segments that cross image edges.

Lin et al. (2004) extended this method to a single, grayscale image based on the same idea: the irradiance values of edge pixels should be a linear combination of the irradiance values of the regions on either side of the edge. However, since color is not available, they look at the histograms of the intensities in patches lying across edges. These histograms should be roughly uniform, because the point spread function, together with the finite pixel size, turns a sharp edge into a smooth gradient. Once again, they formulate an optimization problem where the inverse CRF is the function that maximizes the uniformity of histograms of different “edge” patches.

The methods described in this section are not intended to serve as a complete survey of the range of radiometric calibration methods; rather they are meant to offer insight into the strategies most commonly used. Many other relevant methods have been proposed, such as methods that model and estimate the noise characteristics of the sensor (Tsin et al., 2001; Granados et al., 2010), approaches based on a probabilistic framework (Xiong et al., 2012), and algorithms working with video sequences, where the CRF may also vary from frame to frame (Grundmann et al., 2013).

3.3.2.1 Maximum likelihood estimation

A more theoretically founded approach is to compute the maximum likelihood (ML) estimate of the irradiance $\tilde{E}(p)$ (Tsin et al., 2001; Granados et al., 2010). Given two irradiance estimates E_i(p) = X_i(p)/t_i from two different images in the stack, we seek to compute:

$\begin{array}{l}\hfill \tilde{E}=\underset{E}{arg\; max}p(\tilde{E}|E_1,E_2),\end{array}$

where we omitted the dependence on the pixel p for clarity. We can assume that the observations are drawn from two independent Gaussian distributions $\mathcal{N}(E_i,{\sigma }_i)$. We can then write

$\begin{array}{ll}\hfill p(\tilde{E}|E_1,E_2)& =\frac{p(E_1,E_2|\tilde{E})p(\tilde{E})}{p(E_1,E_2)}\hfill \\ \hfill & \propto p(E_1|\tilde{E})\cdot p(E_2|\tilde{E}),\hfill \end{array}$

where we made the common assumption of a uniform prior distribution. Plugging Eq. (3.21) into Eq. (3.20) and taking the logarithm, we can write

$\begin{array}{ll}\hfill \tilde{E}& =\underset{\tilde{E}}{arg\; max}p(E_1|\tilde{E})\cdot p(E_2|\tilde{E})\hfill \\ \hfill & =\underset{\tilde{E}}{arg\; min}\frac{{\left(E_1-E\right)}^2}{{\sigma }_1^2}+\frac{{\left(E_2-E\right)}^2}{{\sigma }_2^2}.\hfill \end{array}$

We can find the ML estimate for $\tilde{E}$ by setting the derivative with respect to $\tilde{E}$ in Eq. (3.22) to zero:

$\begin{array}{ll}\hfill \tilde{E}& =\frac{{\sigma }_2^2}{{\sigma }_1^2+{\sigma }_2^2}{E}_1+\frac{{\sigma }_1^2}{{\sigma }_1^2+{\sigma }_2^2}{E}_2\hfill \\ \hfill & =\frac{1/{\sigma }_1^2{E}_1+1/{\sigma }_2^2{E}_2}{1/{\sigma }_1^2+1/{\sigma }_2^2},\hfill \end{array}$

from which we can see that $w_{\mathrm{ML}}=1/{\sigma }_i^2$. Several methods build on this result by observing that weights should at least account for the uncertainty of the pixel’s value. The first attempt in this direction was the work of Tsin et al. (2001). After modeling white balance as an affine transformation of the exposure X, and calibrating the sensor for photon shot noise and thermal noise, they define the weights as

$\begin{array}{l}\hfill w_{\mathrm{T}}=\frac{1}{\sigma (Z)},\end{array}$

where σ(Z) is the standard deviation of the signal, measured from the residuals of the irradiance estimation. Kirk and Andersen (2006) used the ML weights as well:

$\begin{array}{l}\hfill w_{\mathrm{KA}}=\frac{1}{\sigma (X_i/t_i)}=\frac{t_i^2}{\sigma (X_i)}\approx \frac{t_i^2}{\sigma (Z_i)g^{\prime }{(Z_i)}^2}.\end{array}$

Arguably, the most complete noise model was proposed by Granados et al. (2010). They too use the ML weights, but improved on previous work by considering both spatial and temporal noise, the latter being also modeled more accurately than in other approaches. Moreover, they precalibrate the camera noise parameters to avoid polluting the irradiance estimate with the uncertainty of the noise estimation.

3.3.2.2 Winner-take-all merging schemes

A few researchers have proposed a different approach to the generation of an HDR map from a stack of LDR images. Such work is based on the observation that the picture with the longest exposure in the stack is also the one with the smallest quantization noise, and the one impacted by the least photon shot noise (see also Section 3.2). Following this logic, in his work preceding the work of Mann and Picard (1995), Madden (1993) suggested combining the different images in an HDR stack by using the longest, nonsaturated exposure available for each pixel p. A similar approach was proposed by Tocci et al. (2011); however, they also suggest blending the irradiance estimates at the very top and bottom of the useful range of each LDR image to prevent banding artifacts in the transition areas. Additionally, Tocci et al. worked in the Bayer domain to prevent artifacts due to demosaicing when a subset of the color channels saturate, and assessed the reliability of a pixel’s estimate also based on its neighborhood.

3.3.2.3 Exposure fusion methods

The approach to HDR we have described so far consists of radiometric calibration, followed by a merging process that produces the final HDR result. To be displayed on regular monitors, the HDR map needs to be tone mapped. An orthogonal approach is that of fusing the images directly in the nonlinear brightness domain. The most popular method in this category is exposure fusion by Mertens et al. (2007). Their simple and effective method sidesteps the estimation of the exposure values X_i, and blends the digital values Z_i directly. To reflect the quality of a pixel value, they define

$\begin{array}{l}\hfill w_{\mathrm{EF}}=w_{\mathrm{s}}\cdot w_{\mathrm{c}}\cdot w_{\mathrm{e}},\end{array}$

where w_s, the color saturation weight, encourages more vivid colors, w_c, the contrast weight, penalizes low contrast, and w_e, the well-exposedness weight, prefers pixels close to the middle of the range. A naïve application of the method directly to the image may cause visible seams due to abrupt changes in the values of the weights of neighboring pixels. To prevent such artifacts, Mertens et al. decompose the image into a Laplacian pyramid, and combine it with a Gaussian pyramid decomposition of the weight maps to create the final image. Once again, unlike Eq. (3.14), w_EF is used in the weighted average of the digital values Z_i. Merging images directly in the nonlinear brightness domain has advantages and disadvantages. In general, it creates natural-looking results, whereas the tone mapping procedure required for the standard HDR pipeline often produces unnaturally contrasted pictures. Moreover, artifacts due to misregistration are often attenuated by the weighting process. At the same time, it never produces an actual HDR irradiance map, which can be beneficial for computer vision tasks. Finally, when the difference in brightness between the images in the stack is too large, it can introduce artifacts caused by the Gaussian pyramid decomposition of the weights. Several methods build on this idea to increase computational efficiency (Gelfand et al., 2010), to embed deghosting (Zhang and Cham, 2010; An et al., 2011; Gallo et al., 2015), or simply to propose different weights (Shen et al., 2011).

3.4.2.1 Rejection methods without a reference image

Rejection methods that do not define a single reference image are based on the observation that small moving objects tend to affect different regions of the images across the stack. Therefore, if the stack of LDR images is large enough (usually five or more images), a pixel p is likely to capture the irradiance from the static parts of the scene in most of the pictures. Methods from this category then propose a model for how pixel p should behave across the stack if it represented a static object, and discard the values Z_i(p) across the stack that do not follow this model, as they are likely to be affected by motion. However, these methods run into the problem that neighboring pixels may come from a different subset of exposures where objects might be in different positions, which would introduce visible discontinuities. To minimize these effects, these algorithms generally identify clusters or groups of pixels that can be drawn coherently from one (or more) of the input LDR images.

One of the first methods to do this was described in Section 4.7 of the book by Reinhard et al. (2005). In this method, the CRF is assumed to be known so that the LDR images Z_i can be converted into their corresponding irradiance images E_i. The different images E_i should theoretically be the same, except for noise, saturation, and motion, which may alter some of the pixels’ values from image to image. Therefore, Reinhard et al. propose computing the weighted normalized variance of the values at each pixel p to determine which pixels are affected by motion:

$\begin{array}{l}\hfill {\sigma }^2(p)=\frac{\sum _{i=0}^N{w}_i(p)E_i{(p)}^2/\sum _{i=0}^N{w}_i(p)}{{\left(\sum _{i=0}^N{w}_i(p)E_i(p)\right)}^2/{\left(\sum _{i=0}^N{w}_i(p)\right)}^2}-1.\end{array}$

This equation, explained only verbally by Reinhard et al. (2005) and later presented mathematically by Jacobs et al. (2008), uses weights w_i to exclude overexposed or underexposed pixels from the computation as their divergence from the true irradiance may bias the estimate of the variance. Note that unlike traditional variance, the variance in Eq. (3.27) is normalized to the actual size of the signal.

The key observation is that when one is looking across the image stack, pixels that are not affected by motion should have a smaller variance than those that measure irradiance from different objects. Of course, one could set a simple threshold for this variance to distinguish between these two cases. However, this naïve approach has the problem that the image would suffer from discontinuity artifacts when neighboring pixels are selected from different images with different objects.

To avoid this problem, rejection methods that do not define a reference image must group pixels together into larger clusters, where all the pixels in a cluster are drawn coherently from the same image in the stack. In the particular case of Reinhard et al. (2005), morphological operators such as erosion and dilation are used to grow the binary image after thresholding of the variance to create larger, contiguous regions that are identified to have motion. To decide which exposure to use in each region, they generate a histogram of irradiance values in each region and find the maximum value that is not in the top 2%, which they consider to be outliers. They then find the longest exposure that still includes this maximum value within its valid range, and interpolate between this exposure and the original HDR result using the per-pixel variance as a mixing coefficient. In this way, pixels with lower variance across the stack will use the original HDR result, while pixels with larger variance will use the single exposure. This algorithm is able to produce deghosted images and, at the same time, ensure that each region is drawn coherently from one exposure.

In another method, Eden et al. (2006) first use a SIFT-based feature registration technique to align the input images in the presence of varying exposure levels. Once the stack has been aligned, they map the images to the irradiance domain, where they draw each pixel of the final composite from one of the input images. This is done in two steps. In the first step, they use a subset of the aligned input images to create a reference panorama that covers the full angular extent of all the inputs using graph cuts (Boykov et al., 2001). However, because of overexposure or underexposure this reference image could have areas of missing information, so they introduce detail from images that are better exposed while solving for a smooth transition between regions in a second pass. This problem is minimized via max-flow graph cut to produce the final result, which can be smoothed out to remove any remaining seams.

The approach of Khan et al. (2006) attempts to compute a ghost-free image through several iterations of kernel density estimation that modify the blending weights w_i of Eq. (3.14), by assuming that background (static) pixels are the most common. Essentially, they compute the probability that a given pixel is part of the background, and use this weight when blending so pixels from dynamic objects (and not the background) get a smaller weight. To do this, they represent each pixel in the stack of images with a five-dimensional vector x_i(p), where i is the index of the image in the stack and p is the pixel location. This vector contains the three LDR color channels of the pixel value (in Lab space) as well as the coordinates of the pixel on the image.

For a given pixel p, they select all pixels y_j(q) in its 3 × 3 neighborhood over all the images in the stack, denoted by $\mathcal{N}(p)$. Note that the pixels at position p across the stack are not included in this neighborhood. They begin by assuming that all y_j(q) are equally likely to be part of the background. The probability that a pixel p belongs to the background B (given by P(x_i(p) | B)) can then be calculated with a kernel density estimator:

$P\left(x_i\left(p\right)|B\right)=\frac{{\displaystyle \sum _{j,q\in \mathcal{N}\left(p\right)}{w}_{j,q}{K}_H\left(x_i\left(p\right)-y_j\left(q\right)\right)}}{{\displaystyle \sum _{j,q\in \mathcal{N}\left(p\right)}{w}_{j,q}}},$

where the kernel K_H is a five-dimensional multivariate Gaussian density function, and the weight w_j,q indicates the probability of the pixel belonging to the background. For the first iteration, these weights are initialized to a “hat” function similar in spirit to that of Debevec and Malik (1997). For subsequent iterations, the value of the weights can be set to the probability that the pixel belongs to the background, as computed by Eq. (3.28) in the previous iterations. However, each time the newly computed weights are multiplied by the initial weights from the hat function to continually diminish the probability that pixels that are overexposed or underexposed are used in the final estimates. On convergence, the weights are plugged into Eq. (3.14) to merge the LDR images into an HDR result.

Jacobs et al. (2008) extended the deghosting algorithm of Reinhard et al. (2005) in several ways. First, they prealign the images as in the earlier work of Ward (2003), but in this case they iteratively solve for the translation and rotation that maximizes the XOR score between the two median threshold bitmaps. In the second stage, they replace the variance metric of Eq. (3.27) with a local entropy measure that indicates movement in the scene. Specifically, they measure the local entropy at each pixel in the LDR image Z_i by looking at the pixel values z within a two-dimensional window around pixel p:

$\begin{array}{l}\hfill H_i(p)=-\sum _{z}P(Z=z)log\left(P(Z=z)\right),\end{array}$

where the probability function P(Z = z) is computed from the normalized histogram of the intensity values of pixels within the window. Using these entropies, they compute an uncertainty image U, which is the local weighted entropy difference between the images:

$\begin{array}{l}\hfill U(p)=\sum _{i=1}^{N-1}\sum _{j=0}^{i-1}\frac{v_{ij}}{\sum _{i=1}^{N-1}\sum _{j=0}^{i-1}{v}_{ij}}\left|H_i(p)-H_j(p)\right|,\end{array}$

where $v_{ij}=min(w_i(p),w_j(p))$, and weights w_i(p) and w_j(p) are computed with Debevec and Malik’s triangle function in Eq. (3.15), with $Z_{min}=0.05$ and $Z_{max}=0.95$. The intuition is that static regions would have similar local entropy measures across the LDR images, even if they are near edges, which might increase the variance because of slight camera motions. This method also does not need a priori knowledge of the CRF, as the entropy measurement can be done in the LDR domain. As with previous methods, this uncertainty image is thresholded and the resulting binary image is eroded and dilated to produce contiguous regions that are affected by motion. At this point, each region is filled with values from one of the irradiance images E_i that is not overexposed or underexposed in that region and is blended with the original HDR value to avoid artifacts at the borders.

Sidibe et al. (2009) observed that the value of pixel p across the stack should increase with the exposure time, since the camera response curve is monotonically increasing: Z_i(p) ≤ Z_j(p) if t_i < t_j. Therefore, they propose identifying regions where this order relation is broken at least once as ghosted regions. Of course, there might be motions that preserve this order which would not be detected. In the ghosted regions, they use the input images that they deem to have captured the background, which is assumed to appear in most of the images. To do this, they effectively compute the histogram of irradiance values at each ghosted pixel and compute the mode of this distribution, which is the value that appears the most often. The mode is assumed to be the background and the values are merged (ignoring pixel values close to saturation or zero) to form the final HDR image. To have enough samples at each pixel to compute the mode, they require at least five images in the stack.

In another approach, Pece and Kautz (2010) first compute median threshold bitmaps for each image in the stack as proposed by Ward (2003) and accumulate these binary maps for each pixel over all the exposures. Values that are neither 0 nor N are considered motion, and the morphological operators of dilation and erosion are applied to this result to generate the final motion map. Pece and Kautz present results they obtained using exposure fusion (Mertens et al., 2007), where they select the best available exposure for each of the clusters in the motion map to produce their results.

Zhang and Cham (2012) presented a technique similar to exposure fusion (Mertens et al., 2007) (see Section 3.3.2.3) because they fuse the images without generating an HDR image first, but use a novel consistency metric that uses the image gradient to detect movement. To begin, they compute the magnitude M_i(p) and direction θ_i(p) of the gradient around every pixel of each image in the stack. Next, they observe that the magnitude of the gradient can be used to determine saturated or underexposed pixels, as these regions typically have lower gradient magnitude. Therefore, they propose a visibility measure that indicates how well exposed and visible a particular pixel is:

$\begin{array}{l}\hfill V_i(p)=\frac{M_i(p)}{\sum _{i=1}^N{M}_i(p)+\epsilon },\end{array}$

where the ε is a small value (eg, 10⁻²⁵) to avoid division by zero. Finally, they observe that the gradient direction can serve as a consistency measure to detect motion across the exposure stack because of its invariant property over different exposures. Therefore, they compute the gradient direction difference of the ith image with respect to the jth image as follows:

$\begin{array}{l}\hfill d_{ij}(p)=\frac{\sum _{k\in \mathcal{N}}\left|{\theta }_i(p+k)-{\theta }_j(p+k)\right|}{M^2},\end{array}$

where $\mathcal{N}(p)$ is the set of offsets of the pixels in an M × M square neighborhood around pixel p. With this, a consistency score S_i can be computed for every image. One does this by accumulating a Gaussian weight for each pixel based on the difference of its gradient direction across the stack:

$\begin{array}{l}\hfill S_i(p)=\sum _{j=1}^{N}exp\left(-\frac{d_{ij}{(p)}^2}{2\cdot 0.2}\right).\end{array}$

Given these scores, a consistency score for each pixel p in the stack image i can then be calculated as

$\begin{array}{l}\hfill C_i(p)=\frac{S_i(p)\cdot {\alpha }_i(p)}{\sum _{j=1}^N{S}_j(p)\cdot {\alpha }_j(p)+\epsilon },\end{array}$

where α_i(p) is simply 1 if the pixel is well exposed and 0 if it is not. Here, we use the term “well exposed” to define a pixel whose value is in the middle of its range — say, between 0.1 and 0.9 in a normalized pixel value range. These consistency scores can then be used to compute the final weights for the fusion process (Eq. 3.26):

$\begin{array}{l}\hfill w_{\mathrm{EF}}(p)=\frac{V_i(p)\cdot C_i(p)}{\sum _{j=1}^N{V}_j(p)\cdot C_j(p)+\epsilon }.\end{array}$

The final image can then be fused without the need for tone mapping, but does not produce a true HDR result.

Granados et al. (2013) propose using a noise-aware model to determine whether the image stack values for a particular pixel are consistent, which means that they measure the same static irradiance. They observe that for a pixel in a static region, the exposure values across the stack should all be within an error margin based on the noise of the imaging system. Therefore, rather than using an arbitrary threshold to detect motion, they characterize the noise in the imaging system (both shot noise and readout noise) as a Gaussian distribution, which enables them to determine the probability that the difference between two pixel values is caused by scene motion or noise. This idea can be extended to the N images in the stack to produce consistent subsets, which will not introduce ghosting artifacts when combined.

Once these consistent subsets have been identified, the next challenge is to ensure that neighboring pixels are drawn coherently from the subsets to avoid artifacts. To do this, they pose the irradiance reconstruction problem as a labeling problem, solved by minimizing an energy function with two terms. The first, a consistency term, encourages the pixels to be selected from consistent subsets of the image to reduce ghosting. The second is a prior term that penalizes incoherency across neighboring pixels by enforcing that neighboring pixels should be drawn from the same consistent subset. They solve this labeling problem using the expansion-move graph-cuts algorithm and then merge the consistent sets at each pixel to produce the final HDR result. However, despite this graph-cut optimization, their method still cannot always guarantee a semantically consistent result, and thus it requires a manual intervention to resolve remaining issues.

Finally, Oh et al. (2015) recently proposed a clever rank minimization strategy to solve for the final HDR image. They begin by assuming that there are two kinds of motion between the images in the stack. The first is global motion due to camera movement, which they assume can be modeled with a homography. The second is local motion, which they want to eliminate, and is caused by the nonrigid movement of objects in the scene. Their key observation is that if global motion is accounted for, the stack of exposure images X₁,…,X_N should be linearly dependent. In other words, barring local motion, saturation, or noise, the globally aligned exposure images would simply be scaled versions of E (ie, X_i = E ⋅ t_i). Therefore, they attempt to eliminate motion artifacts by enforcing that the matrix whose columns are the input LDR images should be of rank 1 (ie, all columns should be linearly dependent).

Oh et al. first account for the global motion by modeling the hypothetical process of capturing “globally aligned” LDR source images, as if the camera were not moving. This can be written as ${\tilde{Z}}_i=f({\tilde{X}}_i+{\eta }_i)$, where ${\tilde{Z}}_i$ are the LDR images that would have been taken with a static camera, ${\tilde{X}}_i$ is the ideal exposure image that contains only static scene information, and η_i is a “noise” term representing the local motion in the scene. Since we can apply a homography operator ° h_i to perform global alignment on each of the inputs Z_i, we can write ${\tilde{Z}}_i=Z_i°h_i$. Once the camera has been calibrated so that its response curve is linear (see Section 3.3.1), the capture process can be modeled as

$\begin{array}{l}\hfill {\tilde{Z}}_i=Z_i°h_i=a{\tilde{X}}_i+a{\eta }_i.\end{array}$

We can then vectorize the terms in this equation and combine them into matrices using all of the N captured images: Z °h = X + η. Since all the columns of X are simply scaled versions of the static scene irradiance E, it is a rank 1 matrix. At the same time, η is sparse if we assume that most of the scene is static and only a few areas are affected by motion. Therefore, the problem of removing motion artifacts from the HDR image is equivalent to the problem of solving for a rank 1 matrix X and sparse matrix η through the following optimization:

$\begin{array}{ll}\hfill {\mathbf{X}}^{*},{\mathit{\eta }}^{*},{\mathbf{h}}^{*}=& \underset{\mathbf{X},\mathit{\eta },\mathbf{h}}{arg\; min}{p}_2(\mathbf{X})+\lambda \parallel \mathit{\eta }{\parallel }_1\hfill \\ \hfill & \text{subject to}\mathbf{Z}°\mathbf{h}=\mathbf{X}+\mathit{\eta }.\hfill \end{array}$

Here, $p_2(\mathbf{X})={\sum }_{i=2}^N{\sigma }_i(\mathbf{X})$ is the sum of the singular values from the second to the last⁶ which measures the rank of the matrix. The L1 norm ⋅₁ is a measure of sparsity, and the weighting coefficient λ balances the contribution of the two terms. This constrained optimization problem can be solved with augmented Lagrange multipliers (Peng et al., 2012), where the problem is divided into three different subproblems for X, η, and h and minimized iteratively.

As discussed earlier, rejection-based algorithms have their drawbacks, but this subset of algorithms that do not specify a reference image have additional problems. For example, they can often produce images that contain duplicate objects or other artifacts, because the semantic meaning of objects is lost when the consistent sets computed in neighboring pixels are not coherent. These artifacts typically require a manual correction. Furthermore, since they typically strive to use only “background” pixels from each image, rejection methods of this type will suppress dynamic objects from the HDR result.

Finally, because these algorithms produce images that do not adhere to a ground truth reference (ie, an HDR picture taken at a specific moment in time), they cannot be easily extended to the capture of HDR video. The reason for this is twofold. First, they do not guarantee temporal continuity since each frame is individually computed, and may use a pixel cluster that is not temporally coherent with the neighboring frames. Second, even if temporal coherency could be enforced, the fact that dynamic objects are usually suppressed defeats the purpose of taking a video in the first place.

3.4.2.2 Reference-based rejection methods

The algorithms in this category select a single image from the stack as the reference and use it as the foundation of the final image. In other words, the HDR result will be geometrically consistent with this reference, at least in the parts where it is well exposed. The other images in the stack will be tested against the reference, and pixels deemed to have been affected by motion will be rejected. For regions where all the images in the stack are rejected, the HDR result would be reconstructed with use of only the reference.

One of the first examples of these algorithms is the work of Grosch (2006), which takes two differently exposed images and first aligns them using a variant of the method proposed by Ward (2003), extended to consider both translation and rotation. He then computes the CRF on the largely aligned images using the method of Grossberg and Nayar (2003a), and uses the first image (the reference) to predict the estimated values in the second:

$\begin{array}{l}\hfill {\tilde{Z}}_2(p)=f\left(\frac{t_2}{t_1}g(Z_1(p))\right).\end{array}$

If the predicted color ${\tilde{Z}}_2(p)$ is beyond a threshold from the actual color in the second image (ie, $|{\tilde{Z}}_2(p)-Z_2(p)|>\epsilon$), the algorithm assumes that Z₂(p) would introduce motion artifacts, and falls back to using only the first image at these locations. This produces an artifact-free result because it largely follows the reference, and has the advantage that it does not need a priori knowledge of the CRF. However, if the scene contains large moving objects, then the radiometric calibration step could fail as well, unless a more robust calibration procedure, such as the algorithm of Badki et al. (2015), is used (see Section 3.3.1.2).

Gallo et al. (2009) propose a similar approach. They first define the reference as the image in the stack with the fewest overexposed and underexposed pixels; they then compare the values of the pixels of the different images in the stack against it. They perform the comparison in the log-irradiance domain, where the following relationship holds:

$\begin{array}{l}\hfill ln(X_{\mathrm{ref}})=ln(X_i)+ln(t_i/t_{\mathrm{ref}}),\end{array}$

where the dependence on pixel p is omitted for clarity. Pixels whose exposure X_i(p) is farther than a threshold from the value predicted by Eq. (3.39) belong to moving objects. However, for increased robustness, rather than working directly with pixels, Gallo et al. propose working with patches; a patch from the ith image in the stack is merged with the corresponding patch in the reference if the number of its pixels obeying Eq. (3.39) is above the threshold. The patches are defined on a regular grid; because two neighboring patches in the reference image can be merged with a different subset of patches from the stack, visible seams may exist at the patches’ boundaries. To address this issue, the patches are blended with a Poisson solver (Pérez et al., 2003).

Raman and Chaudhuri (2011) extended the work of Gallo et al. (2009) by replacing squared patches with superpixels, which are inherently more edge aware. However, rather than computing the HDR irradiance map, they fuse the images directly using their nonlinear digital pixel values. To begin, they compute the weighted variance proposed by Reinhard et al. (2005), see Eq. (3.27), to identify the pixels that may have measured irradiance from moving objects. Then, using only the pixels that are deemed to have captured static objects, they fit fourth-order polynomials to create a set of N − 1 IMFs that map the pixel values of each exposure in the stack to the reference.

In the next step, all the images but the reference are segmented into superpixels with homogeneous color and texture. The idea is to blend the superpixels that are static with respect to the reference with the well-exposed reference information. To identify a superpixel as static, the authors use the IMF and compare its pixels with those of the superpixel in the reference; to make the process more robust to noise they also threshold the distance of each pixel from the predicted value. If 90% of the pixels are within this threshold, then the superpixel is considered to be static with respect to the reference. These static superpixels are then decomposed into 6 × 6 patches with an overlap of one pixel on each edge. The patches with more than 90% of pixels within the static superpixel are considered static as well, and their gradients are merged by use of a Gaussian weighting function based on exposure. Finally, a Poisson solver is used to reconstruct the final color information from the gradients (Pérez et al., 2003).

Wu et al. (2010) propose a set of criteria for detecting moving pixels. First, they use a criterion that ensures that the pixel values are monotonically increasing as the exposure time is lengthened, similar to the earlier work of Sidibe et al. (2009). Next, they use a criterion similar to that of Grosch (2006) that compares a pixel’s value with that predicted from another exposure after compensation for the CRF and the exposure time ratio. If a pixel violates any of these criteria, then it is considered to be affected by motion. The final motion map is generated by use of the morphological operators, such as opening and closing. Once the pixels affected by motion have been identified, Wu et al. proceed to compute the final HDR image. Specifically, they select image k as a reference and use it to fill in the pixels affected by motion in the neighboring images k − 1 and k + 1 with the value predicted by the camera response curve as in Eq. (3.38). These new images are then used to predict the next images, and so on until the entire stack has been processed. Finally, they correct boundary artifacts near the edges of the regions in the motion map by convolving the images with a low-pass kernel, and using the result to replace the values calculated originally in these regions. The HDR image is then computed with the standard merging equation (Eq. 3.14).

Heo et al. (2010) first globally aligned the images in the stack with the reference image using a homography estimated with SIFT features using RANSAC. Next, N − 1 joint histograms are computed between the values in the reference and those in the other images. These histograms are then converted into smooth joint probability distribution functions through a Parzen windowing process using a 5 × 5 Gaussian filter followed by a normalization to enforce that the subtended area sums to 1. Pixels in the other images in the stack with a joint probability less than a fixed threshold are labeled as ghost pixels. This simple thresholding of the joint probability to determine ghost regions can be very noisy, however, so Heo et al. further refine the ghost regions using an energy minimization that enforces smoothness between neighboring pixels, and which is solved with use of graph cuts.

The refined ghost regions can be used to compute new joint histograms that are not affected by motion artifacts; therefore, the algorithm iteratively alternates between computing the joint probability distribution functions and detecting the ghost regions. The pixels not affected by motion are then used to compute the CRF with the method of Debevec and Malik (1997). To further reduce artifacts, this CRF is used to refine the radiance values of all the pixels in the other images in order to make their values more consistent with the reference image. Finally, the different exposures are blended to generate the final HDR result with a weighted filtering step. These weights are computed by application of a bilateral filter (Tomasi and Manduchi, 1998) to all the samples in a patch around a pixel, with use of a global intensity transfer function to compare the differently exposed pixel values.

Rejection-based methods that use a single reference image generally reduce or completely remove ghosting artifacts from the final HDR image. They do, however, have some of the shortcomings of all the rejection-based methods we discussed earlier, such as not being able to handle dynamic HDR content. Furthermore, if the regions where the reference is overexposed or underexposed are large, these algorithms could have problems recovering the full dynamic range because of their heavy reliance on the reference; see Fig. 3.7.

3.4.3.1 Optical flow and correspondence registration methods

Bogoni (2000) presented perhaps the earliest known method to register a stack of images for HDR reconstruction. First, he applies an affine motion estimation to globally align the images. This process, based on earlier work on registration for image mosaics (Hansen et al., 1994), operates in a multiresolution fashion from coarse to fine, using a Laplacian pyramid scheme. At each iteration, the optical flow field is computed from one image to another by means of local cross-correlation analysis, and then an affine motion model is fit to the flow field by weighted least-squares regression. The affine transform is then used to warp each image to align it roughly with the reference. At this point, a second step performs unconstrained motion estimation with optical flow between each source image and a predefined reference. This resulting field is used to warp the individual sources to compute the final registration with the reference.

Jinno and Okuda (2008) propose addressing the problem of ghosting with Markov random fields. After selecting the reference image, they estimate three arrays (the same size as the images) for each of the other images in the stack. The first is a displacement field d, and the second is a binary occlusion field o that indicates the parts of the reference that are occluded in the second image. This is computed by thresholding the maximum search distance for the displacement field: if a pixel cannot be found in a neighborhood $\mathcal{N}(p)$ around a given pixel that has a luminance within a specific threshold, then pixel p is considered occluded. The third is a saturation field, a binary mask that keeps track of the regions where the second image is overexposed or underexposed. Since these arrays are spatially coherent, they can be modeled as Markov random fields and computed with use of Bayes rule as an estimation problem that finds the most probable fields d, o, and s given observed images Z_ref and Z_i:

$\begin{array}{ll}\hfill \underset{\mathbf{d},\mathbf{o},\mathbf{s}}{max}P(\mathbf{d},\mathbf{o},\mathbf{s}|Z_{\mathrm{ref}},Z_i)& =\underset{\mathbf{d},\mathbf{o},\mathbf{s}}{max}\frac{P(Z_{\mathrm{ref}}|\mathbf{d},\mathbf{o},\mathbf{s},Z_i)P(\mathbf{d},\mathbf{o},\mathbf{s}|Z_i)}{P(Z_{\mathrm{ref}})}\hfill \\ \hfill & =\underset{\mathbf{d},\mathbf{o},\mathbf{s}}{max}\frac{P(Z_{\mathrm{ref}}|\mathbf{d},\mathbf{o},\mathbf{s},Z_i)P(\mathbf{d}|\mathbf{o},\mathbf{s},Z_i)P(\mathbf{o}|\mathbf{s},Z_i)P(\mathbf{s}|Z_i)}{P(Z_{\mathrm{ref}})}.\hfill \end{array}$

This problem is analogous to that of finding $\underset{\mathbf{d},\mathbf{o},\mathbf{s}}{max}P(Z_{\mathrm{ref}}|\mathbf{d},\mathbf{o},\mathbf{s},Z_i)P(\mathbf{d}|\mathbf{o},\mathbf{s},Z_i)P(\mathbf{o}|\mathbf{s},Z_i)P(\mathbf{s}|Z_i)$, which they approximate by first finding s through thresholding, and then iteratively solving for d and o. Once they have these fields, they can use them during the merging stage to produce the final HDR result.

Zimmer et al. (2011) align images in the stack with a specified reference using an energy-based optical flow optimization that is more tolerant to changes in exposure. To achieve this invariance, they define an energy function that leverages the gradient constancy, similar to that of Brox et al. (2009) and that of Brox and Malik (2011). Specifically, for each image i in the stack they compute a dense displacement field u_i(p) = [u_i(p),v_i(p)]^T that specifies an offset at every pixel by minimizing an energy function of the form

$\begin{array}{l}\hfill E({\mathbf{u}}_i(p))=\sum _{p\in \Omega }D({\mathbf{u}}_i(p))+\lambda S(\nabla {\mathbf{u}}_i(p)),\end{array}$

where D(u_i(p)) is the data term that tries to align the image with the reference, and S is the smoothness term (regularizer) that encourages smooth flow in places where the reference image is unreliable (ie, overexposed or underexposed). Because the brightness constancy across the stack is violated in this application, they propose that the data term D(u_i(p)) should try to match the gradient of the offset region in image Z_i with that of the reference:

$\begin{array}{l}\hfill D({\mathbf{u}}_i(p))=\Psi \left(\frac{1}{n_x}{\left|\frac{\partial }{\partial x}{Z}_i(p+{\mathbf{u}}_i(p))-\frac{\partial }{\partial x}{Z}_{\mathrm{ref}}(p)\right|}^2+\frac{1}{n_y}{\left|\frac{\partial }{\partial y}{Z}_i(p+{\mathbf{u}}_i(p))-\frac{\partial }{\partial y}{Z}_{\mathrm{ref}}(p)\right|}^2\right),\end{array}$

where Ψ is regularized L₁ norm $\Psi (s^2)=\sqrt{s^2+0.001^2}$ and n_x and n_y are normalization factors. For the smoothness term S(∇δp_i(p)), they use a regularizer based on total variation:

$\begin{array}{l}\hfill S(\nabla {\mathbf{u}}_i(p))=\Psi (|\nabla u_i(p){|}^2+|\nabla v_i(p){|}^2).\end{array}$

The energy equation (Eq. 3.41) is then optimized by a semi-implicit gradient descent scheme, and the final flows are used to warp each of the input images in the stack, which are then merged with use of the method of Robertson et al. (1999).

Later, Hu et al. (2012) proposed using the patch-based, nonrigid dense correspondence method of HaCohen et al. (2011) to compute dense correspondences between the reference image and the other images in the stack, called the source images. They then use this correspondence field to warp pixels in the source images to match the appearance of the reference. However, because of occlusions and disocclusions, as well as brightness changes, the correspondences are generally incomplete; this can result in “holes” (ie, regions where the pixels’ values are undefined).

To address this problem, Hu et al. first propose a robust strategy to estimate the IMFs (Grossberg and Nayar, 2003a) using the known pixel correspondences. For each hole in the warped source, they then attempt to paste the pixels from the original source image; however, to compensate for motion, they first apply a local projective transformation to align them with those in the reference. To ensure that the pasted pixels cause no artifacts, Hu et al. take a bounding box larger than the hole to be transformed and pasted. If the pixels within the box, but outside the hole, do not match, the region is considered to be affected by motion. In these cases, the pixels are pasted directly from the reference after their brightness values have been appropriately corrected with the estimated IMF.

Another method based on a flavor of optical flow is the approach of Gallo et al. (2015). The method is based on the observation that modern cameras offer fast bursts modes, which make arbitrarily large displacements unlikely. Therefore, instead of computing the optical flow at each pixel, they suggest computing it only at sparse locations, and then to propagate it to the rest of the pixels. Specifically, there are four stages to the algorithm.

Gallo et al. first describe a novel method to find and match corners across two images in the stack, one chosen to be the reference and the other being the source image. Their corners are based on the changes of average brightness around different pixels, which can be computed efficiently with integral images. The second stage identifies and removes matches that are either incorrect or belong to structures that move in a highly nonrigid fashion. To achieve this, Gallo et al. observe that good matches should be locally consistent with a homography, and propose a modification of the RANSAC algorithm to isolate those that are not. The set of matches that are both correct and rigid offers an estimate of the flow at sparse locations, which can be propagated to the rest of the pixels with use of the reference image as a guide to an edge-aware diffusion algorithm. With the dense flow, the source image can be warped to be geometrically consistent with the reference. However, to account for possible errors in the flow propagation, Gallo et al. modify the algorithm of Mertens et al. (2007), see Section 3.3.2.3, by adding a fourth weight. Specifically, they use the structural similarity index proposed by Wang et al. (2004) to account for the quality of the registration at different locations, and reduce the contribution of regions that are not correctly registered. For a pair of five-megapixel images, Gallo et al. report execution times of less than 150 ms on a desktop computer and less than 700 ms on a commercial tablet computer. For reference, the methods described in Section 3.4.3.2 are several orders of magnitude slower. As mentioned before, this large speedup is possible thanks to the observation that arbitrarily large displacements are unlikely when the stack is captured in a fast burst.

Compared with rejection-based approaches for HDR reconstruction, these alignment methods, which rely on correspondences between the different images in the stack, have the advantage that they can move content around. This allows them to handle dynamic objects with HDR illumination. However, finding reliable correspondences, especially in cases of complex motion and deformation, is quite difficult and can introduce new artifacts. These problems can largely be resolved by use of the patch-based synthesis methods discussed next.

3.4.3.2 Patch-based synthesis methods

The most successful kind of HDR deghosting algorithms are perhaps those that align the stack of images together by using patch-based synthesis to generate plausible images that are registered to the reference (Sen et al., 2012; Hu et al., 2013; Kalantari et al., 2013). Indeed, a recent state-of-the-art report on HDR deghosting techniques has shown that these algorithms produce the best results for general scenes (Tursun et al., 2015). These methods can also be considered a new kind of algorithm (different from the rejection and registration algorithms we have discussed) because they can solve for the aligned images and the HDR reconstruction simultaneously. Although patch-based synthesis had previously been shown to be very powerful for various computational imaging tasks (such as image hole filling (Wexler et al., 2007), image summarization and editing (Simakov et al., 2008; Barnes et al., 2010), morphing (Shechtman et al., 2010), and finding dense correspondences between images (HaCohen et al., 2011)), these new methods apply it to HDR reconstruction by posing the problem as an energy optimization.

To use patch-based synthesis for HDR reconstruction, the two independent methods of Sen et al. (2012) and Hu et al. (2013) make similar observations: after registration, each image Z_i from the stack should look as if it were taken at the same time as the reference Z_ref, but should be photometrically consistent with the original Z_i, thereby capturing all of the additional dynamic range information contained in the original image.

Sen et al. (2012) propose doing this using a new optimization equation that codifies the objective of reference-based HDR reconstruction algorithms: (1) to produce an HDR image that resembles the reference in portions where the reference is well exposed, and (2) to leverage well-exposed information from other images in the stack in places where the reference is not well exposed. This results in what they call the HDR image synthesis equation, which contains two terms:

$\begin{array}{l}\hfill \text{Energy}(E)=\sum _{p\in \text{pixels}}\left[{\alpha }_{\mathrm{ref}}(p)\cdot {\left(g(Z_{\mathrm{ref}}(p))/t_{\mathrm{ref}}-E(p)\right)}^2+(1-{\alpha }_{\mathrm{ref}}(p))\cdot E_{\mathrm{MBDS}}(H|L_1,\dots ,L_N)\right].\end{array}$

The first term states that the ideal HDR image E should be close in an L₂ sense to the LDR reference Z_ref mapped to the linear irradiance domain. This should be done only for the pixels where the reference is properly exposed, as given by the α_ref(p) term, which is a trapezoid function in the pixel intensity domain that favors intensities near the middle of the pixel value range.

In the parts where the reference image Z_ref is poorly exposed as indicated by the 1 − α_ref(p) term, the algorithm draws information from the other images in the stack using a novel multisource bidirectional equation E_MBDS that extends the bidirectional similarity metric of Simakov et al. (2008):

$\begin{array}{l}\hfill \text{BDS}(T|S)=\frac{1}{|S|}\sum _{P\in S}\underset{Q\in T}{min}d(P,Q)+\frac{1}{|T|}\sum _{Q\in T}\underset{P\in S}{min}d(Q,P).\end{array}$

The original function of Simakov et al. takes a pair of images (source S and target T) and ensures that all of the patches (small blocks of pixels) in S can be found in T (first term, called “completeness”) and vice versa (second term, called “coherence”). Note that the coherence term ensures that the final target does not contain objectionable artifacts, as these artifacts are not found in the original source.

However, Eq. (3.45) does not work for HDR reconstruction directly; sometimes content that should be visible in the ith exposure when “aligned” with the reference exposure might be occluded in Z_i and needs to be drawn from a different image. So rather than using a pairwise bidirectional similarity metric, Sen et al. introduce a multisource bidirectional similarity metric E_MBDS that draws information from all the images in the stack simultaneously.

To optimize Eq. (3.44), Sen et al. introduce auxiliary variables ${\tilde{Z}}_i$ that represent the different LDR images in the stack after they have been aligned with the reference. This equation can then be solved with an iterative, two-stage algorithm that solves for the ${\tilde{Z}}_1,\dots ,{\tilde{Z}}_N$ and E simultaneously:

Stage 1: The algorithm first solves for the aligned LDR images ${\tilde{Z}}_1,\dots ,{\tilde{Z}}_N$ with a bidirectional search-and-vote process (Simakov et al., 2008) accelerated by PatchMatch (Barnes et al., 2009). This process draws information into each of the aligned LDR images from the entire stack, which has been adjusted to match the corresponding exposure level. To produce images aligned with the reference, the irradiance image E from the previous iteration (which has been injected with the reference in step 2) is used as the initial target for the search-and-vote process.

Stage 2: Next, the algorithm optimizes for E by merging the aligned images ${\tilde{Z}}_1,\dots ,{\tilde{Z}}_N$ using a standard HDR merging process (Section 3.3) and then injects the portions of the reference image where it is well exposed into the result.

Once the new E has been computed, it is used to extract the new image targets for the next iteration, and the algorithm goes back to stage 1. These two stages are performed at every iteration of the algorithm until it converges. Furthermore, as is common for patch-based methods such as this (eg, Simakov et al., 2008), this core algorithm is performed at multiple scales, starting at the coarsest resolution and working to the finest. Once the algorithm has converged, it returns both the desired HDR image E and the “aligned” images at each exposure ${\tilde{Z}}_1,\dots ,{\tilde{Z}}_N$. A result produced with this algorithm is shown in Fig. 3.8. This algorithm was later extended by Kalantari et al. (2013) to reconstruct HDR video, as described in Chapter 4.

Hu et al. (2013) propose a different patch-based synthesis algorithm, which, unlike the algorithm of Sen et al., does not require the camera calibration curve to be known a priori. Specifically, they calculate the aligned images ${\tilde{Z}}_i$ as

$\begin{array}{l}\hfill {\tilde{Z}}_i=arg\underset{{\tilde{Z}}_i,\tau ,\mathbf{u}}{min}\left(C_{\mathrm{r}}({\tilde{Z}}_i,Z_{\mathrm{ref}},\tau )+C_{\mathrm{t}}({\tilde{Z}}_i,Z_i,\mathbf{u})\right),\end{array}$

where u is the displacement field that “warps” image Z_i to match the geometric appearance of the reference, and τ is the IMF between the source image Z_i and the reference Z_ref. In Eq. (3.46), the first term, C_r for “radiance consistency,” encourages the aligned image ${\tilde{Z}}_i$ to be geometrically consistent with the reference, Z_ref:

$\begin{array}{l}\hfill C_{\mathrm{r}}({\tilde{Z}}_i,Z_{\mathrm{ref}},\tau )=\sum _p\left(\parallel {\tilde{Z}}_i(p)-\tau (Z_{\mathrm{ref}}(p)){\parallel }^2+\alpha \parallel \nabla {\tilde{Z}}_i(p)-\nabla \tau (Z_{\mathrm{ref}}(p)){\parallel }^2\right).\end{array}$

Note that both the images and their gradients are accounted for in Eq. (3.47). The second term in Eq. (3.46), C_t, is what Hu et al. call the “texture consistency” term:

$\begin{array}{l}\hfill C_{\mathrm{t}}({\tilde{Z}}_i,Z_i,\mathbf{u})=\frac{1}{k}\sum _p\left(\parallel P_{\tilde{Z}}(p)-P_{Z_i}(p+\mathbf{u}(p)){\parallel }^2+\alpha \parallel \nabla (P_{\tilde{Z}}(p)-\nabla P_{Z_i}(p+\mathbf{u}(p)){\parallel }^2\right),\end{array}$

where k is a normalization factor. The texture consistency term enforces similarity between the patch around pixel p in the warped source, $P_{\tilde{Z}}(p)$, and the corresponding patch in the source image, $P_{Z_i}(p+\mathbf{u}(p))$. This helps enforce that the synthesized content is plausible and free of artifacts.

Hu et al. tackle this optimization iteratively using a coarse-to-fine approach, which helps in two ways. First, it prevents the optimization from falling in a local minimum. Second, it allows the algorithm to deal with large oversaturated or undersaturated regions: a patch that is entirely saturated at the finest level could include information from neighboring nonsaturated pixels at coarser levels, thus allowing information to propagate inward. To do this, they propose an iterative, three-stage algorithm:

Stage 1: First, they estimate τ using the intensity histograms of the images (Grossberg and Nayar, 2003a) at the coarsest level of the pyramid and initialize ${\tilde{Z}}_i=\tau (Z_i)$ for the same level. The displacement u, which appears only in C_t, can then be estimated with PatchMatch (Barnes et al., 2009).

Stage 2: In a second step, they propose refining the estimate of ${\tilde{Z}}_i$ by minimizing C_r. However, for the areas where the reference image is overexposed or underexposed, they average τ(Z_ref(p)) with the corresponding location in the source image Z_i(p + u(p)) with a weight that accounts for how likely the latter is to become overexposed or underexposed in the reference image.

Stage 3: In the third and last step, with the new ${\tilde{Z}}_i$, they refine the IMF, τ. Moving to the next finer level, they leave τ unchanged and linearly interpolate u. The latent image ${\tilde{Z}}_i$, instead, is initialized with a weighted average of τ(Z_ref) and Z_i(p + u(p)).

Results from this approach can be seen in Fig. 3.9.

As discussed earlier, these patch-based synthesis methods have the advantage that they work very well for scenes with complex, arbitrarily large motion where other algorithms would normally fail. However, they are expensive and require considerable time and hardware resources for evaluation: the reference implementations provided by Hu et al. take more than 1 min for VGA images. Furthermore, although they can produce plausible results, they are only hallucinating the final result as compared with the true HDR result that would have been captured by a hypothetical HDR camera.