2.1 HDR Video Compression Methods

In this section, the focus is on HDR video compression methods apart from JPEG HDR, which is a still image method extendable to video. Both the video and still image version are used in the results section. Banterle et al. [1] provides a comprehensive overview of HDR still image, video, and texture compression. As mentioned earlier, HDR video compression methods can be broadly classified into two groups: higher bit-depth single stream compression methods and backwards compatible double stream methods.

Compression methods following the first approach produce a single raw video stream which allocates higher bit-depth, that is, bit-depth >8, typically 10–14 for the luma channel, and 8 bits for the two chroma channels. Using a range of reversible transfer functions to convert 16-bit floating point luminance to n-bit luma where n ∈ [10, 14], this approach takes advantage of encoders that are able to support up to 14 bits/pixel/channel. Compression methods following this approach include Perception Motivated HDR video encoding [6], which converts real-world luminance values to 11-bit luma, using perceptual encoding [6], Temporally Coherent Luminance to Luma conversion for HDR video proposed [7], which uses a temporally coherent extension of adaptive logLUV encoding [8] to convert real-world luminance to 12-bit luma, and HVS based HDR video compression proposed by Zhang et al. [9] which converts real-world luminance values to 14-bit luma values using a nonlinear approach similar to Lloyd Max quantization [10]. The computation of chroma values in all the abovementioned approaches are similar to chroma encoding mentioned in logLUV [5].

A few drawbacks of these approaches when used in conjunction with state-of-the-art encoders are that the compressed video files cannot be played back using legacy video players due to the lack of higher bit-depth support, the maximum value of the chroma information, although limited to 8 bits, needs to be passed to the encoder as 11, 12, and 14 bits, respectively because encoders expect similar bit-depths for both luma and chroma and that most existing hardware-based encoders and decoders are limited to 8 bits. Therefore, wide-scale adoption of these compression methods in practice seems unlikely in the near future.

The second group consists of compression methods which split the input HDR data into primary and secondary streams, typically allocating 8 bits/pixel/channel to each of the streams. The primary stream, in a backwards-compatible double stream method is typically a tone-mapped LDR stream, which enables it to be played back using any legacy video player. The secondary stream consists of additional information that can be used to reconstruct the HDR frames. Compression methods following this approach include JPEG HDR [11], HDR MPEG [12], and Rate Distortion Optimized HDR video encoding [13].

JPEG HDR extends the widely used 8 bit/pixel/channel JPEG image format which uses subsampled additional information with precorrection and postcorrection techniques in order to reconstruct HDR frames on the decoder. Originally designed for still HDR images, a fairly straightforward modification can lead to an effective backwards-compatible HDR video compression method. The primary stream consists of LDR frames, created using any given tone mapper. The secondary stream consists of frames created by taking the ratio between the luminance of the input HDR frame and the luminance of the corresponding LDR frame such that $RI(x,y)=\frac{L_{\mathrm{hdr}}(x,y)}{L_{\mathrm{ldr}}(x,y)}$. The luminance of both the HDR and LDR frame are computed using the BT/REC. 709 primaries. The ratio frame is subsequently log encoded, discretized (RI(x, y) ∈ [0, 255]) and subsampled in order to minimize storage requirements.

HDR MPEG, much like JPEG HDR, splits the input HDR frame into two streams. The primary LDR stream is created using a given tone mapping operator and passed through encoder and decoder blocks. Subsequently, both the input HDR and decoded LDR frames and passed through color transformation functions in order to transform both HDR and LDR frames into a similar color space. This transformation helps in the creation of a monotonically increasing reconstruction function (RF), which is subsequently used to predict an HDR frame from its LDR counterpart. Finally, the residual luma differences between the input HDR frame and the predicted HDR frame is quantized using a quantization function and stored as the secondary stream. The quantization function (QF) ensures the residual image is always in unsigned integer range. The algorithm also outputs an auxiliary stream, which stores the RF and QF values per frame to be subsequently used during reconstruction.

The rate distortion optimized HDR video compression method follows a method very similar to JPEG-HDR. The primary stream is created using a temporally coherent TMO, proposed by Lee and Kim [14], essentially a temporally coherent extension of the Fattal TMO [15], which also adds the option of managing color saturation of the LDR frames; although as with the other methods, other tone mapping functions could be used. A log-encoded ratio frame is computed such that $RI(x,y)=\log (\frac{L_{\mathrm{hdr}}(x,y)}{L_{\mathrm{ldr}}(x,y)})$. Subsequently, the minima and maxima of each ratio frame is stored as an auxiliary data structure before the frames are scaled, such that RI(x, y) ∈ [0, 1] and filtered using a cross-bilateral filter [16] in order to reduce noise. Finally, the scaled and filtered ratio frames are discretized and passed to the encoder, and the auxiliary data structure is also stored as a separate file. On the decompression side, the primary frames and secondary frames are read back followed by a saturation correction of the primary frames and inverse scaling of the secondary frames, using information stored in the auxiliary data structure. Finally, the primary and secondary frames are multiplied to reconstruct the output HDR frames.

The primary advantage of the backwards compatible approaches is that the video files can be encoded and decoded using any legacy software/hardware based encoders and decoders, which means hardware support is easy to find and uptake can be quicker because part of the content can be successfully played on legacy LDR software and displays.

3.1 Encoding Method

Fig. 1 illustrates the general encoding approach used by pBC. This presents a robust, yet simple implementation of the generalized method described above. Two operations are performed initially, both aimed at fulfilling the variables in Eq. (1). Following the extraction of L(S_HDR) (luma stream) and the computation of S_LDR (backwards compatible stream) from Eq. (1), the two streams are prepared for legacy encoding, whereby any LDR encoder can be used to encode the two streams. pBC is based around computing the backwards compatible layer based on the following equation:

$\begin{array}{l}\hfill S_{\mathrm{LDR}}=\frac{S_{\mathrm{HDR}}}{L(S_{\mathrm{HDR}})+1},\end{array}$

where S_HDR is the original HDR image/frame and L() is a function that extracts the luminance of the HDR image/frame using any luminance computation equation such as L(S_HDR) = 0.2126 × S_HDR^R + 0.7152 × S_HDR^G + 0.0722 × S_HDR^B where S_HDR^R, S_HDR^G, and S_HDR^B are, respectively, the red, green, and blue channels of the HDR image/frame. The equation is applied directly to each pixel. This function is similar to the sigmoid functions used in other methods, but has the unique characteristic of directly computing the color channels through the direct use of the HDR content and its luminance.

The resulting image/frame, S_LDR is a full color image/frame that can be encoded with a legacy encoder and that can be viewed on a traditional display. S_LDR is encoded as a stream or image and L(S_HDR) is also encoded. L(S_HDR) can be log encoded to conform with the human visual system’s perception of brightness which follows a logarithmic scale or any other encoding mechanism that reduces the range. The resulting L(S_HDR) is also encoded using a traditional LDR legacy encoder. In the results presented in Section 4, log encoding of the luminance frame is used.

3.2 Channel Scaling

The dynamic range of S_LDR can, on occasion, be larger than 1, a function can be applied to ensure the range is maintained between 0 and 1. In the case of the results presented here, a further computation occurs to the S_LDR stream if the values of any of the channels of S_LDR exceed 1. This is computed as

$\begin{array}{l}\hfill {\mathrm{SC}}_{\mathrm{LDR}}=\frac{S_{\mathrm{LDR}}}{(1+S_{\mathrm{LDR}}\times \mathrm{clamp}((max(S_{\mathrm{LDR}})*\mathrm{IQR}(L(S_{\mathrm{HDR}}),0,1))},\end{array}$

where SC_LDR is the corrected LDR data, which will not exceed the value of 1. The clamp(value, min, max) function is used to clamp results to 1 when required. IQR(X) computes the interquartile range of the computed HDR luminance channel, and as defined as follows:

$\begin{array}{l}\hfill \mathrm{IQR}(X)=\mathrm{median}(X_R)-\mathrm{median}(X_L),\end{array}$

where median() computes the median of a set of data, and X_R is the subset of X, which take values greater than median(X). X_L is defined analogously. If this step is applied, the corrected data SC_LDR is stored as the LDR part instead of S_LDR.

3.3 Decoding Method

The decoding method is illustrated in Fig. 2 for viewers/displays that support HDR video and Fig. 3 for LDR viewers/displays. The main difference between the two is that when decoding for HDR viewers, the backwards compatible stream is enhanced with the luma stream, while for the LDR viewers the backwards compatible stream is played directly.

When enhancing the backwards compatible stream with the luma stream, Eq. (1) is inverted to reconstruct the HDR stream as S_HDR^DEC = S_LDR × (L(S_HDR) + 1). If required, Eq. (3) is also inverted and computed prior to inverting Eq. (1). When the luma stream is log encoded (as is the case for the results presented in the next section), then this is log decoded prior to computing any equations. Overall, the decoding process is efficient and easy to implement, making it amiable to hardware implementations.

4.1 Implementation

All the methods were implemented in Matlab using the same framework for most intermediate routines, not specific to an individual method. For each method, two LDR streams are produced and the content of these is explained for each method below. pBC is a direct implementation of the method described in Section 3. The main base stream of pBC contains the backwards compatible stream and the extension stream the luminance. The secondary stream is log encoded to account for the human visual systems response to luminance. JPEG HDRv is a video implementation of the JPEG HDR method [22], as it was seen to produce very good results for video. The method chosen for tone mapping was the Reinhard tone mapper [23], as it has been shown to perform well in psychophysics experiments [17]. The ratio stream is not limited in size, as is the case for still images in JPEG HDR, although the streams produced are typically very small; it is log encoded, as was the case of the extension layer for pBC. HDR MPEG makes use of a temporally-coherent video tone mapper [26], as our implementation using the traditional Photographic Tone Reproduction operator was found to produce poor results for this method. Rate Distortion [13] uses the temporal tone mapper used in the original paper [14]. The bit rate optimization introduced in this method is not employed, as the idea was to compare methods; arguably this optimization could be employed by other methods also. As mentioned above, Inverse (Reinhard) uses the Photographic Tone Mapping Reproduction operator and its inverse. The same sigmoid that inspired pBC is used for Inverse (Sigmoid). The secondary stream for both inverse methods consists of a residual computed from a reconstructed HDR frame’s difference from the original.

4.2 Method and Results

The HDR video compression comparison is based around Fig. 5. All HDR frames were encoded with one of the chosen methods, a YUV file was generated and a traditional legacy encoder was used to compress the two YUV files into legacy streams. High Efficiency Video Encoding (HEVC) was chosen as the means of legacy encoding for this set of comparisons, as it represents the upcoming standard for traditional video compression. The x265 application was used to represent HEVC. The settings were set as default maximum quality compression settings (very slow). Each method was encoded with different quantization parameters for each scene with settings of 40, 35, 30, 25, 20, 15, 10, 5, 2, 1. Once encoded, streams were decoded into YUV streams, which were then converted into streams that were decoded by the various HDR decoding methods, producing a series of HDR frames. The resultant HDR frames were compared with the original frames using the three metrics. Bitrates of the resultant encoded streams were used for computing output bitrates used in the results. The YUV subsampling format of 4:4:4 was chosen with 8-bit depth for both streams. The QP slices were also set equally for both streams. Results are subsequently averaged across all six scenes for each of the metrics.

Fig. 6 shows results for the three metrics. pBC and HDR MPEG perform best overall across all three metrics; pBC performs better than HDR MPEG at PSNR for lower bitrates, and with all metrics at middle bitrates, and HDR MPEG performs best for higher bitrates. JPEG HDRv also performs well, as does Rate Distortion. Rate Distortion would probably perform better with the optimization method included. Both Inverse methods perform relatively poorly. Section 4.4 provides further analysis of the results.

4.3 Still Images

pBC could also be used for the compression of still HDR images. To demonstrate the approach, the same method used by Ward and Simmons for JPEG HDR [22] is adopted. In this method, a ratio image is encoded in the subband of the JPEG image, and the tone mapped image is stored in the body of the JPEG container. The ratio image is decreased to fit within the subband, typically reduced to <64 KB. pBC was adapted to use the same approach for the results presented in this section. The luminance content of the method is reduced to <64 KB and is stored in the subband. Results are presented for pBC, versus the original JPEG HDR method in Fig. 7 for the 20 images shown if Fig. 8 averaged across all 20 images for the previously used metrics. The images shown in Fig. 7 were all computed with pBC. Bits per pixel (bpp) were computed from the resultant encoded images. Quality parameters were set to 1, 4, 16, 32, 48, 64, 80, 96, and a final no compression value represented as 100. As with JPEG HDRv, the Reinhard tone mapper was used for JPEG HDR. Results again demonstrate a good performance for pBC overall.

4.4 Discussion

While details of the methods may vary in terms of implementations, parameters used, tone mappers used, etc., which may modify the results, pBC performs relatively well for a straightforward method, and indicates that there is scope for methods that are unsophisticated. The Inverse methods demonstrate that this straightforward approach is not enough in isolation, and care must be taken with the design of a practical encoding method, which pBC has provided. HDR MPEG, JPEG HDR (and JPEG HDRv), and Rate Distortion, as well as the Inverse methods have the advantage of being able to use different tone mapping operators and therefore, offer more flexibility, but some of this comes at an added cost, potentially in the tone mappers used, for example, local tone mappers may be quite expensive and in the use of complex operators, for example, Rate Distortion makes use of a cross bilateral filter [16]. The Inverse methods lose some of the flexibility, as they require methods that are invertible. Issues like computational complexity play an important role in HDR transmission, which will be a requirement for HDR to succeed in the broadcast domain. With this plethora of options, we believe pBC has an opportunity to succeed as a practical straightforward method that produces good consistent results with little complexities.