1.2.1.1 Gauss Function

The zero-mean Gauss function is defined as follows:

1.2

A convolution of an image with the Gauss function produces smoothed images

1.3

also known as Gaussians, for . (We stay with symbol here as introduced by Lindeberg (1994) for “layer”; a given context will prevent confusion with the left image of a stereo pair.)

1.2.1.2 Edges

Step-edges in images are detected based on first- or second-order derivatives, such as values of the gradient or the Laplacian given by

1.4

Local maxima of - or -magnitudes or , or zero-crossings of values are taken as an indication for a step-edge. The gradient or Laplacian is commonly preceded by smoothing, using a convolution with the zero-mean Gauss function.

Alternatively, Phase-congruency edges in images are detected based on local frequency-space representations (Kovesi 1993).

1.2.1.3 Corners

Let , , , and denote the second-order derivatives of image . Corners in images are localized based on high curvature of intensity values, to be identified by two large eigenvalues of the Hessian matrix

1.5

at a pixel location in a scalar image (see Harris and Stephens (1988)). Figure 1.1 shows the corners detected by FAST. Corner detection is often preceded by smoothing using a convolution with the zero-mean Gauss function.

1.2.1.4 Scale Space and Key Points

Key points or interest points are commonly detected as maxima or minima in a subset of the scale space of a given image (Crowley and Sanderson 1987; Lindeberg 1994). A finite set of differences of Gaussians

1.6

produces a DoG scale space. These differences are approximations to Laplacians of increasingly smoothed versions of an image (see Figure 1.5 for an example of such Laplacians forming an LoG scale space).

1.2.1.5 Features

An image feature is finally a location (an interest point), defined by a key point, edge, corner, and so on, together with a descriptor, usually given as a data vector (e.g., in case of scale-invariant feature transform (SIFT) of length 128 representing local gradients), but possibly also in other formats such as a graph. For example, the descriptor of a step-edge can be mean and variance of gradient values along the edge, and the descriptor of a corner can be defined by the eigenvalues of the Hessian matrix.

1.2.2.1 Pinhole-type Camera

The -axis models the optical axis. Assuming an ideal pinhole-type camera, we can ignore radial distortion and can have undistorted projected points in the image plane with coordinates and . The distance between the image plane and the projection center is the focal length.

A visible point in the world is mapped by central projection into pixel location in the undistorted image plane:

1.7

with the origin of image coordinates at the intersection point of the -axis with the image plane.

The intersection point of the optical axis with the image plane in coordinates is called the principal point. It follows that . A pixel location in the 2D image coordinate system has 3D coordinates in the camera coordinate system.

1.2.2.2 Intrinsic and Extrinsic Parameters

Assuming multiple cameras , for some indices (e.g., just and for binocular stereo), camera calibration specifies intrinsic parameters such as edge lengths and of camera sensor cells (defining the aspect ratio), a skew parameter , coordinates of the principal point where optic axis of camera and image plane intersect, the focal length , possibly refined as and , and lens distortion parameters starting with and . In general, it can be assumed that lens distortion has been calibrated before and does not need to be included anymore in the set of intrinsic parameters. Extrinsic parameters are defined by rotation matrices and translation vectors, for example, matrix and vector for the affine transform between camera coordinate systems and , or matrix and vector for the affine transform between camera coordinate system and .

1.2.2.3 Single-Camera Projection Equation

The camera projection equation in homogeneous coordinates, mapping a 3D point into image coordinates of the th camera, is as follows:

1.8

1.9

where is a scaling factor. This defines a matrix of intrinsic camera parameters and a matrix of extrinsic parameters (of the affine transform) of camera . The camera matrix is defined by 11 parameters if we allow for an arbitrary scaling of parameters; otherwise it is 12.

1.2.3.1 Optimizing a Labeling Function

Labels are assigned to all the pixels in the carrier by a labeling function . Solving a labeling problem means to identify a labeling that approximates somehow an optimum of a defined error or energy

1.10

where is a weight. Here, is the data-cost term and is the smoothness-cost term. A decrease in works toward reduced smoothing of calculated labels. Ideally, we search for an optimal (i.e., of minimal total error) in the set of all possible labelings, which defines a total variation (TV).

We detail Eq. (1.10) by adding costs at pixels. In a current image, label is assigned by the value of labeling function at pixel position . Then we have that

1.11

where is an adjacency relation between pixel locations.

In optical flow or stereo vision, label (i.e., optical flow vector or disparity) defines a pixel in another image (i.e., in the following image, or in the left or right image of a stereo pair); in this case, we can also write instead of .

1.2.3.2 Invalidity of the Intensity Constancy Assumption

Data-cost terms are defined for windows that are centered at the considered pixel locations. The data in both windows, around the start pixel location , and around the pixel location in the other image, are compared for understanding “data similarity.”

For example, in the case of stereo matching, we have in the right image and in the left image , for disparity , and the data in both windows are identical if and only if the data-cost measure

1.12

results in value 0, where SSD stands for sum of squared differences.

The use of such a data-cost term would be based on the intensity constancy assumption (ICA), that is, intensity values around corresponding pixel locations and are (basically) identical within a window of specified size. However, the ICA is invalid for real-world recording. Intensity values at corresponding pixels and in their neighborhoods are typically impacted by lighting variations, or just by image noise. There are also impacts of differences in local surface reflectance, differences in cameras when comparing images recorded by different cameras, or effects of perspective distortion (the local neighborhood around a surface point is differently projected into different cameras). Thus, energy optimization needs to apply better data measures compared to SSD, or other measures are also defined based on the ICA.

1.2.3.3 Census Data-Cost Term

The census-cost function has been identified as being able to compensate successfully bright variations in input images of a recorded video (Hermann and Klette 2009; Hirschmüller and Scharstein 2009). The mean-normalized census-cost function is defined by comparing a window centered at pixel location in frame with a window of the same size centered at a pixel location in frame . Let be the mean of the window around for or . Then we have that

1.13

with

1.14

Note that value 0 corresponds to consistency in both comparisons. If the comparisons are performed with respect to values and , rather than the means and , then we have the census-cost function as a candidate for a data-cost term.

Let be the vector listing results in a left-to-right, top-to-bottom order (with respect to the applied window), where is the signum function; lists values . The mean-normalized census data-cost equals the Hamming distance between vectors and .

1.3.1.1 Stereo Vision

We address the detection of corresponding points in a stereo image , a basic task for distance calculation in vehicles using binocular stereo.

Corresponding pixels define a disparity, which is mapped based on camera parameters into distance or depth. There are already very accurate solutions for stereo matching, but challenging input data (rain, snow, dust, sunstroke, running wipers, and so forth) still pose unsolved problems (see Figure 1.6 for an example of a depth map).

1.3.1.2 Binocular Stereo Vision

After camera calibration, we have two virtually identical cameras and , which are perfectly aligned defining canonical stereo geometry. In this geometry, we have an identical copy of the camera on the left translated by base distance along the -axis of the camera coordinate system of the left camera. The projection center of the left camera is at and the projection center of the cloned right camera is at . A 3D point is mapped into undistorted image points

1.15

1.16

in the left and right image planes, respectively. Considering and in homogeneous coordinates, we have that

1.17

for the 3 3 bifocal tensor , defined by the configuration of the two cameras. The dot product defines an epipolar line in the image plane of the right camera; any stereo point corresponding to needs to be on that line.

1.3.1.3 Binocular Stereo Matching

Let be the base image and be the match image. We calculate corresponding pixels and in the image coordinates of carrier following the optimization approach as expressed by Eq. (1.11). A labeling function assigns a disparity to pixel location , which specifies a corresponding pixel .

For example, we can use the census data-cost term as defined in Eq. (1.13), and for the smoothness-cost term, either the Potts model, linear truncated cost, or quadratic truncated costs is used (see Chapter 5 in Klette (2014)). Chapter 6 of Klette (2014) discusses also different algorithms for stereo matching, including belief-propagation matching (BPM) (Sun et al. 2003) and dynamic-programming stereo matching (DPSM). DPSM can be based on scanning along the epipolar line only using either an ordering or a smoothness constraint, or it can be based (for symmetry?) on scanning along multiple scanlines using a smoothness constraint along those lines; the latter case is known as semi-global matching (SGM) if multiple scanlines are used for error minimization (Hirschmüller, 2005). A variant of SGM is used in Daimler's stereo vision system, available since March 2013 in their Mercedes cars (see also Chapter 2 by U. Franke in this book).

Iterative SGM (iSGM) is an example for a modification of baseline SGM; for example, error minimization along the horizontal scanline should in general contribute more to the final result than optimization along other scanlines (Hermann and Klette, 2012). Figure 1.7 also addresses confidence measurement; for a comparative discussion of confidence measures, see Haeusler and Klette (2012). Linear BPM (linBPM) applies the MCEN data-cost term and the linear truncated smoothness-cost term (Khan et al. 2013).

1.3.1.4 Performance Evaluation of Stereo Vision Solutions

Figure 1.8 provides a comparison of iSGM to linBPM on four frame sequences each of 400 frames length. It illustrates that iSGM performs better (with respect to the used measure, see the following section for its definition) on the bridge sequence that is characterized by many structural details in the scene, but not as good as linBPM on the other three sequences. For sequences dusk and midday, both performances are highly correlated, but not for the other two sequences. Of course, evaluating on only a few sequences of 400 frames each is insufficient for making substantial evaluations, but it does illustrate performance.

The diagrams in Figure 1.8 are defined by the normalized cross-correlation (NCC) between a recorded third-frame sequence and a virtual sequence calculated based on the stereo matching results of two other frame sequences. This third-eye technology (Morales and R 2009) also uses masks such that only image values are compared which are close to step-edges (e.g., see Figure 1.5 for detected edges at bright pixels in LoG scale space) in the third frame. It enables us to evaluate performance on any calibrated trinocular frame sequence recorded in the real world.

1.3.2.1 Dense or Sparse Motion Analysis

Dense motion analysis aims at calculating approximately correct motion vectors for “basically” every pixel location in frame (see Figure 1.10 for an example). Sparse motion analysis is designed for having accurate motion vectors at a few selected pixel locations.

Motion analysis is a difficult 2D correspondence problem, and it might become easier by having recorded high-resolution images at a higher frame rate in future. For example, motion analysis is approached by a single-module solution by optical flow calculation, or as a multi-module solution when combining image segmentation with subsequent estimations of motion vectors for image segments.

1.3.2.2 Optical Flow

Optical flow is the result of dense motion analysis. It represents motion vectors between corresponding pixels in frames and . Figure 1.10 shows the visualization of an optical flow map.

1.3.2.3 Optical Flow Equation and Image Constancy Assumption

The derivation of the optical flow equation (Horn and Schunck 1981)

1.18

for and first-order derivatives , , and follows from the ICA, that is, by assuming that corresponding 3D world points are represented in frame and by the same intensity. This is actually not true for computer vision in vehicles. Light intensities change frequently due to lighting artifacts (e.g., driving below trees), changing angles to the Sun, or simply due to sensor noise. However, the optical flow equation is often used as a data-cost term in an optimization approach (minimizing energy as defined in Eq. (1.10)) for solving the optical flow problem.

1.3.2.4 Examples of Data and Smoothness Costs

If we accept Eq. (1.18)due to Horn and Schunck (and thus the validity of the ICA) as data constraint, then we derive

1.19

as a possible data-cost term for any given time .

We introduced above the zero-mean-normalized census-cost function . The sum can replace in an optimization approach as defined by Eq. (1.10) (see Hermann and Werner (2013)). This corresponds to the invalidity of the ICA for video data recorded in the real world.

For the smoothness-error term, we may use

1.20

This smoothness-error term applies squared penalties to first-order derivatives in the sense. Applying a smoothness term in an approximate sense reduces the impact of outliers (Brox et al. 2004).

Terms and define the optimization problem as originally considered by Horn and Schunck (1981).

1.3.2.5 Performance Evaluation of Optical Flow Solutions

Apart from using data with provided ground truth (see EISATS and KITTI and Figure 1.4), there is also a way for evaluating calculated flow vectors on recorded real-world video assuming that the recording speed is sufficiently large; for calculated flow vectors for frames and , we calculate an image “half-way” using the mean of image values at corresponding pixels and we compare this calculated image with frame (see Szeliski (1999)). Limitations for recording frequencies of current cameras make this technique not yet practically appropriate, but it is certainly appropriate for fundamental research.

1.3.3.1 Measures for Object Detection

Let tp or fp denote the numbers of true-positives or false-positives, respectively. Analogously we define tn and fn for the negatives; tn is not a common entry for performance measures.

Precision (PR) is the ratio of true-positives compared to all detections. Recall (RC) (or sensitivity) is the ratio of true-positives compared to all potentially possible detections (i.e., to the number of all visible objects):

1.21

The miss rate (MR) is the ratio of false-negatives compared to all objects in an image. False-positives per image (FPPI) is the ratio of false-positives compared to all detected objects in an image:

1.22

In case of multiple images, the mean of measures can be used (i.e., averaged over all the processed images).

How to decide whether a detected object is true-positive? Assume that objects in images have been locally identified manually by bounding boxes, serving as the ground truth. All detected objects are matched with these ground-truth boxes by calculating ratios of areas of overlapping regions

1.23

where denotes the area of a region in an image, is the detected bounding box of the object, and is the area of the bounding box of the matched ground-truth box. If is larger than a threshold , say , then the detected object is taken as a true-positive.

1.3.3.2 Object Tracking

Object tracking is an important task for understanding the motion of a mobile platform or of other objects in a dynamic environment. The mobile platform with the installed system is also called the ego-vehicle whose ego-motion needs to be calculated for understanding the movement of the installed sensors in the three-dimensional (3D) world.

Calculated features in subsequent frames can be tracked (e.g., by using RANSAC for identifying an affine transform between feature points) and then used for estimating ego-motion based on bundle adjustment. This can also be combined with another module using nonvisual sensor data such as GPS or of an inertial measurement unit (IMU). For example, see Geng et al. (2015) for an integration of GPS data.

Other moving objects in the scene can be tracked using repeated detections or by following a detected object in frame to frame . A Kalman filter (e.g., linear, general, or unscented) can be used for building a model for the motion as well as for involved noise. A particle filter can also be used based on extracted weights for potential moves of a particle in particle space. Kalman and particle filters are introduced, with references to related original sources, in Klette (2014).

1.3.4.1 Environment Analysis

There are static (i.e., fixed with respect to the Earth) or dynamic objects in a scenario which need to be detected, understood, and possibly further analyzed.

A flying helicopter (or just multi-copter) should be able to detect power lines or other potential objects defining a hazard. Detecting traffic signs or traffic lights, or understanding lane borders of highways or suburban roads are examples for driving vehicles. Boats need to detect buoys and beacons.

Pedestrian detection became a common subject for road-analysis projects. After detecting a pedestrian on a pathway next to an inner-city road, it would be helpful to understand whether this pedestrian intends to step onto the road in the next few seconds.

After detecting more and more objects, we may have the opportunity to model and understand a given environment.

1.3.4.2 Performance Evaluation of Semantic Segmentation

There is a lack of provided ground truth for semantic segmentations in traffic sequences. Work reported in current publications on semantic segmentation, such as Floros and Leibe (2012) and Ohlich et al. (2012), can be used for creating test databases. There is also current progress in available online data; see www.cvlibs.net/datasets/kitti/eval_road.php, www.cityscapes-dataset.net, and (Ros et al. 2015) for a study for such data.

Barth et al. (2010) proposed a method for segmentation, which is based on evaluating pixel probabilities of whether they are in motion in the real world or not (using scene flow and ego-motion). Barth et al. (2010) also provides ground truth for image segmentation in Set 7 of EISATS, illustrated by Figure 1.12. Figure 1.12 also shows resulting SGM stereo maps andsegments obtained when following the multi-module approach briefly sketched earlier.

Modifications in the involved modules for stereo matching and optical flow calculation influence the final result. There might be dependencies between performances of contributing programs.