Definition 5.1

The parameter set is said to be KLID-identifiable at , if and only if , , implies .

By the definition, if parameter set is KLID-identifiable at (we also say is KLID-identifiable), then for any , , we have , and hence . Therefore, any change in the parameter yields changes in the output density.

The identifiability can also be defined in terms of the information divergence rate.

Definition 5.2

The parameter set is said to be KLIDR-identifiable at , if and only if , the KL information divergence rate (KLIDR) exists, and implies .

Let be the -neighborhood of , where denotes the Euclidean norm. The local KLID (or local KLIDR)-identifiability is defined as follows.

Definition 5.3

The parameter set is said to be locally KLID (or locally KLIDR)-identifiable at , if and only if there exists , such that , (or ) implies .

Here, we give some assumptions that will be used later on.

Assumption 5.1

, , the KLID always exists.

Remark:

Let be the probability space of the output sequence with parameter , where is the related measurable space, and is the probability measure. is said to be absolutely continuous with respect to , denoted by , if for every such that . Clearly, the existence of implies . Thus by Assumption 5.1, , we have , .

Assumption 5.2

The density function is at least twice continuously differentiable with respect to , and , the following interchanges between integral (or limitation) and derivative are permissible:

(5.3)

(5.4)

Remark:

The interchange of differentiation and integration can be justified by bounded convergence theorem for appropriately well-behaved PDF . Similar assumptions can be found in Ref. [220]. A sufficient condition for the permission of interchange between differentiation and limitation is the uniform convergence of the limitation in .

Theorem 5.1

Assume that is an open subset of . Then, will be locally KLID-identifiable if the FIM is positive definite.

Proof:

As is an open subset, an obvious sufficient condition for to be locally KLID-identifiable is that , and . This can be easily proved. By Assumption 5.2, we have

(5.7)

On the other hand, we can derive

(5.8)

Theorem 5.2

Assume that is an open subset of . Then will be locally KLIDR-identifiable if the FIRM is positive definite.

Proof:

By Theorem 5.1 and Assumption 5.2, we have

(5.9)

and

(5.10)

Thus, is locally KLIDR-identifiable.

Suppose the observation sequence () is a stationary zero-mean Gaussian process, with power spectral . According to Theorem 2.7, the spectral expressions of the KLIDR and FIRM are as follows:

(5.11)

(5.12)

In this case, we can easily verify that , and . In fact, we have

(5.13)

and

(5.14)

where .

Remark:

Theorems 5.1 and 5.2 indicate that, under certain conditions, the positive definiteness of the FIM (or FIRM) provides a sufficient condition for the local KLID (or local KLIDR)-identifiability.

Example 5.1

Consider the following second-order state-space model () [120]:

(5.19)

where is a zero-mean white Gaussian process with unit power. Then the output sequence with is

(5.20)

It is easy to obtain the RCV:

(5.21)

The Jacobian matrix can then be calculated as:

(5.22)

Clearly, we have for all with . So this parameterization is locally KLID-identifiable provided . The identifiability can also be checked from the transfer function. The transfer function of the above system is:

(5.23)

, , define . Then , we have provided the following two conditions are met:

According to the Definition 1 of Ref. [219], this system is also locally identifiable from the transfer function provided .

The KLID-identifiability also has connection with the LS-identifiability [217]. Consider the signal-plus-noise model:

(5.24)

where is a parameterized deterministic signal, is a zero-mean white Gaussian noise, ( is an -dimensional identity matrix), and is the noisy observation. Then we have

(5.25)

By Eq. (5.16), we derive

(5.26)

where . The above KLID is equivalent to the LS criterion of the deterministic part. In this case, the KLID-identifiability reduces to the LS-identifiability of the deterministic part.

Next, we show that for a stationary Gaussian process, the KLIDR-identifiability is identical to the identifiability from the output spectral density [219].

Let () be a parameterized zero-mean stationary Gaussian process with continuous spectral density (-dimensional matrix). By Theorem 2.7, the KLIDR between and exists and is given by:

(5.27)

Theorem 5.3

, with equality if and only if , .

Proof:

, the spectral density matrices and are positive definite. Let and be two normally distributed -dimensional vectors, and , respectively. Then we have

(5.28)

Combining Eqs. (5.28) and (5.27) yields

(5.29)

It follows easily that , with equality if and only if for almost every (hence , ).

By Theorem 5.3, we may conclude that for a stationary Gaussian process, is KLIDR-identifiable if and only if , implies . This is exactly the identifiability from the output spectral density.

Definition 5.4

Assume that is KLID-identifiable. Then the MIH is [120]:

(5.30)

where .

The MIH is the minimum length of the observation sequence from which can be uniquely identified. If the MIH is known, we could identify with the least observation data. In general, it is difficult to obtain the exact value of MIH. In some special situations, however, one can derive an upper bound on the MIH. For a parameterized Markov process, this upper bound is straightforward. In the theorem below, we show that for a -order strictly stationary Markov process, the number provides an upper bound on the MIH.

Theorem 5.4

If the observation sequence () is a -order strictly stationary Markov process (), and the parameter set is KLID-identifiable at , then we have .

Proof:

As parameter set is KLID-identifiable at , by Definition 5.1, there exists a number , such that , . Let us consider two cases, one for which and the other for which .

Example 5.2

Consider the first-order AR model () [120]:

(5.36)

where is a zero-mean white Gaussian noise with unit power. Assume that the system has reached steady state when the observations begin. The observation sequence will be a first-order stationary Gaussian Markov process, with covariance matrix:

(5.37)

(, ), we have

(5.38)

where are the RCVs. And hence, .

The following corollary is a direct consequence of Theorem 5.4.

Corollary 5.1

For a -order strictly stationary Markov process , the parameter set is KLID-identifiable at if and only if , implies .

From the theory of stochastic process, for a -order strictly stationary Markov process , under certain conditions (see Ref. [221] for details), the conditional density will determine uniquely the joint density . In this case, the KLID-identifiability and the KLIDR-identifiability are equivalent.

Theorem 5.5

Assume that the observation sequence is a -order strictly stationary Markov process (), whose conditional density uniquely determines the joint density . Then, , is KLID-identifiable if and only if it is KLIDR-identifiable.

Proof:

We only need to prove .

Definition 5.5

The parameter set is said to be asymptotic KLID-identifiable at , if there exists a sequence of density estimates , such that (convergence in probability), where is the minimum KLID estimator, .

Theorem 5.6

Assume that the parameter space is a compact subset, and the density estimate sequence satisfies . Then, will be asymptotic KLID-identifiable provided it is KLID-identifiable.

Proof:

Since , for and arbitrarily small, there exists an such that for ,

Theorem 5.7

If is asymptotic KLID-identifiable, then it is KLID-identifiable provided

Proof:

If is asymptotic KLID-identifiable, then for and arbitrarily small, there exists an such that for ,

Theorem 5.10

Assume that and satisfy:

Then, the mean value of the optimal error PDF () is .

Proof:

Using reduction to absurdity. Suppose , and let . Denote and . According to the assumptions, , and is an even function. Then we have

Theorem 5.11

Let the reference PDF be a zero-mean Gaussian PDF with variance . If the error PDF satisfies

(5.70)

where is a positive constant, then the error entropy satisfies

(5.71)

where is the solution of the following equation:

(5.72)

Proof:

Denote the collection of all the error PDFs that satisfy . Clearly, this is a convex set. , we have

(5.73)

In order to solve the error distribution that achieves the maximum entropy, we create the Lagrangian:

(5.74)

where and are the Lagrange multipliers. When , is a concave function of . If is a function such that for sufficiently small, the Gateaux derivative of with respect to is given by:

(5.75)

If it is zero for all , we have

(5.76)

Thus, if (such that is a concave function of ), the error PDF that achieves the maximum entropy exists and is given by

(5.77)

According to the assumptions, . It follows that

(5.78)

where . Obviously, is a Gaussian density, and we have

(5.79)

So can be determined as

(5.80)

In order to determine the value of , we use the Kuhn–Tucker condition:

(5.81)

When , we have , that is,

(5.82)

Therefore, is the solution of the Eq. (5.72).

Define the function . It is easy to verify that is continuous and monotonically decreasing over interval . Since , , and , the equation certainly has a solution in .

From the previous derivations, one may easily obtain:

(5.83)

The above theorem indicates that, under the KLID constraint , the error entropy is upper bounded by the reference entropy plus a certain constant. In particular, when , we have and . Therefore, if one chooses a reference PDF with small entropy, the error entropy will also be confined within small values. In practice, the reference PDF is, in general, chosen as a PDF with zero mean and small entropy (e.g., the distribution at zero).

In most practical situations, the error PDF is unknown and needs to be estimated from samples. There is always a bias in the density estimation; in order to offset the influence of the bias, one can use the same method to estimate the reference density based on the samples drawn from the reference PDF. Let and be respectively the actual and reference error samples. The KDEs of and will be

(5.84)

where and are corresponding kernel widths. Using the estimated PDFs, one may obtain the empirical KLID criterion .

Theorem 5.12

The empirical KLID , with equality if and only if .

Proof:

, we have , with equality if and only if . And hence

(5.85)

with equality if and only if .

Lemma 5.3

If two PDFs and are bounded, then [231]:

(5.86)

where .

Theorem 5.13

If is a Gaussian kernel function, , then

(5.87)

Proof:

Since Gaussian kernel is bounded, and , the kernel-based density estimates and will also be bounded, and

(5.88)

By Lemma 5.3, we have

(5.89)

where

(5.90)

Then we obtain Eq. (5.87).

The above theorem suggests that convergence in KLID ensures the convergence in distance ().

Before giving Theorem 5.14, we introduce some notations. By rearranging the samples in and , one obtains the increasing sequences:

(5.91)

where , . Denote , and .

Theorem 5.14

If is a Gaussian kernel function, , then if and only if .

Proof:

By Theorem 5.12, it suffices to prove that if and only if .

Sufficiency: If , we have

(5.92)

Necessity: If , then we have

(5.93)

Let , where . Then,

(5.94)

where , , . Since , , we have . It follows that

(5.95)

As is a symmetric and positive definite matrix (), we get , that is

(5.96)

Thus, , and

(5.97)

This completes the proof.

Theorem 5.14 indicates that, under certain conditions the zero value of the empirical KLID occurs only when the actual and reference sample sets are identical.

Based on the sample sets and , one can calculate the empirical distribution:

(5.98)

where , and . According to the limit theorem in probability theory [224], we have

(5.99)

If , and is large enough, we have , and hence . Therefore, when the empirical KLID approaches zero, the actual error PDF will be approximately identical with the reference PDF.

Remark:

Strictly speaking, if one selects the function as the reference distribution, the information divergence will be undefined (ill-posed). In practical applications, however, we often use the estimated information divergence as an alternative cost function, where the actual and reference error distributions are both estimated by KDE approach (usually with the same kernel width). It is easy to verify that, for the distribution, the estimated PDF is actually the kernel function. In this case, the estimated divergence will always be valid.

Example 5.3

Consider the FIR system identification [126]:

(5.105)

where the true weight vector . The input signal is assumed to be a zero-mean white Gaussian process with unit power.

We show that the optimal solution under information divergence criterion may be not unique. Suppose the reference PDF is Gaussian PDF with zero mean and variance . The J-information divergence between and can be calculated as:

(5.106)

Clearly, there are infinitely many weight pairs that satisfy . In fact, any weight pair that lies on the circle will be an optimal solution. In this case, the system parameters are not identifiable. However, when , the circle will shrink to a point and all the solutions will converge to a unique solution . For the case , the 3D surface of the J-information divergence is depicted in Figure 5.2. Figure 5.3 draws the convergence trajectories of the weight pair learned by the SIDG algorithm, starting from the initial point . As expected, these trajectories converge to the circles centered at . When , the weight pair will converge to (1.0047, 0.4888), which is very close to the true weight vector.

Example 5.4

Identification of the hybrid system (switch system) [125]:

(5.107)

where is the state variable, is the input, is the process noise, and is the measurement noise. This system can be written in a parameterized form ( merging into ) [125]:

(5.108)

where is the mode index, , , , and . In this example, and . Based on the parameterized form (Eq. (5.108)), one can establish the noisy hybrid decoupling polynomial (NHDP) [125]. By expanding the NHDP and ignoring the higher-order components of the noise, we obtain the first-order approximation (FOA) model. In Ref. [125], the SIDG algorithm (based on the FOA model) was applied to identify the above hybrid system. Figure 5.4 shows the identification performance for different measurement noise powers . For comparison purpose, we also draw the performance of the least mean square (LMS) and algebraic geometric approach [232]. In Figure 5.4, the identification performance is defined as:

(5.109)

In the simulation, the function is selected as the reference PDF for SIDG algorithm. Simulation results indicate that the SIDG algorithm can achieve a better performance.

To further verify the performance of the SIDG algorithm, we consider the case in which and is uniformly distributed in the range of . The reference samples are set to according to some preliminary identification results. Figure 5.5 shows the scatter graphs of the estimated parameter vector (with 300 simulation runs), where (A) and (B) correspond, respectively, to the LMS and SIDG algorithms. In each graph, there are two clusters. Evidently, the clusters generated by SIDG are more compact than those generated by LMS, and the centers of the former are closer to the true values than those of the latter (the true values are and ). The involved error PDFs are illustrated in Figure 5.6. As one can see, the error distribution produced by SIDG is closer to the desired error distribution.

Example 5.5

Identifying the following unknown system [233]:

(5.118)

The adaptive model is chosen to be the reduced order IIR filter

(5.119)

The main goal is to determine the values of the coefficients (or weights) , such that the EDC is minimized. Assume that the error is Gaussian distributed, . Then, the empirical EDC can be approximately calculated as [233]:

(5.120)

where is the kernel width. Figure 5.7 shows the contours of the EDC performance surface in different (the input signal is assumed to be a white Gaussian noise with zero mean and unit variance). As one can see, the local minima of the performance surface have disappeared with large kernel width. Thus, by carefully controlling the kernel width, the algorithm can converge to the global minimum. The convergence trajectory of the adaptation process with the weight approaching to the global minimum is shown in Figure 5.8.