30 3. STATE ESTIMATION OF CYBER-PHYSICAL VEHICLE SYSTEMS
is recurrence relation is initialized at the final layer as
s
M
i
D
@
O
F
@n
M
i
D
@..t a/
T
.t a//
@n
M
i
D
@
P
S
M
j D1
.t
j
a
j
/
2
@n
M
i
D 2
.
t
i
a
i
/
@a
i
@n
M
i
D 2
.
t
i
a
i
/
P
f
m
n
m
i
:
(3.24)
us, the recurrence relation of the sensitivity matrix can be expressed as
s
M
D 2
P
F
M
.n
M
/.t a/: (3.25)
e overall BP learning algorithm is now finalized and can be summarized as the follow-
ing steps: (1) propagate the input forward through the network; (2) propagate the sensitivities
backward through the network from the last layer to the first layer; and, finally (3) update the
weights and biases using the approximate steepest descent rule.
3.3 LEVENBERG–MARQUARDT BACKPROPAGATION
While backpropagation is a steepest descent algorithm, the Levenberg–Marquardt algorithm
is derived from Newton’s method that was designed for minimizing functions that are sums of
squares of nonlinear functions [68, 69].
Newton’s method for optimizing a performance index F .x/ is
x
kC1
D x
k
A
1
k
g
k
(3.26)
A
k
r
2
F .x/j
XDX
k
(3.27)
g
k
rF .x/j
XDX
k
; (3.28)
where r
2
F .x/ is the Hessian matrix and rF .x/ is the gradient.
Assume that F.x/ is a sum of squares function:
F .x/ D
N
X
iD1
v
2
i
.x/ D v
T
.x/v.x/ (3.29)
then the gradient and Hessian matrix are
rF.x/ D 2J
T
.x/v.x/ (3.30)
r
2
F .x/ D 2J
T
.x/J.x/ C 2S.x/; (3.31)
3.3. LEVENBERG–MARQUARDT BACKPROPAGATION 31
where J.x/ is the Jacobian matrix
J.x/ D
2
6
6
6
6
6
6
6
6
6
4
@v
1
.x/
@x
1
@v
1
.x/
@x
2
@v
1
.x/
@x
n
@v
2
.x/
@x
1
@v
2
.x/
@x
2
@v
2
.x/
@x
n
:
:
:
:
:
:
:
:
:
@v
N
.x/
@x
1
@v
N
.x/
@x
2
@v
N
.x/
@x
n
3
7
7
7
7
7
7
7
7
7
5
(3.32)
and
S.x/ D
N
X
iD1
v
i
.x/r
2
v
i
.x/: (3.33)
If S.x/ is assumed to be small then the Hessian matrix can be approximated as
r
2
F .x/ Š 2J
T
.x/J.x/: (3.34)
Substituting Equations (3.30) and (3.34) into Equation (3.26), we achieve the Gauss–
Newton method as:
x
k
D
J
T
.
x
k
/
J
.
x
k
/
1
J
T
.
x
k
/
v
.
x
k
/
: (3.35)
One problem with the Gauss–Newton method is that the matrix may not be invertible.
is can be overcome by using the following modification to the approximate Hessian matrix:
G D H C I: (3.36)
is leads to the Levenberg–Marquardt algorithm [70]:
x
k
D
J
T
.
x
k
/
J
.
x
k
/
C
k
I
1
J
T
.
x
k
/
v
.
x
k
/
: (3.37)
Using this gradient direction, and recompute the approximated performance index. If a
smaller value is yield, then the procedure is continued with the
k
divided by some factor # > 1.
If the value of the performance index is not reduced, then
k
is multiplied by # for the next
iteration step.
e key step in this algorithm is the computation of the Jacobian matrix. e elements of
the error vector and the parameter vector in the Jacobian matrix (3.32) can be expressed as
v
T
D
Œ
v
1
v
2
: : : v
N
D
e
1;1
e
2;1
: : : e
S
M
;1
e
1;2
: : : e
S
M
;Q
(3.38)
x
T
D
Œ
x
1
x
2
: : : x
N
D
h
w
1
1;1
w
1
1;2
: : : w
1
S
1
;R
b
1
1
: : : b
1
S
1
w
2
1;1
: : : b
M
S
M
i
; (3.39)
where the subscript N satisfies:
N D Q S
M
(3.40)
32 3. STATE ESTIMATION OF CYBER-PHYSICAL VEHICLE SYSTEMS
and the subscript n in the Jacobian matrix satisfies:
n D S
1
.R C 1/ C S
2
S
1
C 1
C C S
M
S
M 1
C 1
: (3.41)
Making these substitutions into Equation (3.32), then the Jacobian matrix for multilayer
network training can be expressed as
J.x/ D
2
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
4
@e
1;1
@w
1
1;1
@e
1;1
@w
1
1;2
@e
1;1
@w
1
S
1
;R
@e
1;1
@b
1
1
@e
2;1
@w
1
1;1
@e
2;1
@w
1
1;2
@e
2;1
@w
1
S
1
;R
@e
2;1
@b
1
1
:
:
:
:
:
:
:
:
:
:
:
:
@e
S
M
;1
@w
1
1;1
@e
S
M
;1
@w
1
1;2
@e
S
M
;1
@w
1
S
1
;R
@e
S
M
;1
@b
1
1
@e
1;2
@w
1
1;1
@e
1;2
@w
1
1;2
@e
1;2
@w
1
S
1
;R
@e
1;2
@b
1
1
:
:
:
:
:
:
:
:
:
:
:
:
3
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
5
: (3.42)
In standard backpropagation algorithm, the terms in the Jacobian matrix is calculated as
@
O
F .x/
@x
l
D
@e
T
q
e
q
@x
l
: (3.43)
For the elements of the Jacobian matrix, the terms can be calculated by
ŒJ
h;l
D
@v
h
@x
l
D
@e
k;q
@w
i;j
: (3.44)
us, in this modified Levenberg–Marquardt algorithm, we compute the derivatives of
the errors, instead of the derivatives of the squared errors as adopted in standard backpropaga-
tion.
Using the concept of sensitivities in the standard backpropagation process, here we define
a new Marquardt sensitivity as
Qs
m
i;h
@v
h
@n
m
i;q
D
@e
k;q
@n
m
i;q
; (3.45)
where h D .q 1/S
M
C k.
Using the Marquardt sensitivity with backpropagation recurrence relationship, the ele-
ments of the Jacobian can be further calculated by
ŒJ
h;l
D
@e
k;q
@w
m
i;j
D
@e
k;q
@n
m
i;q
@n
m
i;q
@w
m
i;j
D Qs
m
i;h
a
m1
j;q
(3.46)
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.