2

Basics of Differential Games

The theory of games was developed around 1930 by Von Neumann and Morgenstern [136] and differential games approximately around 1950, the same time that optimal control theory was being developed. The theory immediately found application in warfare and economics; for instance, certain types of battles, airplane dog-fighting, a torpedo pursuing a ship, a missile intercepting an aircraft, a gunner guarding a target against an invader [73], are typical models of differential games. Similarly, financial planning between competing sectors of an economy, or competing products in a manufacturing system, meeting demand with adequate supply in a market system, also fall into the realm of differential games.

Game theory involves multi-person decision-making. It comprises of a task, an objective, and players or decision makers. The task may range from shooting down a plane, to steering of a ship, and to controlling the amount of money in an economy. While the objective which measures the performance of the players may vary from how fast? to how cheaply? or how closely? or a combination of these, the task is accomplished by the player(s).

When the game involves more than one player, the players may play cooperatively or noncooperatively. It is noncooperative if each player involved pursues his or her own interest, which are partly conflicting with that of others; and it is dynamic if the order in which the decisions are made is important. Whereas when the game involves only one player, the problem becomes that of optimal control which can be solved by the Pontryagin’s “maximum principle” or Bellman’s “dynamic programming principle” [73]. Indeed, the theories of differential games, optimal control, the calculus of variation and dynamic programming are all approaches to solving variational problems, and have now become merged into the theory of “modern control.”

There are two types of differential games that we shall be concerned with in this chapter; namely, nonzero-sum and zero-sum noncooperative dynamic games. We shall not discuss static games in this book. Furthermore, we shall only be concerned with games in which the players have perfect knowledge of the current and past states of the system, and their individual decisions are based on this. This is also known as closed-loop perfect information structure or pattern. On the other hand, when the decision is only based on the current value of the state of the system, we shall add the adjective “memoryless.”

We begin with the fundamental dynamic programming principle which forms the basis for the solution to almost all unconstrained differential game problems.

2.1    Dynamic Programming Principle

The dynamic programming principle was developed by Bellman [64] as the discrete-time equivalent of Hamilton-Jacobi theory. In fact, the term dynamic programming has become synonymous with Hamilton-Jacobi theory. It also provides a sufficient condition for optimality of an optimal decision process. To derive this condition, we consider a discrete-time dynamic system defined on a manifold $\mathcal{X}$ ⊆ ℜⁿ which is open and contains the origin x = 0,

with local coordinates (x₁,…, x_n):

$x_{k+1}=f_k\left(x_k,u_k\right),x_{k_0}=x^0,k\in \left\{k_0,\dots \right\}\subset Z,$

(2.1)

where $u_k\in \mathcal{U}$ is the control input or decision variable which belongs to the set $\mathcal{U}$ of all admissible controls, $f_k:\mathcal{X}\times \mathcal{U}\to \mathcal{X}$ is a $C^0\left(\mathcal{X}\times \mathcal{U}\right)$ function of its arguments for each k ∈ Z. We shall also assume that f_k(., .) satisfies the following global existence and uniqueness conditions for the solutions of (2.1)

Assumption 2.1.1 For the function f_k(x, u) in (2.1), for each k ∈ Z, there exists a constant C_1k (depending on k) such that for any fixed u,

$\Vert f_k\left(x_1,u\right)-f_k\left(x_2,u\right)\Vert \le c_{1k}\Vert x_1-x_2\Vert \mathrm{\forall }{x}_1,x_2\in \mathcal{X,}\mathrm{\forall }u\in \mathcal{U},k\in Z.$

To formulate an optimal decision process (or an optimal control problem), we introduce the following cost functional:

$J\left(x^0,k_0;u_{\left[k_0,K\right]}\right)={\displaystyle \sum _{k=K_0}^K{L}_k\left(x_{k+1},x_{k,}{u}_k\right)\to \min .,}$

(2.2)

where $L_k:\mathcal{X}\times \mathcal{X}\times \mathcal{U}\to \Re ,k=k_0,\mathrm{\dots },K,$ are real C⁰ functions of their arguments, which is to be minimized over the time span {k₀,…, K} as a basis for making the decision. The minimal cost-to-go at any initial state x_k and initial time k ∈ {k₀,…, K} (also known as value-function) is then defined as

$V\left(x,k\right)=\underset{u_{[k,K]}\in \mathcal{U}}{\inf }\left\{{\displaystyle \sum _{j=k}^K{L}_j\left(x_{j+1},x_j,u_j\right)}\right\},$

(2.3)

and satisfies the boundary condition V (x, K + 1) = 0, where x = x_k and u_[k,K] ≜ {u_k,…, u_K}.

Remark 2.1.1 For brevity, we shall use the notation [k₁, k₂] henceforth to mean the subset of the integers Z ⊃ {k₁,…, k₂} and where there will be no confusion.

We then have the following theorem.

Theorem 2.1.1 Consider the nonlinear discrete-time system (2.1) and the optimal control problem of minimizing the cost functional (2.2) subject to the dynamics of the system. Then, there exists an optimal control, k ∈ $u_k^{\star }$ [k₀, K] to the problem if there exist C⁰($\mathcal{X}$ ×[k₀, K+1]) functions V : $\mathcal{X}$ × [k₀, K] which corresponds to the value-function (2.3) for each k, and satisfying the following recursive (dynamic programming) equation subject to the boundary condition:

$V\left(x,k\right)=\underset{u_k}{\inf }\left\{L_k\left(f_k\left(x,u_k\right),x,u_k\right)+V\left(f_k\left(x,u_k\right),k+1\right)\right\};$

(2.4)

V (x, K + 1) = 0.

Furthermore,

$J^{\star }\left(x^0,k_0;u_{\left[k_0,K\right]}^{\star }\right)=V\left(x^0,k_0\right)$

is the optimal cost of the decision process.

Proof: The proof of the above theorem is based on the principle of optimality [164] and can be found in [64, 164]. □

Equation (2.4) is the discrete dynamic programming principle and governs the solution of most of the discrete-time problems that we shall be dealing with in this book.

Next, we discuss the continuous-time equivalent of the dynamic programming equation which is a first-order nonlinear partial-differential equation (PDE) known as the Hamilton-Jacobi-Bellman equation (HJBE). In this regard, consider the continuous-time nonlinear dynamic system defined on an open subset $\mathcal{X}$ ⊆ ℜⁿ containing the origin x = 0, with coordinates (x₁,…, x_n):

$\dot{x}\left(t\right)=f\left(x\left(t\right),u\left(t\right),t\right),x\left(t_0\right)=x_0\text{,}$

(2.5)

where $f:\mathcal{X}\times \mathcal{U}\times \Re \to {\Re }^n$ is a C¹ measurable function, and $\mathcal{U}$ ⊂ ℜ^m is the admissible control set which is also measurable, and x(t₀) = x₀ is the initial state which is assumed known. Similarly, for the sake of completeness, we have the following assumption for the global existence of solutions of (2.5) [234].

Assumption 2.1.2 The function f(., ., .) in (2.5) is piece-wise continuous with respect to t and for each t ∈ [0, ∞) there exist constants C_1t, C_2t such that

$\Vert f\left(x_1,u,t\right)-f\left(x_2,u,t\right)\Vert \le C_{1t}\Vert x_1-x_2\Vert \mathrm{\forall }{x}_1,x_2\in \mathcal{X},\mathrm{\forall }u\in \mathcal{U}.$

For the purpose of optimally controlling the system, we associate to it the following cost functional:

$J\left(x_0,t_0;u_{\left[t_0,T\right]}\right)={\displaystyle \underset{t_0}{\overset{T}{\int }}L\left(x,u,t\right)dt\to \min .,}$

(2.6)

for some real C⁰ function L : $\mathcal{X}$ × $\mathcal{U}$ × ℜ → ℜ, which is to be minimized over a time-horizon [t₀, T ] ⊆ ℜ. Similarly, define the value-function (or minimum cost-to-go) from any initial state x and initial time t as

$V\left(x,t\right)=\underset{u_{\left[t,T\right]}}{\inf }\left\{{\displaystyle \underset{t}{\overset{T}{\int }}L\left(x\left(s\right),u\left(s\right),s\right)ds}\right\}$

(2.7)

and satisfying the boundary condition V (x, T ) = 0. Then we have the following theorem.

Theorem 2.1.2 Consider the nonlinear system (2.5) and the optimal control problem of minimizing the cost functional (2.6) subject to the dynamics of the system and initial condition. Suppose there exists a C¹($\mathcal{X}$ × [t₀, T ]) function V : $\mathcal{X}$ × [t₀, T ] → ℜ, which corresponds to the value-function (2.7), satisfying the Hamilton-Jacobi-Bellman equation (HJBE):

$-V_t\left(x,t\right)=\underset{u_{\left[t_0,T\right]}}{\min }\left\{V_x\left(x,t\right)f\left(x,u,t\right)+L\left(x,u,t\right)\right\},V\left(x,T\right)=0.$

(2.8)

Then, there exists an optimal solution u^⋆ to the problem. Moreover, the optimal cost of the policy is given by

$J_{\star }\left(x_0^{\star },t_0;u_{\left[t_0,T\right]}^{\star }\right)=V\left(x_0,t_0\right).$

Proof: The proof of the theorem can also be found in [47, 175, 164]. □

The above development has considered single player decision processes or optimal control problems. In the next section, we discuss dynamic games, which involve multiple players. Again, we shall begin with the discrete-time case.

2.2    Discrete-Time Nonzero-Sum Dynamic Games

Nonzero-sum dynamic games were first introduced by Isaacs [136, 137] around the years 1954-1956 within the frame-work of two-person zero-sum games. They were later popularized in the works of Starr and Ho [250, 251] and Friedman [107]. Stochastic games were then later developed [59].

We consider a discrete-time deterministic N-person dynamic game of duration K − k₀ described by the state equation (which we referred to as the task):

$x_{k+1}=f_k\left(x_k,u_k^1,\dots ,u_k^N\right),x\left(k_0\right)=x^0,k\in \left\{k_0,\dots ,K\right\},$

(2.9)

where x_k ∈ $\mathcal{X}$ ⊂ ℜ ⁿ is the state-vector of the system for k = k₀,…, K which belong to the state-space $\mathcal{X}$ with x⁰ the initial state which is known a priori; $u_k^i,i=1,\mathrm{\dots },N,N\in \mathbf{\text{N}}$ are the decision variables or control inputs of the players 1,…, N which belong to the measurable control sets $\mathcal{U}$ⁱ ⊂ ℜ^m_i, m_i ∈ N; and $f_k:\mathcal{X}\times {\mathcal{U}}^1\times \mathrm{\dots },{\mathcal{U}}^N\to \Re$ are real C¹ functions of their arguments. To the game is associated the N objectives or pay-offs or cost functionals Jⁱ : Γ¹ ×…× Γ^N → ℜ, where Γⁱ, i = 1. .., N are the permissible strategy spaces of the players,

$J^i\left(u^1,\dots ,u^N\right)={\displaystyle \sum _{k=k_0}^K{L}_k^i\left(x_{k+1,}{x}_k,u_k^1,\dots ,u_k^N\right),i=1,\dots ,N,}$

(2.10)

and where the $u_k^i\in {\text{Γ}}^i,i=1,\mathrm{\dots },N$ are functions of x_k, and $L_k^i:\mathcal{X}\times \mathcal{X}\times {\mathcal{U}}^1\times \mathrm{\dots },{\mathcal{U}}^N\to \Re ,i=1,\mathrm{\dots },N,k=k_0,\mathrm{\dots },K$ are real C¹ functions of their arguments. The players play noncooperatively, and the objective is to minimize each of the above pay-off functions. Such a game is called a nonzero-sum game or Nash-game. The optimal decisions $u_{k,\star }^i,i=1,\mathrm{\dots },N,$ which minimize each pay-off function at each stage of the game, are called Nash-equilibrium solutions.

Definition 2.2.1 An N-tuple of strategies $\{u_{*}^i\in {\text{Γ}}^i,i=1,\mathrm{\dots },N\}$ where Γⁱ, i = 1,…, N is the strategy space of each player, is said to constitute a noncooperative Nash-equilibrium solution for the above N-person game if

$J_{\star }^i:=J^i\left(u_{\star }^1,\dots ,u_{\star }^N\right)\le J^i\left(u_{\star }^1,\dots ,u^i,u_{\star }^{i+1},u_{\star }^N\right)i=1,\dots ,N,$

(2.11)

$where{u}_{\star }^i=\left\{u_{k,\star }^{i}k=k_0,\dots ,K\right\}and{u}^i=\left\{u_{k,\star }^{i}k=k_0,\dots ,K\right\}.$

In this section, we discuss necessary and sufficient conditions for the existence of Nash-equilibrium solutions of such a game under closed-loop (or feedback) no-memory perfect-state-information structure, i.e., when the decision is based on the current state information which is pefectly known. The following theorem based on the dynamic programming principle gives necessary and sufficient conditions for a set of strategies to be a Nash-equilibrium solution [59].

Theorem 2.2.1 For the N-person discrete-time nonzero-sum game (2.9)-(2.11), an N-tuple of strategies $\{u_{\star }^i\in {\text{Γ}}^i,i=1,\mathrm{\dots },N\},$ provides a feedback Nash-equilibrium solution, if and only if, there exist N × (K − k₀) functions Vⁱ : $\mathcal{X}$ × Z → ℜ such that the following recursive equations are satisfied:

$\begin{array}{l}{V}^i\left(x,k\right)=\underset{u_k^i\in {\mathcal{U}}^i}{\min }\left\{L_k^i\left(x_{k+1},x_k,u_{k,\star }^1,\dots ,u_k^i,\dots ,u_{k,\star }^N\right),+V^i\left(x_{k+1},k+1\right)\right\},\\ V^i\left(x,K+1\right)=0,i=1,\dots ,N,k\in \left\{k_0,\dots ,K\right\}\\ =\underset{u_k^i\in {\mathcal{U}}^i}{\min }\left\{L_k^i(f_k\left(x,u_k^1,\dots ,u_k^i,\dots ,u_{k,\star }^N\right),x_k,u_{k,\star }^1,\dots ,u_k^i,\dots ,u_{k,\star }^N)+V^i(f_k\left(x,u_k^1,\dots ,u_k^i,\dots ,u_{k,\star }^N\right),k+1\right\},\\ V^i\left(x,K+1\right)=0,i=1,\dots ,N,k\in \left\{k_0,\dots ,K\right\}.\end{array}$

(2.12)

Notice that, in the above equations, there are N × (K − k₀) functions, since at each stage of the decision process, a different function is required for each of the players. However, it is often the case that N functions that satisfy the above recursive equations can be obtained with a single function for each player. This is particularly true in the linear case.

More specifically, let us consider the case of a two-player nonzero-sum game, of which type most of the problems in this book will be, described by the state equation:

$x_{k+1}=f_x\left(x_k,w_k,u_k\right),x\left(k_0\right)=x^0,k\in \left\{k_0,\dots ,K\right\}$

(2.13)

where w_k ∈ $\mathcal{W}$ ⊂ ℜ^m_w and u_k ∈ $\mathcal{U}$ ⊂ℜ^m_u represent the decisions of the two players respectively. The pay-offs J¹, J² are given by

$J^1\left(w,u\right)={\displaystyle \sum _{k=K_0}^K{L}_k^1\left(x_{k+1},x_k.w_k,u_k\right),}$

(2.14)

$J^2\left(w,u\right)={\displaystyle \sum _{k=K_0}^K{L}_k^2\left(x_{k+1},x_k.w_k,u_k\right),}$

(2.15)

where w = w_[k0,K], u = u_[k0,K]. Then, a pair of strategies w_⋆ := {w_k,⋆, k = k₀,…, K}, u_⋆ := {u_k,⋆, k = k₀,…, K}, will constitute a Nash-equilibrium solution for the game if

$J^1\left(w_{\star },u_{\star }\right)\le J^1\left(w_{\star },u\right)\mathrm{\forall }u\in \mathcal{U},$

(2.16)

$J^2\left(w_{\star },u_{\star }\right)\le J^2\left(w,u_{\star }\right)\mathrm{\forall }w\in \mathcal{W}\text{.}$

(2.17)

Furthermore, the conditions of Theorem 2.2.1 reduce to:

$V^1(x,k)=\underset{u_k\in \mathcal{U}}{min}\{L_k^1(f_k(x,w_{k,\star },u_k),x,w_{k,\star },u_k)+V^1(f_k(x,w_{k,\star },u_k),k+1\};V^1(x,K+1)=0,k=1,\dots ,K$

(2.18)

$V^2\left(x,k\right)=\underset{w_k\in \mathcal{W}}{\text{min}}\left\{L_k^2\left(f_k\left(x,w_k,u_{k,\star }\right),x,w_k,u_{k,\star }\right)+V^2(f_k\left(x,w_k,u_{k,\star }\right),k+1\right\};V^2\left(x,K+1\right)=0,k=1,\dots ,K\text{.}$

(2.19)

Remark 2.2.1 Equations (2.18), (2.19) represent a pair of coupled HJ-equations. They are difficult to solve, and not much work has been done to study their behavior. However, these equations will play an important role in the derivation of the solution to the discrete-time mixed ${\mathcal{H}}_2$/${\mathcal{H}}_{\infty }$ control problem in later chapters of the book.

Another special class of the nonzero-sum games is the two-person zero-sum game, in which the objective functionals (2.14), (2.15) are such that

$J^1\left(w,u\right)=-J^2\left(w,u\right)=J\left(w,u\right):={\displaystyle \sum _{k=k_0}^K\left(x_{k+1},x_k.w_k,u_k\right).}$

(2.20)

If this is the case, then J¹(w_⋆, u_⋆) + J²(w_⋆, u_⋆) = 0 and while u is the minimizer, w is the maximizer. Thus, we might refer to u as the minimizing player and w as the maximizing player. Furthermore, the “Nash-equilibrium” conditions (2.16), (2.17) reduce to the saddle-point conditions:

$J\left(w,u_{\star }\right)\le J\left(w_{\star },u_{\star }\right)\le J\left(w_{\star },u\right),\mathrm{\forall }w\in \mathcal{W},u\in \mathcal{U},$

(2.21)

where the pair (w_*, u_*) is the optimal strategy and is called a “saddle-point.” Consequently, the set of necessary and sufficient conditions (2.18), (2.19) reduce to the following condition given in this theorem [57, 59].

Theorem 2.2.2 For the two-person discrete-time zero-sum game defined by (2.13), (2.20), a pair of strategies (w_⋆, u_⋆) provides a feedback saddle-point solution if, and only if, there exists a set of K − k₀ functions V : $\mathcal{X}$ × Z → ℜ, such that the following recursive equations are satisfied for each k ∈ [k₀, K]:

$\begin{array}{l}V(x,k)=\underset{u_k\in \mathcal{U}}{\min }\underset{w_k\in \mathcal{W}}{\max }\{L_k(f_k(x,w_k,u_k),x,w_k,u_k)+V(f_k(x,w_k,u_k),k+1)\},\hfill \\ =\underset{w_k\in \mathcal{W}}{\max }\underset{u_k\in \mathcal{U}}{\min }\{L_k(f_k(x,w_k,u_k),x,w_k,u_k)+V(f_k(x,w_k,u_k),k+1)\},\hfill \\ =L_k(f_k(x,w_{k,\star },u_{k,\star }),x,w_{k,\star },u_{k,\star })+V(f_k(x,w_{k,\star },u_{k,\star }),k+1),\hfill \\ V(x,K+1)=0,\hfill \end{array}$

(2.22)

where x = x_k. Equation (2.22) is known as Isaacs equation, being the discrete-time version of the one developed by Isaacs [136, 59]. Furthermore, the interchangeability of the “max” and “min” operations above is also known as “Isaacs condition.” The above equation will play a significant role in the derivation of the solution to the ${\mathcal{H}}_{\infty }$-control problem for discrete-time nonlinear systems in later chapters of the book.

2.2.1    Linear-Quadratic Discrete-Time Dynamic Games

We now specialize the above results of the N-person nonzero-sum dynamic games to the linear-quadratic (LQ) case in which the optimal decisions can be expressed explicitly. For this purpose, the dynamics of the system is described by a linear state equation:

$x_{k+1}=f_k\left(x_k,u_k^1,\dots ,u_k^N\right)=A_k{x}_k+{\displaystyle \sum _{i=1}^N{B}_k^i{u}_k^i,k=\left[k_0,K\right]}$

(2.23)

where $A_k\in {\Re }^{n\times n},B_k^i\in {\Re }^{n\times p_i},$ and all the variables have their previous meanings. The pay-offs Jⁱ, i = 1,…, N are described by

$J^i\left(u_{\left[k_0,K\right]}^1,\dots ,u_{\left[k_0,K\right]}^N\right)=N{\displaystyle \sum _{k=1}^{}{L}_k^i\left(x_{k+1},u_k^1,\dots ,u_k^N\right)}$

(2.24)

$L_k^i\left(x_{k+1},u_k^1,\dots ,u_k^N\right)=\frac{1}{2}\left[x_{k+1}^T{Q}_{k+1}^i{x}_{k+1}+{\displaystyle \sum _{j=1}^N{\left(u_k^j\right)}^T{R}_k^{ij}{u}_k^j}\right]$

(2.25)

where $Q_{k+1}^i,R_k^{ij}$ are matrices of appropriate dimensions with $Q_{k+1}^i\ge 0$ symmetric and $R_k^{ii}>0,$ for all k = k₀,…, K, i, j = 1,…, N.

The following corollary gives necessary and sufficient conditions for the existence of Nash-equilibrium for the LQ-game [59].

Corollary 2.2.1 For the N-player LQ game described by the equations (2.23)-(2.25), suppose $Q_{k+1}^i\ge 0$ i = 1,…, N and $R_k^{ii}>0,$ i, j = 1,…, N, j ≠ i, k = k₀,…, K. Then, there exists a unique feedback Nash-equilibrium solution to the game if, and only if, there exist unique solutions $P_k^i,i=1,\mathrm{\dots },N\text{},$ k = k₀,…, K to the recursive equations:

$\left[R_k^{ii}+{\left(B_k^i\right)}^T{Z}_{k+1}^j{B}_k^i\right]P_k^i+{\left(B_k^i\right)}^T{Z}_{k+1}^i{\displaystyle \sum _{j=1,j\ne i}^N{B}_k^j{P}_k^j={\left(B_k^i\right)}^T{Z}_{k+1}^i{A}_k,i=1,\dots ,N,}$

(2.26)

where

$Z_k^i=F_k^T{Z}_{k+1}^i{F}_k+{\displaystyle \sum _{j=1}^N{\left(p_k^j\right)}^T{R}_k^{ij}{P}_k^j+Q_k^i,Z_{K+1}^i=Q_{K+1}^i},i=1,\dots ,N$

(2.27)

$F_k=A_k-{\displaystyle \sum _{i=1}^N{B}_k^i{P}_k,k=k_0,\dots ,K.}$

(2.28)

Furthermore, the equilibrium strategies are given by

$u_k^i=-P_k^i{x}_k,i=1,\dots ,N.$

(2.29)

Remark 2.2.2 The positive-semidefinite conditions for $Q_{k+1}^i$, and $R_k^{ij}$, j ≠ i guarantee the convexity of the functionals Jⁱ and therefore also guarantee the existence of a minimizing solution in (2.12) for all k ∈ [k₀, K]. Furthermore, there has not been much work in the literature regarding the existence of solutions to the coupled discrete-Riccati equations (2.26)-(2.28). The only relevant work we could find is given in [84].

Again, the results of the above corollary can easily be specialized to the case of two players. But more importantly, we shall specialize the results of Theorem 2.2.2 to the case of the two-player LQ zero-sum discrete-time dynamic game described by the state equation:

$x_{k+1}=A_k{x}_k+B_k^1{w}_k+B_k^2{u}_k,x\left(k_0\right)=x^0,k=\left[k_0,K\right]$

(2.30)

and the objective functionals

$-J^1\left(u,w\right)=J^2\left(u,w\right)=J\left(u,w\right)={\displaystyle \sum {}_{k=k_0}^K{L}_k\left(x_{k+1},x_k,u,w\right),}$

(2.31)

$L_k\left(x_{k+1},x_k,u,w\right)=\frac{1}{2}\left[x_{k+1}^T{Q}_{k+1}{x}_{k+1}+u_k^T{u}_k-w_k^T{w}_k\right].$

(2.32)

We then have the following corollary [57, 59].

Corollary 2.2.2 For the two-player LQ zero-sum dynamic game described by (2.30)-(2.31), suppose Q_k+1 ≥ 0, k = k₀,…, K. Then there exists a unique feedback saddle-point solution if, and only if,

$I+{\left(B_k^2\right)}^T{M}_{k+1}{B}_k^2>0,\mathrm{\forall }k=k_0,\dots ,K,$

(2.33)

$I-{\left(B_k^1\right)}^T{M}_{k+1}{B}_k^1>0,\mathrm{\forall }k=k_0,\dots ,K,$

(2.34)

where

$S_k=[I+(B_k^1{B}_k^{1^T}-B_k^2{B}_k^{2^T})M_{k+1}],$

(2.35)

$M_k=Q_k+A_k^T{M}_{k+1}{S}_k^{-1}{A}_k,M_{K+1}=Q_{K+1}.$

(2.36)

Moreover, the unique equilibrium strategies are given by

$u_{k,\star }=-{\left(B_k^2\right)}^T{M}_{k+1}{S}_k^{-1}{A}_k{x}_k,$

(2.37)

$w_{k,\star }={\left(B_k^1\right)}^T{M}_{k+1}{S}_k^{-1}{A}_k{x}_k,$

(2.38)

and the corresponding unique state trajectory is given by the recursive equation:

$x_{k+1}^{\star }=S_k^{-1}{A}_k,x_k^{\star }\text{.}$

(2.39)

In the next section we consider the continuous-time counterparts of the discrete-time dynamic games.

2.3    Continuous-Time Nonzero-Sum Dynamic Games

In this section, we discuss the continuous-time counterparts of the results that we presented in the previous section for discrete-time dynamic games. Indeed the name “differential game” stems from the continuous-time models of dynamic games.

We consider at the outset N-player nonzero-sum differential game defined by the state equation:

$\dot{x}\left(t\right)=f\left(x,u^1,\dots ,u^N,t\right),x\left(t_0\right)=x_0$

(2.40)

where x(t) ∈ $\mathcal{X}$ ⊆ ℜⁿ is the state of the system for any t ∈ℜ which belong to the state-space manifold $\mathcal{X}$ and x(t₀) is the initial state which is assumed to be known a priori by all the players; uⁱ ∈ $\mathcal{U}$ⁱ, i = 1,…, N, N ∈ N are the decision variables or control inputs of the players 1,…, N which belong to the control sets $\mathcal{U}$ⁱ ⊂ ℜ^m_i, m_i ∈ N, which are also measurable; and $f_k:\mathcal{X}\times {\mathcal{U}}^1\times \mathrm{\dots },{\mathcal{U}}^N\times \Re \to \Re$ is a real C¹ function of its arguments.

To each player i = 1,…, N is associated an objective functional or pay-off Jⁱ : Γ¹ ×. .. × Γ^N → ℜ, where Γⁱ, i = 1,…, N is the strategy space of the player, which he tries to minimize, and is defined by

$J^i\left(u^1,\dots ,u^N\right)={\varphi }^i\left(x\left(t_f\right),t_f\right)+{\int }_{t_0}^{t_f}{L}^i\left(x,u^1,\dots ,u^N,t\right)dt,i=1,\dots ,N,$

(2.41)

where $L^i:\mathcal{X}\times {\mathcal{U}}^1\times \mathrm{\dots },{\mathcal{U}}^N\times {\Re }_{+}\to \Re \text{and}{\varphi }^i:\mathcal{X}\times {\Re }_{+}\to \Re ,i=1,\mathrm{\dots },N,$ are real C¹ functions of their arguments respectively. The final or terminal time t_f may be variable or fixed; here, we shall assume it is fixed for simplicity. The functions ϕⁱ(., .), i = 1,…, N are also known as the terminal cost functions.

The players play noncooperatively, but each player knows the current value of the state vector as well as all system parameters and cost functions. However, he does not know the strategies of the rival players. Our aim is to derive sufficient conditions for the existence of Nash-equilibrium solutions to the above game under closed-loop memoryless perfect information structure. For this purpose, we redefine Nash-equilibrium for continuous-time dynamic games as follows.

Definition 2.3.1 An N-tuple of strategies $\left\{u_{\star }^i\in {\text{Γ}}^i,i=1,\mathrm{\dots },N\right\}$, where Γⁱ, i = 1,…, N are the set of strategies, constitute a Nash-equilibrium for the game (2.40)-(2.41) if

$J_{\star }^i:=J^i\left(u_{\star }^1,\dots ,u_{\star }^N\right)\le J^i\left(u_{\star }^1,\dots ,u_{\star }^{i-1},u^i,u_{\star }^{i+1},\dots ,u_{\star }^N\right){\mathrm{\forall }}_i=1,\dots ,N,$

(2.42)

where $u_{*}^i$ = {$u_{*}^i$ (t), t ∈ [t₀, t_f ]} and uⁱ = {uⁱ(t), t ∈ [t₀, t_f ]}.

To derive the optimality conditions, we consider an N-tuple of piecewise continuous

strategies u := {u¹,…, u^N } and the value-function for the i-th player

$V^i\left(x,t\right)=\underset{u\in \mathcal{U}}{\inf }\left\{{\varphi }^i\left(x\left(t_f\right),t_f\right)+{\int }_t^{t_f}{L}^i\left(x,u,\tau \right)dt\right\}={\varphi }^i\left(x\left(t_f\right),t_f\right)+{\int }_t^{t_f^{}}{L}^i\left(x,u_{\star },\tau \right)d\tau .$

(2.43)

Then, by applying Definition 2.3.1 and the dynamic-programming principle, it can be shown that the value-functions V ⁱ, i = 1,…, N are solutions of the following Hamilton-Jacobi PDEs:

$\frac{\partial V^i}{\partial t}=-\underset{u^i\in {\mathcal{U}}^i}{\inf }{H}^i\left(x,t,u_{\star }^1,\dots ,u_{\star }^{i-1},u^i,u_{\star }^{i+1},\dots ,u_{\star }^N,\frac{\partial V^i}{\partial x}\right)$

$V^i(x(t_f),t_f)={\varphi }^i(x(t_f),t_f),$

(2.44)

$H^i\left(x,t,u,{\text{λ}}_i^T\right)=L^i\left(x,u,t\right)+{\lambda }_i^{T}f\left(x,u,t\right),i=1,\dots ,N.$

(2.45)

The strategies $u_{\star }=\left(u_{\star }^1,\mathrm{\dots },u_{\star }^N\right)$ which minimize the right-hand-side of the above equations (2.44) are the equilibrium strategies. The equations (2.44) are integrated backwards in time from the terminal manifold x(t_f ), and at each (x, t) one must solve a static game for the Hamiltonians Hⁱ, i = 1,…, N to find the Nash-equilibrium. This is not always possible except for a class of differential games called normal [250]. For this, it is possible to find a unique Nash-equilibrium point u_⋆ for the vector H = [H¹,…, H^N ] for all x, λ, t such that when the equations

$\frac{\partial V^i}{\partial t}\left(x,t\right)=-H^i\left(x,t,u_{\star }\left(x,t,\frac{\partial V^1}{\partial x},\dots ,\frac{\partial V^N}{\partial x}\right),\frac{\partial V^i}{\partial x}\right)$

(2.46)

$\dot{x}=f\left(x,u^1,\dots ,u^N,t\right)$

(2.47)

are integrated backward from all the points on the terminal surface, feasible trajectories are obtained. Thus, for a normal game, the following theorem provides sufficient conditions for the existence of Nash-equilibrium solution for the N-player game under closed-loop memoryless perfect-state information pattern [57, 250].

Theorem 2.3.1 Consider the N-player nonzero-sum continuous-time dynamic game (2.40)-(2.41) of fixed duration [t₀, t_f ], and under closed-loop memoryless perfect state information pattern. An N-tuple $u_{\star }=\left(u_{\star }^1,\mathrm{\dots },u_{\star }^N\right)$ of strategies provides a Nash-equilibrium solution, if there exist N C¹-functions V ⁱ : $\mathcal{X}$ × ℜ₊ → ℜ, i = 1,…, N satisfying the HJEs (2.44)-(2.45).

Remark 2.3.1 The above result can easily be specialized to the two-person nonzero-sum continuous-time dynamic game. This case will play a significant role in the derivation of the solution to the mixed ${\mathcal{H}}_2$/${\mathcal{H}}_{\infty }$ control problem in a later chapter.

Next, we specialize the above result to the case of the two-person zero-sum continuous-time dynamic game. For this case, we have the dynamic equation and the objective functional described by

$\dot{x}\left(t\right)=f\left(x,u,w,t\right),x\left(t_0\right)=x_0,$

(2.48)

$J^1\left(u,w\right)=-J^2\left(u,w\right):=J\left(u,w\right)=\varphi \left(x\left(t_f\right),t_f\right)+{\int }_{t_0^{}}^{t_f^{}}L\left(x,u,w,t\right)dt,$

(2.49)

where u ∈ $\mathcal{U}$, w ∈ $\mathcal{W}$ are the strategies of the two players, which belong to the measurable sets $\mathcal{U}$ ⊂ ℜ^mu, $\mathcal{W}$ ⊂ ℜ^mw respectively, x₀ is the initial state which is known a priori to both players. Then we have the following theorem which is the continuous-time counterpart of Theorem 2.2.2.

Theorem 2.3.2 Consider the two-person zero-sum continuous-time differential game (2.48)-(2.49) of fixed duration [t₀, t_f], and under closed-loop memoryless information structure. A pair of strategies (u_⋆, w_⋆) provides a Nash-feedback saddle-point solution to the game, if there exists a C¹-function V : X × ℜ → ℜ of both of its arguments, satisfying the PDE:

$-\frac{\partial V\left(x,t\right)}{\partial t}=\underset{u\in \mathcal{U}}{\inf }\underset{w\in \mathcal{W}}{\sup }\left\{\frac{\partial V\left(x,t\right)}{\partial t}f\left(x,u,w,t\right)+L\left(x,u,w,t\right)\right\}$

$=\underset{w\in \mathcal{W}}{\sup }\underset{u\in \mathcal{U}}{\inf }\left\{\frac{\partial V\left(x,t\right)}{\partial t}f\left(x,u,w,t\right)+L\left(x,u,w,t\right)\right\}$

$=\frac{\partial V\left(x,t\right)}{\partial t}f\left(x,u_{\star },w_{\star },t\right)+L\left(x,u_{\star },w_{\star },t\right),V\left(x\left(t_f\right),t_f\right)=\varphi \left(x\left(t_f\right),t_f\right),$

(2.50)

known as Isaacs equation [136] or Hamilton-Jacobi-Isaacs equation (HJIE).

Remark 2.3.2 The HJIE (2.50) will play a significant role in the derivation of the solution to the ${\mathcal{H}}_{\infty }$-control problem for continuous-time nonlinear systems in later chapters.

Next, we similarly specialize the above results to linear systems with quadratic objective functions, also known as the “linear-quadratic (LQ)” continuous-time dynamic game.

2.3.1    Linear-Quadratic Continuous-Time Dynamic Games

In this section, we specialize the results of the previous section to the linear-quadratic (LQ) continuous-time dynamic games. For the linear case, the dynamic equations (2.40), (2.41) reduce to the following equations:

$\dot{x}\left(t\right)=f\left(x,u^1,\dots ,u^N,t\right)=A\left(t\right)x\left(t\right)+{\displaystyle \sum _{i=1}^N{B}^i\left(t\right)u^i\left(t\right),x\left(t_0\right)=x_0}$

(2.51)

$J^i\left(u^1,\dots ,u^N\right)=\frac{1}{2}{x}^T\left(t_f\right)Q_f^{i}x\left(t_f\right)+\frac{1}{2}{\int }_{t_0^{}}^{t_f}\left[x^T\left(t_f\right)Q_f^{i}x\left(t_f\right)+{\displaystyle \sum }_{j=1}^N{\left(u^j\right)}^T\left(t\right)R^{ij}\left(t\right)u^j\left(t\right)\right]dt,$

(2.52)

where A(t) ∈ ℜ^n×n, Bⁱ(t) ∈^n×pi and Qⁱ, Qⁱ_f, R^ij(t) are matrices of appropriate dimensions for each t ∈ [t₀, t_f ], i, j = 1,…, N. Moreover, Qⁱ(t), Qⁱ_f are symmetric positivesemidefinite and Rⁱⁱ is positive-definite for i = 1,…, N. It is then possible to derive a closed-form expression for the optimal strategies as stated in the following corollary to Theorem 2.3.1.

Corollary 2.3.1 Consider the N-person LQ-continuous-time dynamic game (2.51), (2.52). Suppose Qⁱ(t) ≥ 0, Qⁱ_f ≥ 0 and symmetric, R^ij(t) > 0 for all t ∈ [t₀, t_f ], i, j = 1,…, N, i = j. Then there exists a linear feedback Nash-equilibrium solution to the differential game under closed-loop memoryless perfect state information structure, if there exists a set of symmetric solutions P ⁱ(t) ≥ 0 to the N-coupled time-varying matrix Riccati ordinary differential-equations (ODEs):

${\dot{P}}_i\left(t\right)+P^i\left(t\right)\tilde{A}\left(t\right)+{\tilde{A}}^T\left(t\right)P^i\left(t\right)+{\displaystyle \sum _{j=1}^N{P}^j\left(t\right)B^j\left(t\right){\left(R^{jj}\left(t\right)\right)}^{-1}{R}^{ij}\left(t\right){\left(R^{jj}\left(t\right)\right)}^{-1}{\left(B^j\right)}^T{P}^j\left(t\right)+Q^i\left(t\right)=0,P^i\left(t_f\right)=}{Q}_f^i,i=1,\dots ,N$

(2.53)

$\tilde{A}\left(t\right)=A\left(t\right)-{\displaystyle \sum _{i=1}^N{B}^i\left(t\right){\left(R^{ii}\left(t\right)\right)}^T{\left(B^i\right)}^T{P}^i\left(t\right).}$

(2.54)

Furthermore, the optimal strategies and costs for the players are given by

$u_{\star }^t\left(t\right)=-{\left(R^{ii}\left(t\right)\right)}^{-1}{\left(B^i\left(t\right)\right)}^T{P}^i\left(t\right)x\left(t\right),i=1,\dots ,N,$

(2.55)

$J_{\star }^i=\frac{1}{2}{x}_0^T{P}^i\left(0\right)x_0,i=1,\dots ,N,$

(2.56)

The equations (2.53) represent a system of coupled ODEs which could be integrated backwards using numerical schemes such as the Runge-Kutta method. However, not much is known about the behavior of such a system, although some work has been done for the two-player case [2, 108, 221]. Since these results are not widely known, we shall review them briefly here.

Consider the system equation with two players given by

$\dot{x}\left(t\right)=A\left(t\right)x\left(t\right)+B^1\left(t\right)w\left(t\right)+B^2\left(t\right)u\left(t\right),x\left(t_0\right)=x_0,$

(2.57)

and the cost functions:

$J^1\left(u,w\right)=x{\left(t_f\right)}^T{Q}_f^{1}x\left(t_f\right)+{\int }_0^{t_f^{}}\left(x^T{Q}^{1}x+w^T{R}^{11}w+u^T{R}^{12}u\right)dt,$

(2.58)

$J^2\left(u,w\right)=x{\left(t_f\right)}^T{Q}_f^{2}x\left(t_f\right)+{\int }_0^{t_f}\left(x^T{Q}^{2}x+u^T{R}^{22}u+w^T{R}^{21}w\right)dt,$

(2.59)

where Qⁱ_f ≥ 0, Qⁱ ≥ 0, R^ij ≥ 0, i = j, Rⁱⁱ > 0, i, j = 1, 2. Further, assume that the matrices A(t) = A, B¹(t) = B¹, and B²(t) = B² are constant and of appropriate dimensions. Then, the system of coupled ODEs corresponding to the two-player nonzero-sum game is given by

$\begin{array}{l}{\dot{P}}^1=-A^T{P}^1-P^{1}A+P^1{S}^{11}{P}^1+P^1{S}^{22}{P}^2+P^2{S}^{22}{P}^1-P^2{S}^{12}{P}^2-Q^1,\\ P^1\left(t_f\right)=Q_f^1,\end{array}$

(2.60)

$\begin{array}{l}{\dot{P}}^2=-A^T{P}^2-P^{2}A+P^2{S}^{22}{P}^2+P^2{S}^{11}{P}^1+P^1{S}^{11}{P}^2-P^1{S}^{21}{P}^1-Q^2,\hfill \\ P^2(t_f)=Q_f^2,\hfill \end{array}$

(2.61)

where

$S^{ij}=B^j{\left(R^{jj}\right)}^{-1}{R}^{ij}{\left(R^{jj}\right)}^{-1}{\left(B^j\right)}^T,i,j=1,2.$

In [221] global existence results for the solution of the above system (2.60)-(2.61) is established for the special case B¹ = B² and R¹¹ = R²² = −R¹² = −R²¹ = I_n, the identity matrix. It is shown that P ¹ − P ² satisfies a linear Lyapunov-type ODE, while P ¹ + P ² satisfies a standard Riccati equation. Therefore, P ¹ + P ² and hence P ¹, P ² cannot blow up in any finite time, and as t_f → ∞, P ¹(0) + P ²(0) goes to a stabilizing, positive-semidefinite solution of the corresponding algebraic Riccati equation (ARE). In the following result, we derive upper and lower bounds for the solutions P ¹ and P ² of (2.60), (2.61).

Lemma 2.3.1 Suppose P ¹, P ² are solutions of (2.60)-(2.61) on the interval [t₀, t_f ]. Then, $P^1(t)\ge 0,P^2(t)\ge 0\text{for}\text{all}t\in [t_0,t_f].$

Proof: Let x(t₀) = x₁ ≠ 0 and let x be a solution of the initial-value problem

$\dot{x}\left(t\right)=\left[A-S^{11}{P}^1\left(t\right)-S^{22}{P}^2\left(t\right)\right]x\left(t\right),x\left(t_0\right)=x_1.$

(2.62)

Then for i = 1, 2

$\frac{d}{dt}\left[x^T{P}^i\left(t\right)x\right]={\dot{x}}^T{\dot{P}}^i\left(t\right)x+x^T{P}^i\left(t\right)\dot{x}$

$=x^T\left\{A^T{P}^i\left(t\right)-P^1\left(t\right)S^{11}{P}^i\left(t\right)-P^2\left(t\right)S^{22}{P}^i\left(t\right)+P^i\left(t\right)A-P^i\left(t\right)S^{11}{P}^1\left(t\right)-P^i\left(t\right)S^{22}{P}^2\left(t\right)-A^T{P}^i\left(t\right)-P^i\left(t\right)A-Q^i+P^i\left(t\right)S^{ii}{P}^i\left(t\right)-{\displaystyle \sum _{j\ne i}{P}^j\left(t\right)S^{ij}{P}^j\left(t\right)}+{\displaystyle \sum _{j\ne i}\left(P^i\left(t\right)S^{jj}{P}^j\left(t\right)+P^j\left(t\right)S^{jj}{P}^i\left(t\right)\right)}\right\}x$

$=-x^T\left\{Q^i+P^1\left(t\right)S^{i1}{P}^1\left(t\right)+P^2\left(t\right)S^{i2}{P}^2\left(t\right)\right\}x$

where the previous equation follows from (2.60), (2.61) and (2.62). Now integrating from t₀ to t_f yields

$x_1^T{P}^i\left(t_0\right)x_1={\displaystyle {\int }_{t_0}^{t_f}{x}^T\left(\tau \right){\tilde{Q}}^i\left(\tau \right)x\left(\tau \right)d\tau }+x^T\left(t_f\right)Q_f^{i}x\left(t_f\right)\ge 0$

(2.63)

where ${\tilde{Q}}^i$ := Qⁱ + P ¹Sⁱ¹P ¹ + P ²Sⁱ²P ². Since x₁ is arbitrary and Qⁱ, Sⁱ¹, Sⁱ², P ⁱ(t_f ) ≥ 0, the result follows. □

The next theorem gives sufficient conditions for no finite-escape times for the solutions P ¹, P ².

Theorem 2.3.3 Let Q ∈ ℜ^n×n be symmetric and suppose for all t ≤ t_f

$\begin{array}{l}F\left(P^1\left(t\right),P^2\left(t\right),Q\right):=Q+\left(P^1\left(t\right)+P^2\left(t\right)\right)\left(S^{11}+S^{22}\right)+\left(P^1\left(t\right)+P^2\left(t\right)\right)-\\ P^1\left(t\right)S^{21}{P}^1\left(t\right)-P^2\left(t\right)S^{12}{P}^2\left(t\right)-\left(P^1\left(t\right)S^{22}{P}^1\left(t\right)+P^2\left(t\right)S^{11}{P}^2\left(t\right)\right)\ge 0,\end{array}$

(2.64)

and when P ¹(t), P ²(t) exist, then the solutions P ¹(t), P ²(t) of (2.60), (2.61) exist for all t < t_f with

$0\le P^1\left(t\right)+P^2\left(t\right)\le R_Q\left(t\right),$

(2.65)

for some unique positive-semidefinite matrix function R_Q(t) solution to the terminal value problem:

${\dot{R}}_Q\left(t\right)=-R_Q\left(t\right)A-A^T{R}_Q\left(t\right)-\left(Q^1+Q^2+Q\right),R_Q\left(t_f\right)=Q_f^1+Q_f^2$

(2.66)

which exists for all t < t_f.

Proof: The first inequality in (2.65) follows from Lemma 2.3.1. Further

$\left({\dot{P}}^1+{\dot{P}}^2\right)=-A^T\left(P^1+P^2\right)-\left(P^1+P^{2}A-\left(Q^1+Q^2+Q\right)+F\left(P^1\left(t\right),P^2\left(t\right),Q\right)\right),$

and by the monotonicity of solutions to Riccati equations [292], it follows that the second inequality holds while F (P ¹(t), P ²(t), Q) ≥ 0. □

Remark 2.3.3 For a more extensive discussion of the existence of the solutions P ¹, P ² to the coupled Riccati-ODEs, the reader is referred to reference [108]. However, we shall take up the subject for the infinite-horizon case where t_f → ∞ in a later chapter.

We now give an example of a pursuit-evasion problem which can be solved as a two-player nonzero-sum game [251].

Example 2.3.1 We consider a pursuit-evasion problem with the following dynamics

$\dot{r}=v,\dot{v}=a_p-a_e$

where r is the relative position vector of the two adversaries, a_p, a_e are the accelerations of the pursuer and evader respectively, which also serve as the controls to the players.

The cost functions are taken to be

$J_p=\frac{1}{2}{q}_p^2{r}_f^T{r}_f+\frac{1}{2}{\displaystyle {\int }_0^T\left(\frac{1}{c_e}{a}_p^T{a}_p+\frac{1}{c_{ep}}{a}_e^T{a}_e\right)dt}$

$J_e=-\frac{1}{2}{q}_e^2{r}_f^T{r}_f+\frac{1}{2}{\displaystyle {\int }_0^T\left(\frac{1}{c_e}{a}_e^T{a}_e+\frac{1}{c_{ep}}{a}_p^T{a}_p\right)dt}$

where r_f = r(T ) and the final time is fixed, while c_p, c_e, c_pe, c_ep are weighting constants. The Nash-equilibrium controls are obtained by applying the results of Corollary 2.3.1 as

$a_p=-c_p\left[\begin{array}{cc}0& I\end{array}\right]P^p\left[\begin{array}{l}r\\ v\end{array}\right],a_e=c_e\left[\begin{array}{cc}0& I\end{array}\right]P^e\left[\begin{array}{l}r\\ v\end{array}\right],$

where P^p, P^e are solutions to the coupled Riccati-ODES:

${\dot{P}}^p\left(t\right)=-P^p\left[\begin{array}{cc}0& I\\ 0& 0\end{array}\right]-\left[\begin{array}{cc}0& 0\\ I& 0\end{array}\right]P^p+c_p{P}^p\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^p+c_e{P}^p\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^e+c_e{P}^e\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^p-\frac{c_e^2}{c_{pe}}{P}^e\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^e$

${\dot{P}}^e\left(t\right)=-P^e\left[\begin{array}{cc}0& I\\ 0& 0\end{array}\right]-\left[\begin{array}{cc}0& 0\\ I& 0\end{array}\right]P^e+c_e{P}^e\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^e+c_p{P}^e\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^p+c_p{P}^p\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^e-\frac{c_p^2}{c_{ep}}{P}^p\left[\begin{array}{cc}0& 0\\ 0& I\end{array}\right]P^p$

$P^p\left(T\right)=q_p^2\left[\begin{array}{cc}I& 0\\ 0& 0\end{array}\right],P^e\left(T\right)=-q_e^2\left[\begin{array}{cc}I& 0\\ 0& 0\end{array}\right].$

It can be easily verified that the solutions to the above ODEs are given by

$P^p\left(t\right)=\frac{1}{c^p}\tilde{p}\left(t\right)\left[\begin{array}{cc}I& \tau I\\ \tau I& {\tau }^{2}I\end{array}\right],P^e\left(t\right)=\frac{1}{c_e}\tilde{e}\left(t\right)\left[\begin{array}{cc}I& \tau I\\ \tau I& {\tau }^{2}I\end{array}\right]$

where τ = T − t is the time-to-go and $\tilde{p}\left(t\right),\tilde{e}\left(t\right)$ are solutions to the following ODES:

$\frac{d\tilde{p}}{d\tau }=-{\tau }^2\left({\tilde{p}}^2+2\tilde{p}\tilde{e}-a{\tilde{e}}^2\right),\tilde{p}\left(0\right)=c_p{q}_p^2,a=\frac{c_p}{c_{pe}}$

$\frac{d\tilde{e}}{d\tau }=-{\tau }^2\left({\tilde{e}}^2+2\tilde{p}\tilde{e}-b{\tilde{p}}^2\right),\tilde{e}\left(0\right)=c_e{q}_e^2,a=\frac{c_e}{c_{ep}}.$

Again we can specialize the result of Theorem 2.3.2 to the LQ case. We consider for this case the system equation given by (2.57) and cost functionals represented as

$-J^{-1}\left(u,w\right)=J^2\left(u,w\right):=J=\frac{1}{2}{x}^T\left(t_f\right)Q_{f}x\left(t_f\right)+\frac{1}{2}{\displaystyle {\int }_{t_0}^{t_f}\left[x^T\left(t\right)Qx\left(t\right)+u^T\left(t\right)R_{u}u\left(t\right)-w^T\left(t\right)R_{w}w\left(t\right)\right]dt,}$

(2.67)

where u is the minimizing player and w is the maximizing player, and without any loss of generality, we can assume R_w = I. Then if we assume V (x(t_f ), t_f ) = ϕ(x(t_f ), t_f ) = $\frac{1}{2}$ x^T Q_fx is quadratic, and also

$V\left(x,t\right)=\frac{1}{2}{x}^T\left(t\right)P\left(t\right)x\left(t\right),$

(2.68)

where P (t) is symmetric and P (T ) = Q_f, then substituting these in the HJIE (2.50), we get the following matrix Riccati ODE:

$\dot{P}\left(t\right)+P\left(t\right)\tilde{A}\left(t\right)+{\tilde{A}}^T\left(t\right)P\left(t\right)+P\left(t\right)\left[B^1\left(t\right){\left(B^1\left(t\right)\right)}^T-B^2\left(t\right)R_u^{-1}{\left(B^2\left(t\right)\right)}^T\right]P\left(t\right)+Q=0,P\left(t_f\right)=Q_f.$

(2.69)

Furthermore, we have the following corollary to Theorem 2.3.2.

Corollary 2.3.2 Consider the two-player zero-sum continuous-time LQ dynamic game described by equations (2.57) and (2.67). Suppose there exist bounded symmetric solutions to the matrix ODE (2.69) for all t ∈ [t₀, t_f ], then there exists a unique feedback saddle-point solution to the dynamic game under closed-loop memoryless information-structure. Further, the unique strategies for the players are given by

$u_{\star }\left(t\right)=-R_u^{-1}{\left(B^2\left(t\right)\right)}^{T}P\left(t\right)x\left(t\right)\text{,}$

(2.70)

$w_{\star }\left(t\right)={\left(B^1\left(t\right)\right)}^{T}P\left(t\right)x\left(t\right)\text{,}$

(2.71)

and the optimal value of the game is given by

$J\left(u_{\star },w_{\star }\right)=\frac{1}{2}{x}_0^{T}P\left(0\right)x_0.$

(2.72)

Remark 2.3.4 In Corollary 2.3.1, the requirement that Q_f ≥ 0 and Qⁱ ≥ 0, i = 1,…, N has been included to guarantee the existence of the Nash-equilibrium solution; whereas, in Corollary 2.3.2 the existence of bounded solutions to the matrix Riccati ODE (2.69) is used. It is however possible to remove this assumption if in addition to the requirement that Q_f ≥ 0 and Q ≥ 0, we also have

$\left[{\left(B^1\left(t\right)\right)}^T{B}^1\left(t\right)-{\left(B^2\left(t\right)\right)}^T{R}_u^{-1}{B}^2\left(t\right)\right]\ge 0\mathrm{\forall }t\in \left[t_0,t_f\right].$

Then the matrix Riccati equation (2.69) admits a unique positive-semidefinite solution [59].

2.4    Notes and Bibliography

The first part of the chapter on dynamic programmimg principle is based on the books [47, 64, 164, 175], while the material on dynamic games is based in most part on the books by Basar and Bernhard [57] and by Basar and Olsder [59]. The remaining material is based on the papers [250, 251]. On the other hand, the discussion on the existence, definiteness and boundedness of the solutions to the coupled matrix Riccati-ODEs are based on the Reference [108]. Moreover, the results on the stochastic case can also be found in [2].

We have not discussed the Minimum principle of Pontryagin [47] because all the problems discussed here pertain to unconstrained problems. However, for such problems, the minimum principle provides a set of necessary conditions for optimality. Details of these necessary conditions can be found in [59]. Furthermore, a treatment of constrained problems using the minimum principle can also be found in [73].