2.11 The exponential family

2.11.1 Definition

It turns out that many of the common statistical distributions have a similar form. This leads to the definition that a density is from the one-parameter exponential family if it can be put into the form

or equivalently if the likelihood of n independent observations from this distribution is

It follows immediately from Neyman’s Factorization Theorem that is sufficient for θ given X.

2.11.2 Examples

Normal mean. If with known then

which is clearly of the above form.

Normal variance. If with θ known then we can express the density in the appropriate form by writing

Poisson distribution. In the Poisson case, we can write

Binomial distribution. In the binomial case we can write

2.11.3 Conjugate densities

When a likelihood function comes from the exponential family, so

there is an unambiguous definition of a conjugate family – it is defined to be the family Π of densities such that

This definition does fit in with the particular cases we have discussed before. For example, if x has a normal distribution with unknown mean but known variance, the conjugate family as defined here consists of densities such that

If we set , we see that

which is a normal density. Although the notation is slightly different, the end result is the same as the one we obtained earlier.

2.11.4 Two-parameter exponential family

The one-parameter exponential family, as its name implies, only includes densities with one unknown parameter (and not even all of those which we shall encounter). There are a few cases in which we have two unknown parameters, most notably when the mean and variance of a normal distribution are both unknown, which will be considered in detail in Section 2.12. It is this situation which prompts us to consider a generalization. A density is from the two-parameter exponential family if it is of the form

or equivalently if, given n independent observations , the likelihood takes the form

Evidently the two-dimensional vector is sufficient for the two-dimensional vector of parameters given X. The family of densities conjugate to such a likelihood takes the form

While the case of the normal distribution with both parameters unknown is of considerable theoretical and practical importance, there will not be many other two-parameter families we shall encounter. The idea of the exponential family can easily be extended to a k-parameter exponential family in an obvious way, but there will be no need for more than two parameters in this book.