2.11 The exponential family
2.11.1 Definition
It turns out that many of the common statistical distributions have a similar form. This leads to the definition that a density is from the one-parameter exponential family if it can be put into the form
or equivalently if the likelihood of n independent observations from this distribution is
It follows immediately from Neyman’s Factorization Theorem that is sufficient for θ given X.
2.11.2 Examples
Normal mean. If with known then
which is clearly of the above form.
Normal variance. If with θ known then we can express the density in the appropriate form by writing
Poisson distribution. In the Poisson case, we can write
Binomial distribution. In the binomial case we can write
2.11.3 Conjugate densities
When a likelihood function comes from the exponential family, so
there is an unambiguous definition of a conjugate family – it is defined to be the family Π of densities such that
This definition does fit in with the particular cases we have discussed before. For example, if x has a normal distribution with unknown mean but known variance, the conjugate family as defined here consists of densities such that
If we set , we see that
which is a normal density. Although the notation is slightly different, the end result is the same as the one we obtained earlier.
2.11.4 Two-parameter exponential family
The one-parameter exponential family, as its name implies, only includes densities with one unknown parameter (and not even all of those which we shall encounter). There are a few cases in which we have two unknown parameters, most notably when the mean and variance of a normal distribution are both unknown, which will be considered in detail in Section 2.12. It is this situation which prompts us to consider a generalization. A density is from the two-parameter exponential family if it is of the form
or equivalently if, given n independent observations , the likelihood takes the form
Evidently the two-dimensional vector is sufficient for the two-dimensional vector of parameters given X. The family of densities conjugate to such a likelihood takes the form
While the case of the normal distribution with both parameters unknown is of considerable theoretical and practical importance, there will not be many other two-parameter families we shall encounter. The idea of the exponential family can easily be extended to a k-parameter exponential family in an obvious way, but there will be no need for more than two parameters in this book.