Search in book...
Toggle Font Controls
Create new playlist

Name your new playlist

Playlist description (optional)
Sign In

Email address

Password

Forgot Password?

or

Continue with Facebook

Continue with Google
Sign Up

Full Name

Email address

Confirm Email Address

Password

or

Continue with Facebook

Continue with Google

20.1 Probability Review

In this section we briefly introduce the concepts from probability needed for what follows. An understanding of probability and the various identities that arise is essential for the development of entropy.

Consider an experiment $X$ $X$ with possible outcomes in a finite set $X$ $X$ . For example, $X$ $X$ could be flipping a coin and $X = {heads, tails}$ $X = {heads, tails}$ . We assume each outcome is assigned a probability. In the present example, $p (X = heads) = 1 / 2$ $p (X = heads) = 1 / 2$ and $p (X = tails) = 1 / 2$ $p (X = tails) = 1 / 2$ . Often, the outcome $X$ $X$ of an experiment is called a random variable.

In general, for each $x \in X$ $x \in X$ , denote the probability that $X = x$ $X = x$ by

p_{X} (x) = p_{x} = p (X = x) .

$p_{X} (x) = p_{x} = p (X = x) .$

Note that $\sum_{x \in X} p_{x} = 1$ $\sum_{x \in X} p_{x} = 1$ . If $A \subseteq X$ $A \subseteq X$ , let

p (A) = \sum_{x \in A} p_{x},

$p (A) = \sum_{x \in A} p_{x},$

which is the probability that $X$ $X$ takes a value in $A$ $A$ .

Often one performs an experiment where one is measuring several different events. These events may or may not be related, but they may be lumped together to form a new random event. For example, if we have two random events $X$ $X$ and $Y$ $Y$ with possible outcomes $X$ $X$ and $Y$ $Y$ , respectively, then we may create a new random event $Z = (X, Y)$ $Z = (X, Y)$ that groups the two events together. In this case, the new event $Z$ $Z$ has a set of possible outcomes $Z = X \times Y$ $Z = X \times Y$ , and $Z$ $Z$ is sometimes called a joint random variable.

Example

Draw a card from a standard deck. Let $X$ $X$ be the suit of the card, so $X = {clubs, diamonds, hearts, spades}$ $X = {clubs, diamonds, hearts, spades}$ . Let $Y$ $Y$ be the value of the card, so $Y = {two, three, \dots, ace}$ $Y = {two, three, \dots, ace}$ . Then $Z$ $Z$ gives the 52 possibilities for the card. Note that if $x \in X$ $x \in X$ and $y \in Y$ $y \in Y$ , then $p ((X, Y) = (x, y)) = p (X = x, Y = y)$ $p ((X, Y) = (x, y)) = p (X = x, Y = y)$ is simply the probability that the card drawn has suit $x$ $x$ and value $y$ $y$ . Since all cards are equally probable, this probability is 1/52, which is the probability that $X = x$ $X = x$ (namely 1/4) times the probability that $Y = y$ $Y = y$ (namely 1/13). As we discuss later, this means $X$ $X$ and $Y$ $Y$ are independent.

Example

Roll a die. Suppose we are interested in two things: whether the number of dots is odd and whether the number is at least 2. Let $X = 0$ $X = 0$ if the number of dots is even and $X = 1$ $X = 1$ if the number of dots is odd. Let $Y = 0$ $Y = 0$ if the number of dots is less than 2 and $Y = 1$ $Y = 1$ if the number of dots is at least 2. Then $Z = (X, Y)$ $Z = (X, Y)$ gives us the results of both experiments together. Note that the probability that the number of dots is odd and less than 2 is $p (Z = (1, 0)) = 1 / 6$ $p (Z = (1, 0)) = 1 / 6$ . This is not equal to $p (X = 0) \cdot p (Y = 0)$ $p (X = 0) \cdot p (Y = 0)$ , which is $(1 / 2) (1 / 6) = 1 / 12$ $(1 / 2) (1 / 6) = 1 / 12$ . This means that $X$ $X$ and $Y$ $Y$ are not independent. As we’ll see, this is closely related to the fact that knowing $X$ $X$ gives us information about $Y$ $Y$ .

We denote

p_{X, Y} (x, y) = p (X = x, Y = y) .

$p_{X, Y} (x, y) = p (X = x, Y = y) .$

Note that we can recover the probability that $X = x$ $X = x$ as

p_{X} (x) = \sum_{y \in Y} p_{X, Y} (x, y) .

$p_{X} (x) = \sum_{y \in Y} p_{X, Y} (x, y) .$

We say that two random events $X$ $X$ and $Y$ $Y$ are independent if

p_{X, Y} (x, y) = p_{X} (x) p_{Y} (y)

$p_{X, Y} (x, y) = p_{X} (x) p_{Y} (y)$

for all $x \in X$ $x \in X$ and all $y \in Y$ $y \in Y$ . In the preceding example, the suit of a card and the value of the card were independent.

We are also interested in the probabilities for $Y$ $Y$ given that $X = x$ $X = x$ has occurred. If $p_{X} (x) > 0$ $p_{X} (x) > 0$ , define the conditional probability of $Y = y$ $Y = y$ given that $X = x$ $X = x$ to be

p_{Y} (y | x) = \frac{p_{X, Y} (x, y)}{p_{X} (x)} .

$p_{Y} (y | x) = \frac{p_{X, Y} (x, y)}{p_{X} (x)} .$

One way to think of this is that we have restricted to the set where $X = x$ $X = x$ . This has total probability $p_{X} (x) = \sum_{y} p_{X, Y} (x, y)$ $p_{X} (x) = \sum_{y} p_{X, Y} (x, y)$ . The fraction of this sum that comes from $Y = y$ $Y = y$ is $p_{Y} (y | x)$ $p_{Y} (y | x)$ .

Note that $X$ $X$ and $Y$ $Y$ are independent if and only if

p_{Y} (y | x) = p_{Y} (y)

$p_{Y} (y | x) = p_{Y} (y)$

for all $x, y$ $x, y$ . In other words, the probability of $y$ $y$ is unaffected by what happens with $X$ $X$ .

There is a nice way to go from the conditional probability of $Y$ $Y$ given $X$ $X$ to the conditional probability of $X$ $X$ given $Y$ $Y$ .

Bayes’s Theorem

If $p_{X} (x) > 0$ $p_{X} (x) > 0$ and $p_{Y} (y) > 0$ $p_{Y} (y) > 0$ , then

p_{X} (x | y) = \frac{p_{X} (x) p_{Y} (y | x)}{p_{Y} (y)} .

$p_{X} (x | y) = \frac{p_{X} (x) p_{Y} (y | x)}{p_{Y} (y)} .$

The proof consists of simply writing the conditional probabilities in terms of their definitions.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for 20.1 Probability Review

Create new playlist

Sign In

Sign Up

Table of Contents for
20.1 Probability Review