Chapter Four

Keener’s Method

James P. Keener proposed his rating method in a 1993 SIAM Review article [42]. Keener’s approach, like many others, utilizes nonnegative statistics that result from contests (games) between competitors (teams) to create a numerical rating for each team. In some circles these are called power ratings. Of course, once a numerical rating for each team is established, then ranking the teams in order of their ratings is a natural consequence.

Keener’s method is to relate the rating for a given team to the absolute strength of the team, which in turn depends on the relative strength of the team—i.e., the strength of the team relative to the strength of the teams that it has played against. This will become more clear as we proceed. Keener bases everything on the following two stipulations that govern the relationship between a team’s strength and its rating.

Strength and Rating Stipulations

1. The strength of a team should be gauged by its interactions with opponents together with the strength of these opponents.

2. The rating for each team in a given league should be uniformly proportional to the strength of the team. More specifically, if s_i and r_i are the respective strength and rating values for team i, then there should be a proportionality constant λ such that s_i = λr_i, and λ must have the same value for each team in the league.

Selecting Strength Attributes

To turn the two rules above into a mechanism that can be used to compute a numerical rating for each team, let’s assign some variables to various quantities in question. First, settle on some attribute or statistic of the competition or sport under consideration that you think will be a good basis for making relative comparisons of team strength, and let

a_ij = the value of the statistic produced by team i when competing against team j.

For example one of the simplest statistics to consider is the number of times that team i has defeated team j during the current playing season. For ties, assign the participants a value of 1/2 for each time they have tied. In other words, if W_ij is the number of times that team i has recently beaten team j, and if T_ij is the number of times that teams i and j have tied, then you might set

If you think that a more relevant attribute that reflects team i’s power relative to that of team j is the number of points S_ij that i scores against j, then you might define

a_ij = S_ij.

If teams i and j play against each other more than once, then S_ij is the cumulative number of points scored by i against j.

Rather than game scores or number of victories, you might choose to use other attributes of the competition. For example, if you are an old-school football fan who believes that the stronger football teams are those that excel in the running game, then you may want to set

a_ij = the number of rushing yards that team i accumulates against team j.

Similarly, a contemporary connoisseur of the NFL could be convinced that it’s all about passing, in which case

a_ij = the number of passing yards that team i gained against team j.

Regardless of what attribute you select to define your a_ij’s you will have to continually update them throughout the playing season (unless of course you are only evaluating the league after all competition is finished). But as you update your a_ij’s, you have to decide if the value of an attribute at the beginning of the season should have as much influence as its value near the end of the season—i.e., do you want to weight the a_ij’s as a function of time?

All of this flexibility allows for a lot of fiddling and tweaking, and this is what makes building rating and ranking models so much fun. As you can already begin to see, Keener’s method is particularly ripe with opportunities for tinkering, and even more such opportunities will emerge in what is to come.

Laplace’s Rule of Succession

Regardless of the attribute that you select, the statistics concerning your attribute can rarely be used in their raw form to create a successful rating technique. For example, consider scores S_ij. If teams i and j each have a weak defense but a good offense, then they are prone to rack up big scores against each other. On the other hand, if teams p and q each have a strong defense but a weak offense, then their games are likely to be low scoring. Consequently, the large S_ij and S_ji can have a disproportionate effect in any subsequent rating system compared to the small S_pq and S_qp. When comparing teams i and j, it is better to take into account the total number of points scored by setting

Similar remarks hold for other attributes—e.g., rushing yards, passing yards, turnovers, etc.

Proportions a_ij generated by using ratios such as (4.1) may be better than using raw values, but, as Keener points out, we should really be using something like

The motivation for this is Laplace’s rule of succession [29], and the intuition is that if

and p_ij can be interpreted as the probability that team i will defeat team j in the future. If team i defeated team j by a cumulative score of X to 0 in past games where X > 0, then it might be reasonable to conclude that team i is somewhat better than team j (slightly better if X is small or much better if X is large). However, regardless of the value of X > 0, we have that

This suggests that it is impossible for team j to ever beat team i in the future, which is clearly unrealistic. On the other hand, if (4.2) is interpreted as the probability p_ij that team i will defeat team j in the future, then you can see from (4.2) that 0 < p_ij < 1, and if team i defeated team j by a cumulative score of X to 0 in the past, then

which is more reasonable than (4.3). Moreover, if S_ij ≈ S_ji and both are large, then p_ij ≈ 1/2, but as the difference S_ij − S_ji > 0 increases, p_ij gets closer to 1, which makes sense. Consequently, using (4.2) is preferred over (4.1).

To Skew or Not to Skew?

Skewing concerns the issue of how to compensate for the “just because we could” situation in which a stronger team mercilessly runs up the score against a weaker opponent to either enhance their own rating or perhaps to just “rub it in.” If (4.2) is used, then Keener suggests applying a nonlinear skewing function such as

to each of the a_ij’s to help mitigate differences at the upper and lower ends. In other words, replace a_ij by h(a_ij). The graph of h(x) is shown in Figure 4.1, and, as you can see, this function has the properties that h(0) = 0, h(1) = 1, and h(1/2) = 1/2. Using the h(a_ij)’s in place of the a_ij’s has the intended effect of somewhat moderating differences at the upper and lower ranges. Skewing with h(x) also introduces an artificial separation between two raw values near 1/2, which might be helpful in distinguishing between teams of nearly equal strength.

Once you have seen Keener’s skewing function in (4.4) and have played with it a bit, you should be able to construct many other skewing functions of your own. For example, you may want to customize your skewing function so that it is more or less exaggerated in its skewing ability than is h(x), or perhaps you need your function to affect the upper ranges of the a_ij’s differently than how it affects the lower ranges. The value of Keener’s skewing idea is its ability to allow you to “tune” your system to the particular competition being modeled. Skewing is yet another of the countless ways to fiddle with and tweak Keener’s technique.

Figure 4.1 Graph of the skewing function h(x)

Skewing is not always needed. For example, skewing did not significantly affect the NFL rankings in the studies in [34, 35, 1]. This is probably due to the fact that the NFL was well balanced for the years under consideration, so the need for fine tuning was not so great. Of course, things are generally much different for NCAA sports or other competitions, so don’t discount Keener’s skewing idea.

Normalization

Having settled on the attribute that defines your nonnegative a_ij’s, and having decided whether or not you want to skew them by redefining a_ij ← h(a_ij), one last bit of massaging (or normalization) of the a_ij’s is required for situations in which not all teams play the same number of games. In such a case, make the replacement

To understand why this is necessary, suppose that you are using scores S_ij to define your a_ij’s along the lines of (4.1) or (4.2). Teams playing more games than other teams have the possibility of producing more points, and thus inducing larger values of a_ij. This in turn will affect any measure of “strength” that is eventually derived from your a_ij’s. The same is true for statistics other than scores—e.g., if number of yards gained or number of passes completed is your statistic, then teams playing more games can accumulate higher values of these statistics, which in turn will affect any ratings that these statistics produce.

Caution! Skewing has a normalization effect, so if you are using a skewing function and there is not a large discrepancy in the number of games played, then (4.5) may not be necessary—you run the risk of “over normalizing” your data. Furthermore, different situations require different normalization strategies. Using (4.5) is the easiest thing to do and is a good place to start, but you may need to innovate and experiment with other strategies to obtain optimal results—yet another place for fiddling.

Chicken or Egg?

Once the a_ij’s are defined, skewed, and normalized, organize them into a square matrix

A = [a_ij]_m×m, where m = the number of teams in the league.

Representing the data in such a way allows us to apply some extremely powerful ideas from matrix theory to quantify “strength” and to generate numerical ratings. While it is not apparent at this point, it will be necessary to make a subtle distinction between the “strength value” for each team and its “ratings value.” This will become more apparent in the sequel. In spite of the fact that the “strength value” and the “rating value” are not equal, they are of course related. This presents us with a chicken-or-egg situation because we would like to use the “rating values” to measure “strength values” but the “strength” of each team will affect the value of a team’s “rating.”

Ratings

Let’s just jump in blindly and start with ratings. We would like to construct a rating value for each of m teams based on their perceived “strength” (which we still don’t have a handle on) at some time t during the current playing season. Let

r_j(t) = the numerical rating of team j at time t,

and let r(t) denote the column containing each of these m ratings. It is understood that the values r_j(t) in the rating vector r(t) change in time, so the explicit reference to time in the notation can be dropped—i.e., simply write r_j in place of r_j(t) and set

Even though the values in r are unknown to us at this point, the assumption is that they exist and can somehow be determined.

Strength

Now relate the ratings in (4.6) (which at this point exist in theory only) to the concept of “strength.” Recall that Keener’s first stipulation is that the strength of a team should be measured by how well it performs against opponents but tempered by the strength of those opponents. In other words, winning ten games against powder-puff competition shouldn’t be considered the same as winning ten games against powerful opponents.

How well team i performs against team j is precisely what your statistic a_ij is supposed to measure. How powerful (or powerless) team j is is what the rating value r_j is supposed to gauge. Consequently, it makes sense to adopt the following definition.

It is natural to consider the overall or absolute strength of team i to be the sum of team i’s relative strengths compared to all other teams in the league. In other words, it is reasonable to define absolute strength (or simply the strength) as follows.

The Keystone Equation

Keener’s second stipulation concerning the relationship between strength and rating requires that the strength of each team be uniformly proportional to the team’s rating in the sense that there is a proportionality constant λ such that s_i = λr_i for each i. In terms of the rating and strength vectors r and s in (4.6) and (4.7) this says that s = λr for some constant λ. However, (4.8) shows that s = Ar, so the conclusion is that the ratings vector r is related to the statistics a_ij in matrix A by means of the equation

This equation is the keystone of Keener’s method!

In the language of linear algebra, equation (4.9) says that the ratings vector r must be an eigenvector, and the proportionality constant λ must be an associated eigenvalue for matrix A. Information concerning eigenvalues and eigenvectors can be found in many places on the Internet, but a recommended treatment from a printed source is [54, page 489].

At first glance, it appears that the problem of determining the ratings vector r is not only solved, but the solution from (4.9) seems remarkably easy—simply find the eigenvalues and eigenvectors for some matrix A. However, if determining r could not be narrowed down beyond solving a general eigenvalue problem Ar = λr, then we would be in trouble for a variety of reasons, some of which are listed below.

1. For a general m × m matrix A, there can be as many as m different values of λ that will emerge from the solution (4.9). This would present us with the dilemma of having to pick one value of λ over the others. The choice of λ will in turn affect the ratings vector r that is produced because once λ becomes fixed, the ratings vector r is married to it as the solution of the equation (A − λI)r = 0.

2. In spite of the fact that the matrix A of statistics contains only real numbers, it is possible that some (or even all) of the λ’s that emerge from (4.9) will be complex numbers, which in turn can force the associated eigenvectors r to contain complex numbers. Any such r is useless for the purpose of rating and ranking anything. For example, it is meaningless for one team to have rating of 6 + 5i while another team’s rating is 6 − 5i because it is impossible to compare these two numbers to decide which is larger.

3. Even in the best of circumstances in which real values of λ emerge from the solution of (4.9), they could be negative numbers. But even if a positive eigenvalue λ pops out, an associated eigenvector r can (and usually will) contain some negative entries. While having negative ratings is not as bad as having complex ratings, it is nevertheless not optimal.

4. Finally, you have to face the issue of how you are going to actually compute the eigenvalues and eigenvectors of your matrix so that you can extract r. The programming and computational complexity involved in solving Ar = λr for a general square matrix A of significant size requires most people to buy or have access to a software package that is designed for full-blown eigen computations—most such packages are so expensive that they should be shipped in gold-plated boxes.

Constraints

Keener avoids all of the above stumbling blocks by imposing three mild constraints on the amount of interaction between teams and on the resulting statistics a_ij in A_m×m.

   I. Nonnegativity. Whatever attribute you use to determine the statistics a_ij, and however you massage, skew, or normalize them, in the end you must ensure that each statistic a_ij is a nonnegative number—i.e., A = [a_ij] ≥ 0 is a nonnegative matrix.

  II. Irreducibility. There must be enough past competition within your league to ensure that it is possible to compare any pair of teams, even if they had not played against each other. More precisely, if teams i and j are any two different teams in the league, then they must be “connected” by a a series of past contests involving other teams {k₁, k₂, . . . , k_p} such that there is a string of games

The technical name for this constraint is to say that the competition (or the matrix A) is irreducible.

III. Primitivity. This is just a more stringent version of the irreducibility constraint in II in that we now require each pair of teams to be connected by a uniform number of games. In other words, the connection between teams i and j in (4.10) involves a chain of p games with positive statistics. For a different pair of teams, II only requires that they be connected, but they could be connected by a chain of q games with positive statistics in which p ≠ q. Primitivity requires that all teams must be connected by the same number of games—i.e., there must exist a single value p such that (4.10) holds for all i and j. The primitivity constraint is equivalent to insisting that A^p > 0 for some power p. The reason to require primitivity is because it makes the eventual computation of the ratings vector r easy to execute.

Perron–Frobenius

By imposing constraints I and II described above, the powerful Perron–Frobenius theory can be brought to bear to extract a unique ratings vector from Keener’s keystone equation Ar = λr. There is much more to the complete theory than can be presented here (and more than is needed), but its essence is given below—the complete story is given in [54, page 661].

Important Properties

Notice that by virtue of being defined by the Perron vector for A, the ratings vector r is not only uniquely determined by the statistics a_ij in A, but r also has the properties that each team rating is such that

This is important because it puts everything on a level playing field in the sense that it allows for an interpretation of strength in terms of percentages—e.g., a nice pie chart can be made to graphically illustrate the strength of one team relative to the rest of the league. Having the ratings sum to one means that whenever the ratings for a particular team increases, it is necessary for one or more ratings of other teams to decrease, and thus a balance is maintained throughout a given playing season, or from year-to-year, or across different playing seasons. Furthermore, this balance allows level comparisons of ratings derived from two different attributes. For example, if we constructed one set of Keener ratings based on scores S_ij and another set of Keener ratings based on wins W_ij (and ties), then the two ratings and resulting rankings can be compared, reconciled, and even aggregated to produce what are sometimes called consensus (or ensemble) ratings and rankings. Techniques of rank aggregation are discussed in detail in Chapter 14.

Computing the Ratings Vector

Given that you have ensured that all statistics a_ij are nonnegative (i.e., A ≥ 0), then the first thing to do is to check that the irreducibility requirement described in constraint II is satisfied. Let’s assume that it is. If it isn’t, something will have to be done to fix this, and possible remedies are presented on page 39. Once you are certain that A is irreducible, then there are two options for computing r.

Forcing Irreducibility and Primitivity

As explained in the discussion of the constraints on page 35, irreducibility and primitivity are conditions that require sufficient interaction between the teams under consideration, and these interactions are necessary for the ratings vector r to be well defined and to be computable by the power method. This raises the natural question: “How do we actually check to see if our league (or our matrix A) has these kinds of connectivity?”

There is no short cut for checking irreducibility. It boils down to checking the definition—i.e., for each pair of teams (i, j) in your league, you need to verify that there has been a series of games

i ↔ k₁ ↔ k₂ ↔ . . . ↔ k_p ↔ j with a_ik1 > 0, a_k1k2 > 0, . . . , a_kpj > 0

that connects them. Similarly, for primitivity you must verify that there is a uniform number of games that connects each pair of teams in your league. These are tedious tasks, but a computer can do the job as long as the number of teams m is not too large.

However, the computational aspect of checking connectivity is generally not the biggest issue because for most competitions it is rare that both kinds of connectivity will be present. This is particularly true when you are building ratings (and rankings) on a continual basis (e.g., week-to-week) from the beginning of a competition and updating them throughout the playing season. It is highly unlikely that you will have sufficient connectivity to ensure irreducibility and primitivity in the early parts of the playing season, and often these connectivity conditions will not be satisfied even near or at the end of a season.

So, the more important question is: “How can we force irreducibility or primitivity into a competition so that we don’t ever have to check for it?” One particularly simple solution is to replace A by a small perturbation of itself by redefining

in which e is a column of all ones and > 0 is a number that is small relative to the smallest nonzero entry in the unperturbed matrix A. The effect is to introduce an artificial game between every pair of teams such that the statistics of the artificial games are small enough to not contaminate the story that the real games are trying to tell you. Notice that in the perturbed competition every team is now directly connected (with positive statistics) to every other team,¹ and therefore both irreducibility and primitivity are guaranteed.

A less stringent perturbation can be used to ensure primitivity when you know that there has been enough games played to guarantee irreducibility. If the competition (or the matrix A) is irreducible, then adding any value > 0 to any single diagonal position in A will produce primitivity. In other words, adding one artificial game between some team and itself with a negligibly small game statistic, or equivalently, redefining A as

A ← A + E, where E = e_i in which = (0, 0, . . . , 1, 0, . . . , 0)

will produce primitivity [54, page 678].

Summary

Now that all of the pieces are on the table, let’s put them together. Here is a summary of how to build rating and ranking systems from Keener’s application of the Perron–Frobenius theory.

1. Begin by choosing one particular attribute of the competition or sport under consideration that you think will be a good basis for making comparisons of each team’s strength relative to that of the other teams in the competition. Examples include the number of times team i has defeated (or tied) team j, or the number of points that team i has scored against team j.

— More than one scheme can be constructed for a given competition by taking more specific features into account—e.g., the number of rushing (or passing) yards that one football team makes against another, or the number of three-point goals or free throws that one basketball team makes against another. Even defensive attributes can be used—e.g., the number of pass interceptions that one football team makes against another, or the number of shots in basketball that team i has blocked when playing team j.

— The various ratings (and rankings) obtained by using the finer aspects of the competition can be aggregated to form consensus (or ensemble) ratings (or rankings). For example, you can construct one or more Keener schemes based on offensive attributes and others based on defensive attributes, and then aggregate the resulting ratings into one master rating and ranking list. The details describing how this is accomplished are presented in Chapter 14. Before straying off into rank aggregation, first nail down a solid system that is based on one specific attribute.

2. Whatever attribute is chosen, compile statistics from past competitions, and set

a_ij = the value of the statistic produced by team i when competing against team j.

It is absolutely necessary that each a_ij be a nonnegative number!

3. Massage the raw statistics a_ij in step 2 to account for anomalies. For example, if you use a_ij = S_ij = the number of points that team i scores against team j, then as explained on page 31, you should redefine a_ij to be

4. If after massaging the data you feel that there is an imbalance in the sense that some a_ij’s are very much larger (or smaller) than they should be (perhaps due to a team artificially running up the statistics on a weak opponent), then restore a balance by constructing a skewing function such as the one in the discussion on page 31 given by

and make the replacement a_ij ← h(a_ij).

5. If not all teams have played the same number of games, then account for this by normalizing the a_ij’s in step 4 above by making the replacement

and organize these numbers into a nonnegative matrix A = [a_ij] ≥ 0.

6. Either check to see that there has been enough competition in your league to guarantee that it (or matrix A) satisfies the irreducibility and primitivity conditions defined on page 36, or else perturb A by adding some artificial games with negligible game statistics so as to force these conditions as described on page 39.

— If you are not sure (or just don’t want to check) if constraints I and II hold, then both irreducibility and primitivity can be forced by making the replacement A ← A + ee^T, where e is a column of all ones and > 0 is small relative to the smallest positive a_ij in the unperturbed A.

— If irreducibility is already present—i.e., if for each pair of teams i and j there has been a series of games such that

i ↔ k₁ ↔ k₂ ↔ . . . ↔ k_p ↔ j with a_ik1 > 0, a_k1k2 > 0, . . . , a_kpj > 0

for some p (which can vary with the pair (i, j) being considered)—then the league (or matrix A) can be made primitive (i.e., made to have a uniform p that works for all pairs (i, j)) by simply adding > 0 to any single diagonal position in A. This creates an artificial game between some team and itself with a negligible statistic, and it amounts to making the replacement A ← A + e_i, where > 0 and = (0, 0, . . . , 1, 0, . . . , 0) for some i.

7. Compute the rating vector r by using the power method described on page 38. If you have forced irreducibility or primitivity as summarized in step 6 above, then slightly modify the power iteration (4.12) on page 38 as shown below.

— For the perturbed matrix A + ee^T, execute the power method as follows.

• Initially set r ← (1/m)e, where e is a column of all ones.

• Repeat the following steps until the entries in r converge to a prescribed number of significant digits.

BEGIN
             
REPEAT

— The power method changes a bit when the perturbed matrix A + e_i is used.

• Initially set r ← (1/m)e, where e is a column of all ones.

• Repeat the following until each entry in r converges to a prescribed number of significant digits.

BEGIN
          
REPEAT

— These steps are simple enough that they can be performed manually in any good spreadsheet environment.

The 2009–2010 NFL Season

To illustrate the ideas in this chapter we used the scores for the regular seventeen-week 2009–2010 NFL season to build Keener ratings and rankings. The teams were ordered alphabetically as shown in the following list of team names.

Using this ordering, we set

S_ij = the cumulative number of points scored by team i against team j

during regular-season games. The matrix containing these raw scores using our ordering is shown below. For example, S₁₂ = 10 in the following matrix indicates that the BEARS scored 10 points against the BENGALS during the regular season, and S₂₁ = 45 means that the BENGALS scored 45 points against the BEARS.

NFL 2009–2010 regular season scores

From these raw scores we used (4.2) on page 31 to construct the matrix

and then we applied Keener’s skewing function h(x) in (4.4) on page 31 to each entry in this matrix to form the nonnegative matrix

Normalization as described on page 32 is not required because all teams played the same number of games (16 of them—each team had one bye). Furthermore, A is primitive (and hence irreducible) because A² > 0, so there is no need for perturbations. The Perron value for A is λ ≈ 15.832 and the ratings vector r (to 5 significant digits) is shown below.

NFL 2009–2010 Keener ratings

After the ratings are sorted in descending order, the following ranking of the NFL teams for the 2009–2010 season is produced.

NFL 2009–2010 Keener rankings

Most who are familiar with the 2009–2010 NFL season would probably agree that this ranking (and the associated ratings) looks pretty good in that it is an accurate reflection of what happened in the post-season playoffs. The SAINTS ended up being ranked #1, and, in fact, the SAINTS won the Super Bowl! They defeated the COLTS by a score of 31 to 17. Furthermore, as shown in Figure 5.2 on page 61, the top ten teams in our rankings each ended up making the playoffs during 2009–2010, and this alone adds to the credibility of Keener’s scheme.

The COLTS were #5 in our ratings, but they probably would have rated higher if they had not rolled over and deliberately given up their last two games of the season to protect their starters from injury. It might be an interesting and revealing exercise to either delete or somehow weight the scores from the last two regular season games for some (or all) of the teams and compare the resulting ratings and rankings with those above—we have not done this, so if you follow up on your own, then please let us know the outcome.

On the other hand, after reviewing a replay of Super Bowl XLIV, it could be argued that the COLTS actually looked like a #5 team against the SAINTS, and either a healthy PATRIOTS or CHARGERS team might well have given the SAINTS a greater challenge than did the COLTS. Another interesting project would be to weight the scores so that those at the beginning of the season and perhaps some at the end do not count for as much as scores near the crucial period after midseason—and this might be done on a team-by-team basis.

Jim Keener vs. Bill James

Wayne Winston begins his delightful book [83] with a discussion of what is often called the Pythagorean theorem for baseball that was formulated by Bill James, a well-known writer, historian, and statistician who has spent much of his time analyzing baseball data. James discovered that the percentage of wins that a baseball team has in one season is closely approximated by a Pythagorean expectation formula that states

This is a widely used idea in the world of quantitative sports analysis—it has been applied to nearly every major sport on the planet. However, each different sport requires a different exponent in the formula. In other words, after picking your sport you must then adapt the formula by replacing ρ² with ρ^x, where the value of x is optimized for your sport. Winston suggests using the mean absolute deviation (or MAD) for this purpose. That is, if

and

and if there are m teams in the league, then the mean absolute deviation for a given value of x is

or equivalently, in terms of the vector 1-norm [54, page 274],

Once you choose a sport then it’s your job to find the value x* that minimizes MAD(x) for that sport. And if you are really into tweaking things, then “points” in (4.16) can be replaced by other aspects of your sport—e.g., in football it is an interesting exercise to see what happens when you use

Let’s try out James’s Pythagorean idea for the 2009–2010 NFL regular season. In other words, estimate

and where x is determined from the 2009–2010 NFL scoring data given on page 43. Then let’s compare these results with how well the Keener ratings in (4.13) and (4.14) estimate the winning percentages. The percentage of wins for each NFL team during the regular 2009–2010 season is shown in the following table,² and this is our target.

Percentage of regular-season wins for the NFL 2009–2010 season

The optimal value x* of the exponent x in (4.17) is determined by brute force. In other words, compute MAD(x) for sufficiently many different values of x and then eyeball the results to identify the optimal x*. Using the 2009–2010 NFL scoring data on page 43 we computed MAD(x) for 1500 equally spaced values of x between 1 and 4 to estimate

The graph of MAD(x) is shown in Figure 4.2. This means that the predictions produced by James’s generalized Pythagorean formula 1/(1 + ρ^x*) were only off by an average 6.21% per team for the 2009–2010 NFL season. Not bad for such a simple formula!

Figure 4.2 Optimal Pythagorean exponent for the NFL

Winston reports in [83] that Daryl Morey, the General Manager of the Houston Rockets, made a similar calculation for earlier NFL seasons (it’s not clear which ones), and Morey arrived at a Pythagorean exponent of 2.37. Intuition says that the optimal x* should vary with time. However, Morey’s results together with with Winston’s calculations and our value suggest that the seasonal variation of x* for the NFL may be relatively small. Moreover, Winston reports that MAD(2.37) = .061 for the 2005–2007 NFL seasons, which is pretty close to our MAD(2.27) = .062 value for the 2009–2010 season.

Now let’s see if the Keener ratings r in (4.13) on page 43 can produce equally good results. There is nothing to suggest that Keener’s ratings are Pythagorean entities that should work in any kind of nonlinear estimation formula. However, the correlation coefficient between the ratings r in (4.13) and the win percentages w in (4.18) is

This indicates a strong linear relationship between r and the data in w [54, page 296] so rather than trying to force the ratings r_i into a Pythagorean paradigm, we should be looking for optimal linear estimates of the form α + βr_i ≈ ω_i, or equivalently αe + βr ≈ w. The Gauss–Markov theorem [54, page 448] says that “best” values of α and β are those that minimize , where A = , x = , and w = . Finding α and β boils down to solving the normal equations A^TAx = A^T w [54, page 226], which is two equations in two unknowns in this case. Solving the normal equations for x is a straightforward calculation that is hardwired into most decent hand-held calculators and spreadsheets. For the 2009–2010 NFL data, the solution (to five significant places) is α = −1.2983 and β = 57.545, so the least squares estimate for the percentage of regular season wins based on the Keener (Perron–Frobenius) ratings is (rounded to five places)

Est Win% for team i = α + βr_i = −1.2983 + 57.545 r_i.

The following table shows the actual win percentage (rounded to three places) along with the Pythagorean and Keener estimates for each NFL team in the 2009–2010 regular season.

And now the punch line. The MAD for the Keener estimate is (to three significant places)

In other words, by using the Keener ratings to predict the percentage of wins in the 2009–2010 NFL season, we were only off by an average of 5.91% per team, whereas the optimized Pythagorean estimate was off by an average of 6.28% per team. MAD was used because Winston uses it in [83], but the mean squared error (or MSE) that averages the sum of squares of the errors is another common metric. For the case at hand we have

MSE_Pythag = .0065 and MSE_Keener = .0050.

• Bottom Line. Keener (Perron–Frobenius really) wins the shootout for 2009–2010!

It is reasonable that Perron–Frobenius (aka James Keener) should be at least as good as Pythagoras (aka Bill James) because the latter does not take into account the level of difficulty (or ease) required for a team to score or to give up points whereas the former does. Is Keener’s margin of victory significant? You decide. If your interest is piqued, then conduct more experiments on your own, and let us know what you discover.

Back to the Future

Suppose that you were able to catch a ride with Marty McFly in Dr. Emmett Brown’s plutonium powered DeLorean back to the beginning of the 2009–2010 NFL season, and, just like Biff Tannen who took the 2000 edition of Grays Sports Almanac back to 1955 to make a fortune, you were able to jot Keener’s ratings (4.13) on page 43 on the palm of your hand just before you hopped in next to the flux capacitor. How many winners would you be able to correctly pick by using these ratings? Of course, the real challenge is to try to predict the future and not the past, but the exercise of looking backwards is nevertheless a valuable metric by which to gauge a rating scheme and compare it with others.

Since most ratings do not account for a home-field advantage, making either hindsight or foresight predictions usually means that you must somehow incorporate a “home-field factor.” If you could take the Keener ratings (4.14) on page 44 back to the beginning of the 2009–2010 NFL season to pick winners from all 267 games during the 2009–2010 NFL season, you would have to add a home-field advantage factor of .0008 to the Keener rating for the home team to account for the fact that the average difference between the home-team score and the away-team score was about 2.4.³ By doing so you would correctly pick 196 out of the 267 games for a winning percentage of about 73.4%. This is the hindsight accuracy.

It is interesting to compare Keener’s hindsight predictions with those that might have been obtained had you taken back one of the other ratings that are generally available at the end of the season. There is no shortage of rating sites available on the Internet, but most don’t divulge the details involved in determining their ratings. If you are curious how well many of the raters performed for the 2009–2010 NFL season, see Todd Beck’s prediction tracker Web site at www.thepredictiontracker.com/nflresults.php, where many different raters are compared.

Can Keener Make You Rich?

The Keener ratings are pretty good at estimating the percentage of wins that a given team will have, but if you are a gambler, then that’s probably not your primary concern. Making money requires that a bettor has to outfox the bookies, and this usually boils down to beating their point spreads (Chapter 9 on page 113 contains a complete discussion of point spreads and optimal “spread ratings”). The “players” who are reading this have no doubt already asked the question, “How do I use the Keener ratings to predict the spread in a given matchup?” Regrettably, you will most likely lose money if you try to use Keener to estimate point spreads. For starters, the team-by-team scoring differences are poorly correlated with the corresponding differences in the Keener ratings. In fact, when 264 games from the 2009–2010 NFL season were examined, the correlation between differences in game scores and the corresponding differences in the Keener ratings was calculated from (4.20) to be only about .579. In other words, there is not much in the way of a linear relation between the two.

In fact, we would go so far as to say that it is virtually impossible for any single list of rating values to generate accurate estimates of point spreads in American football. General reasons are laid out in detail in Chapter 9 on page 113, but while we are considering Keener ratings, we might ask why they in particular should not give rise to accurate spread estimates. Understanding the answer, requires an appreciation of what is hiding underneath the mathematical sheets. In a nutshell, it is because Keener’s technique is the first cousin of a Markov chain defined by making a random walk on the thirty-two NFL teams, and as such, the ratings only convey information about expected wins over a long time horizon. Markovian methods are discussed in detail in Chapter 6 on page 67.

To understand this from an intuitive point of view, suppose that a rabid fan forever travels each week from one NFL city to another and the decision of where to go next week is determined by the entries in the matrix A = [a_ij] that contains your statistics. If you normalize the columns of A to make each column sum to one, then, just as suggested on page 31, a_ij can be thought of as the probability that team i will beat team j. But instead of using this interpretation, you can also think of a_ij as being

a_ij = Probability of traveling to city i given that the fan is currently in city j.

If a_ij > a_kj, then, relative to team j, team i is stronger than k. But in terms of our traveling fan this means that he is destined to spend a greater portion of his life in city i than city k after being in city j. Applying this logic across all cities j suggests that, in the long run, more time is spent in cities with winning teams—the higher the winning percentage, the more time spent. In other words, if we could track our fan’s movements to determine exactly what proportion of his existence is spent in each city, then we have in effect gauged the proportion of wins that each team is expected to accumulate in the long run.

The intuition is as follows. The sequence x₀, x₁, x₂, . . . generated by power method (4.12) on page 38 converges to the Keener ratings r_K. Normalizing the Keener matrix at the outset to make each column sum equal one obviates the need to normalize the iterates (i.e., dividing by ν_k in (4.12)) at each step. The power method simplifies to become

and the iterates x_k estimate the path of our fan. The x₀ initiates his trek by giving him an equal probability of starting in any given city. Markov chain theory [54, page 687] ensures that entry i in x_k is the probability that the fan is in city i on week k. If x_k → r_F, then [r_F]_i is the expected percentage of time that the fan spends in city i over an infinite time horizon. The different normalization strategies produce small differences between r_K and r_F,⁴ but the results are qualitatively the same. Thus Keener’s ratings mirror the fan’s expected time in each city, which in turn implies that Keener reflects only long-run win percentages.

Conclusion

Keener’s ratings are in essence the result of a limiting process that reflects long-term winning percentages. They say nothing about the short-term scheme of things, and they are nearly useless for predicting point spreads. So, NO—Keener’s idea won’t make you rich by helping you beat the spread, but it was pivotal in propelling two young men into becoming two of the worlds’s most wealthy people—if you are curious, then read the following aside.

ASIDE: Leaves in the River

¹It is permissible to put zeros in the diagonal positions of E if having an artificial game between each team and itself bothers you.

²For convenience, percent numbers *% are converted by multiplying decimal values by 100.

³This is not exactly correct because while a “home team” is always designated, three games—namely on 10/25/09 (BUCS–PATRIOTS), 12/3/09 (BILLS–JETS), and 2/7/10 (COLTS–SAINTS)—were played on neutral fields, but these discrepancies are not significant enough to make a substantial difference.

⁴The intuition is valid for NFL data because the variation in the column sums of Keener’s A is small, so the alternate normalization strategy amounts to only a small perturbation. The difference between r_K and r_M can be larger when the variation in the column sums in A is larger.