Chapter Seven

The Offense–Defense Rating Method

A natural approach in the science of ratings is to first rate individual attributes of each team or participant, and then combine these to generate a single number that reflects overall strength. In particular, success in most contests requires a strong offense as well as a strong defense, so it makes sense to try to rate each separately before drawing conclusions about who is #1.

But this is easier said than done, especially with regard to rating offensive strength and defensive strength. The problem is, as everyone knows, playing against a team with an impotent defense can make even the most anemic offense to appear stronger than it really is. And similarly, the apparent defensive prowess of a team is a direct consequence on how strong (or weak) the opposition’s offense is. In other words, there is a circular relationship between perceived offensive strength and defensive strength, so each must take the other into account when trying to separately rate offensive power and defensive power. How this is accomplished is the point of this chapter.

OD Objective

The objective is to assign each team (or participant) both an offensive rating (denoted by o_i > 0 for team i) and a defensive rating (denoted by d_i > 0 for team i) that are derived from statistics accumulated from past play. Given any particular sport or contest, there are generally many statistics that might be considered, but scores are the most basic statistics, and actually they do a pretty good job. Moreover, once you understand how our OD method uses scores to produce offensive and defensive ratings, you will then easily see how to use statistics other than scores to build your own systems. For these reasons, we will describe our OD rating technique in terms of scores.

OD Premise

Suppose that m teams are to be rated, and use scoring as the primary measure of performance. Team j is considered to be a good offensive team and deserves a high offensive rating when it scores many points against its opponents, particularly opponents with high defensive ratings. Similarly, team i is a good defensive team and deserves a high defensive rating when it holds its opponents, particularly strong offensive teams, to low scores. Consequently, offensive and defensive ratings are intertwined in the sense that each offensive rating is some function f of all defensive ratings, and each defensive rating is some function g of all offensive ratings.

So, the issue now boils down to determining how the functions f and g should be defined. Since scoring is going to be the primary measure of performance, organize past scores into a score matrix A = [a_ij] in which a_ij is the score that team j generated against team i (average the results if the teams met more than once, and set a_ij = 0 if the two teams did not play each other). Notice that a_ij has a dual interpretation in the sense that

In other words, depending on how it is viewed, a_ij simultaneously reflects relative offensive and defensive capability. This dual interpretation is used to formulate the following definitions of f and g for each team.

Reflect on these definitions for a moment to convince yourself that they make sense by reasoning as follows. To understand the offensive rating, suppose that team * has a weak defense. Consequently, d_* is relatively large. If team j runs up a big score against team *, then a_*j is large. But this large score is moderated by the large d_* in the term a_*j/d_* that appears in the definition of o_j. In other words, scoring a lot of points against a weak defense will not contribute much to the total offensive rating. But if team * has a strong defense so that d_* is relatively small, then a large score against * will not be mitigated by a small d_*, and a_*j/d_* will significantly increase the offensive rating of team j.

The logic behind the defensive rating is similar. Holding a strong offense to a low score drives the term a_i*/o_* downward in the defensive rating for team i, but allowing a weak offense to run up the score produces a larger value for a_i*/o_*, which contributes to a larger defensive rating for team i.

But Which Comes First?

Since the OD ratings are circularly defined, it is impossible to directly determine one set without knowing the other. This means that computing the OD ratings can only be accomplished by successive refinement in which one set of ratings is arbitrarily initialized and then successive substitutions are alternately performed to update (7.2) and (7.3) until convergence is realized.

Alternating Refinement Process

Initialize the defensive ratings with positive values stored in a column vector d₀. These initial ratings can be assigned arbitrarily—we will generally set

Use the values in d₀ to compute a first approximation to the offensive ratings from (7.2), and store the results in a column vector o₁. Now use these values to compute refinements of the defensive ratings from (7.3), and store the results in a column vector d₁. Repeating this alternating substitution process generates two sequences of vectors

If these sequences converge, then the respective offensive and defensive ratings are the components in the vectors

To better analyze this alternating refinement process it is convenient to describe the vectors in this sequence by using matrix algebra along with a bit of nonstandard notation. Given a column vector with positive entries

If A = [a_ij] is the score matrix defined in (7.1), then the OD ratings in (7.2) and (7.3) are written as the two matrix equations

and

Consequently, with d₀ = e, the vectors in the OD sequences (7.4) are defined by the alternating refinement process

and

The Divorce

The interdependence between the o_k’s and d_k’s is easily broken so that the computations need not bounce back and forth between offensive and defensive ratings. This is accomplished with a simple substitution of (7.8) into (7.7) to produce the following two independent iterations.

where o₀ = e = d₀.

However, there is no need to use both of these iterations because if we can produce ratings from one of them, the other ratings are obtained from (7.5) or (7.6). For example, if the offensive iteration (7.9) converges to o, then plugging it into (7.6) produces d. Similarly, if we only iterate with (7.10), and it converges to d, then o is produced by (7.5).

Combining the OD Ratings

Once a set of offensive and defensive ratings for each team or participant has been determined, the next problem is to decide how the two sets should be combined to produce a single rating for each team? There are many methods for combining or aggregating rating lists—in fact, Chapters 14 and 15 are devoted to this issue. But for the OD ratings, the simplest (and perhaps the most sensible) thing to do is to define an aggregate or overall rating to be

In vector notation, the overall rating vector can be expressed as r = o/d, where / denotes component-wise division. Remember that strong offenses are characterized by big o_i’s while strong defenses are characterized by small d_i’s, so dividing the offensive rating by the corresponding defensive rating makes sense because it allows us to retain the “large value is better” interpretation of the overall ratings in r.

Our Recurring Example

Put the scores in Table 1.1 from our recurring five-team example on page 5 into a scoring matrix

Starting with d₀ = e and implementing the alternating iterative scheme in (7.8)–(7.7) requires k = 9 iterations to produce convergence of both of the OD vectors to four significant digits. The results are shown in the following table.

OD ratings for the 5-team example

It is clear from these OD ratings and rankings that UVA, UNC, and DUKE should be respectively ranked third, fourth, and fifth overall. However, the OD rankings for VT and MIAMI are flip flopped, so their positions in the overall rankings are not evident. But using the OD ratio in (7.11) resolves the issue. The overall ratings are shown below.

Overall ratings for the 5-team example

Scoring vs. Yardage

As mentioned in the introduction to this chapter, almost any statistic derived from a competition could be used in place of scores. For football, it is interesting to see what happens when scores are replaced by yards gained (or given up) in our OD method. As part of his M.S. thesis [41] at the College of Charleston, Luke Ingram used the 2005 ACC Football scoring and yardage data to investigate this issue. The scoring data in our small five-team recurring example is a subset of Luke’s experiments. Using a similar subset of Luke’s yardage data for the same five teams yields the yardage matrix

in which

Starting with d₀ = e and implementing the alternating iterative scheme in (7.8)–(7.7) requires k = 6 iterations to produce convergence of both of the OD vectors to four significant digits when the yardage data is used. The results are shown below.

OD ratings for the 5-team example when yardage is used

This time things are not as clearly defined as when scores are used. The overall ratings obtained from the yardage data is as follows.

Overall ratings for the 5-team example when yardage is used

The results in (7.12) are similar to those in (7.13) except that MIAMI and VT have traded places. Experiments tend to suggest that scoring data provides slightly better overall rankings (based on the final standings) for NCAA football than does yardage data. Perhaps this is because the winner is the team with the most points, not necessarily the team with the most total yardage. However, OD ratings based on yardage can be a good indicator of overall strength at particular times in the midst of a season. Of course, results will differ in other sports settings or competitions.

The 2009–2010 NFL OD Ratings

Since we have used the 2009–2010 NFL data in some examples in other places in this book, you might be wondering what the OD ratings look like for the 2009–2010 NFL season. The offensive and defensive ratings were computed using the scoring data for all games (including playoff games), and scores were averaged when two teams played against each other more than once. The results are shown in the following tables.

Offensive ratings for the 2009–2010 NFL season when scoring is used

Defensive ratings for the 2009–2010 NFL season when scoring is used

If the ratio r_i = o_i/d_i is used to aggregate the offensive and defensive ratings to produce a single overall rating for each team, then the results for the 2009–2010 NFL season are as follows.

Overall ratings for the 2009–2010 NFL season determined by the ratio O/D

However, using this simple ratio is not the only (nor necessarily the best) way to aggregate the OD ratings. For example, it puts as much importance on offense as defense. If you want to experiment with some more complicated strategies, then you might try to build some sort of regression model such as

α(o_h − o_w) + β(d_h − d_a) = s_h − s_a,

where o_h, d_h, o_a, d_a are the respective OD ratings for the home and away teams, and s_h and s_a are the respective scores. Warning! Once you start tinkering with ideas like this, you will find that they become infectious. In fact, they are quicksand, and your personal relationships along with your performance at your day job may suffer.

The NFL is not be the best example to illustrate the benefits of our OD method because the NFL is extremely well balanced—much more so than most other sports. The league goes to great lengths to insure this. Consequently, the differentiation between weak and strong offenses (and defenses) is not nearly as pronounced as it is in NCAA competition or in other sports. This means that you won’t be too far off the mark by just using raw score totals (without regard to the level of the competition) to gauge offensive and defensive prowess in the NFL, but in other sports or competitions this may not be the case. For example, when the same scoring matrix A that produces the NFL OD ratings shown above is used to compute

and

the following results are produced.

Average points scored per game during the 2009–2010 NFL season

Average points allowed per game during the 2009–2010 NFL season

If you take a moment to compare the offensive ratings in (7.14) with average points scored in (7.17), you see that while they do not provide exactly the same rankings, they are nevertheless more or less in line with each other—as they should be. Something would be amiss if this were not the case because the same data produced both sets of rankings. The difference is that offensive ratings in (7.14) take the level of competition into account whereas the raw scores in (7.17) do not. Similarly, the defensive ratings (7.15) are more or less in line with the average points allowed shown in (7.18)—all is right in the world.

Mathematical Analysis of the OD Method

The algorithm for computing the OD ratings is really simple—you just alternate the two multiplications (7.7) and (7.8) as described on page 81. Among all of the rating schemes discussed in this book, our OD method is one of the easiest to implement—an ordinary spread sheet can comfortably do this for you, or, if there are not too many competitors, a hand calculator or even pencil-and-paper will do the job. However, the simplicity of implementation belies some of the deeper mathematical issues surrounding the theory of the OD algorithm.

The main theoretical problem concerns the existence of the OD ratings. Since the vectors o and d that contain the OD ratings are defined to be the limits

where

there is a real possibility that o and d may not exist—indeed, there are many cases in which these sequences fail to converge.

A theorem is needed to tell us when convergence is guaranteed and when convergence is impossible. Before developing such a theorem, some terminology, notation, and background information are required. Things hinge on the realization that the OD method is closely related to a successive scaling or balancing procedure on the data matrix A_m×m. To understand this, suppose that the columns of A are first scaled (or balanced) to make all column sums equal to one, and then the resulting matrix is row-scaled to make all row sums equal to one. This can be described as multiplication with diagonal scaling matrices C₁ and R₁ such that AC₁ has all column sums equal to one followed by R₁(AC₁) to force all row sums equal to one. In other words, if

and

then

R₁AC₁e = e (i.e., all row sums are one).

Unfortunately, scaling the rows of AC₁ destroys the column balance achieved in the previous step when we multiplied A by C₁—i.e., the column sums of R₁AC₁ are generally not equal to one. However, the column sums in R₁AC₁ are often closer to being equal to one than those of A. Consequently, the same kind of alternate scaling is performed again, and again, and again, until the data is perfectly balanced in the sense that the row sums as well as the column sums are all equal to one (or at least close enough to one within some prescribed tolerance). Successive column scaling followed by row scaling generates a sequence of diagonal scaling matrices C_k and R_k such that

S_k = R_k · · · R₂R₁AC₁C₂ · · · C_k ≥ 0 where S_ke = e (all row sums are one).

A square matrix S having nonnegative entries and row sums equal to one is called a stochastic matrix. When the row sums as well as the column sums in S are all equal to one, S is said to be a doubly stochastic matrix.

Thus the matrix S_k = R_k · · · R₂R₁AC₁C₂ · · · C_k resulting from successive column and row scalings is a stochastic matrix for every k = 1, 2, 3, . . . . The hope is that S = lim_k→∞ S_k exists and S is doubly stochastic. Alas, this does not always happen, but all is not lost because there is a theorem that tells us exactly when it will happen. Understanding the theorem requires knowledge of the “diagonals” in a square matrix.

Diagonals

The main diagonal of a square matrix A is the set of entries {a₁₁, a₂₂, . . . a_mm}, and this is familiar to most people. However, there are other “diagonals” in A that receive less notoriety. In general, if σ = (σ₁, σ₂, . . . , σ_m) is a permutation of the integers (1, 2, . . . , m), then the diagonal of A associated with σ is defined to be the set

{a_1σ1, a_2σ2, . . . a_mσm}.

For example, in the main diagonal {1, 5, 9} corresponds to the natural order (1, 2, 3), while this and all other diagonals are derived from the six permutations as shown below.

Sinkhorn–Knopp

In 1967 Richard Sinkhorn and Paul Knopp [70] proved a theorem that perfectly applies to the OD theory. Their theorem is as follows.

OD Matrices

As discussed on page 83, statistics other than game scores (such as yardage in football) can be used to compute OD ratings. Throughout the rest of the OD development, it will be understood that A_m×m is a matrix of nonnegative statistics (scores, yardage, etc.), and such a matrix will be called an OD matrix. The major observation is that applying the Sinkhorn–Knopp process to an OD matrix produces OD ratings by reciprocation. This is developed in the following theorem.

The OD Ratings and Sinkhorn–Knopp

In subsequent discussions it is convenient to identify column vectors x with associated diagonal matrices by means of the mapping

Notice that diag {x} e = x = e^T diag {x}. Furthermore, for vectors v with nonzero components and for diagonal matrices D with nonzero diagonal elements, it is straightforward to verify that

We are now in a position to prove the primary OD theorem that tells us when the OD rating vectors exist and when they do not.

Proof. As in earlier discussions, let C_k and R_k denote the respective products

C_k = C₁C₂ · · · C_k and R_k = R_k · · · R₂R₁

of the diagonal scaling matrices generated by the Sinkhorn–Knopp process. The key is to establish a connection between these products and the OD sequences in (7.21). To do this, recall that the diagonal scaling matrices in the Sinkhorn–Knopp process are generated as follows.

Observe that the products C_k and R_k can be expressed as

where r₀ = e, and

This can be verified by writing out the first few iterates. For example, c₁ = (A^T e)^÷, so it is clear from (7.22) that C₁ = diag {c₁}, and C₁e = c₁, and thus

R₁ = diag (AC₁e)^÷ = diag (Ac₁)^÷ = diag {r₁}.

Use this along with (7.20) to write

so that C₂ = C₁C₂ = diag {c₂}. Similarly, this together with (7.20) yields

so that R₂ = R₂R₁ = diag {r₂}. Proceeding inductively establishes (7.23). Now observe that the sequences in (7.24) are simply the reciprocals of the OD sequences in (7.21). In other words,

Again, this is easily seen by writing out the first few terms as shown below and proceeding inductively.

The Sinkhorn–Knopp Theorem on page 89 says that the limits

exist if and only if A has total support. Consequently, (7.23) means that the limits

exist if and only if A has total support, and thus (7.25) ensures that the limits

exist if and only if A has total support.

Cheating a Bit

Not all OD matrices will have the total support property—i.e., not every positive entry will lie on a positive diagonal. Without total support, the OD rating vectors cannot exist because the defining sequences will not converge, so it is only natural to try to cheat a bit in order to force convergence. One rather obvious way of cheating a little is to simply add a small number > 0 to each entry in A. That is, make the replacement

A ← A + ee^T > 0.

This forces all entries to be positive, so total support is automatic, and thus convergence of the OD sequences is guaranteed. If it bothers you to add something to the main diagonal of A because it suggests that a team has beaten itself, you can instead make the replacement

A ← A + (ee^T − I),

which will also guarantee total support.

If is small relative to the nonzero entries in the original OD matrix, then the spirit of the original data will not be overly compromised, and the OD ratings produced by the slightly modified matrix will still do a good job of reflecting relative offensive and defensive strength. However, there is no free lunch in the world, and there is a price to be paid for this bit of deception.

While relative offensive and defensive strengths are not overly distorted by a small perturbation, the rate of convergence can be quite slow. It should be intuitive that as becomes smaller, the number of iterations needed to produce viable results grows larger. In other words, you need to walk a tightrope by balancing the desired quality of your OD ratings against the time you are willing to spend to generate them. However, keep in mind that you may not need many significant digits of accuracy to produce useable ratings.

As demonstrated in (7.25), the OD sequences are nothing more (or less) than the reciprocals of the Sinkhorn–Knopp sequences in (7.24), so the rate of convergence of the OD sequences is the same as convergence rates of the Sinkhorn–Knopp procedure. While estimates of convergence rates of the Sinkhorn–Knopp have been established, they are less than straightforward. Some require you to know the end result of the process so that you cannot compute an a priori estimate while others are “order estimates” where the order constants are not known or not computable. A detailed discussion of convergence characteristics would take us too far astray and would add little to the application of the OD method. Technical convergence discussions are in [30, 46, 72, 73]. Additional information and applications involving the OD method can be found in Govan’s work [34, 35].

ASIDE: HITS

ASIDE: The Birth Of OD

ASIDE: Individual Awards in Sports

ASIDE: More Movie Rankings: OD Meets Netflix

    u = e;
    for i = 1 : maxiter
         m = A u;
         m = 5 (m−min(m)) / (max(m) −min(m)) ;
         u = 1 / ((R −(R>0) . * (em′)).^2)e
    end

¹It was surprising for us to learn just how many power users there are in the Netflix dataset. There are over 13,000 users who have rated more than a thousand movies.