Fiu, Cirminiello, Mitchell on TV - Campus Insiders | Buy College Football Tickets

Compu-Picks: 2011 - How it Did Part 3

Mr Pac Ten
Posted Mar 15, 2012


2011 Compu-Picks 2011 Review - How it Did, Part Three: Reviewing Compu-Picks' win probability distribution

As you may remember, one of the things I did when creating pre-season projection using Compu-Picks was show win distribution probabilities. Below is a summary of how those probability distributions did.

Overall, the system did extremely well evaluating probability distributions, neither being biased towards the middle or towards the tail. The below table shows a breakdown of how many teams finished in each range, compared to the theoretical expectation (where an even number of teams would finish in each probability bucket). There's plenty of noise, but there's no particular bias towards the tails (19 teams finished under the 10th percentile or over the 90th percentile, compared to the theoretical expectation of 24) or the middle (22 teams finished between the 40th and 60th percentile compared to the theoretical expectation of 24).

Min Max Number Expected
0.0% 5.0% 2 6
5.0% 10.0% 7 6
10.0% 15.0% 10 6
15.0% 20.0% 8 6
20.0% 25.0% 6 6
25.0% 30.0% 6 6
30.0% 35.0% 4 6
35.0% 40.0% 8 6
40.0% 45.0% 5 6
45.0% 50.0% 9 6
50.0% 55.0% 5 6
55.0% 60.0% 3 6
60.0% 65.0% 4 6
65.0% 70.0% 3 6
70.0% 75.0% 7 6
75.0% 80.0% 9 6
80.0% 85.0% 3 6
85.0% 90.0% 11 6
90.0% 95.0% 4 6
95.0% 100.0% 6 6

To clarify exactly what this table means and how it was calculated, a bit of explanation is in order. I'll use LSU, who went a perfect 12-0 in the regular season, as an example. The system projected them at 7.93 wins, so clearly they did much better than projected, but what does that mean on a probibility basis? In this case, they were projected to have a 5% chance of going 12-0, so they did better than 95% of the simulations, and equalled 5% of them. For the sake of this calculation, that would mean that they finished at the 97.5% percentile (95% + 5%/2). The actual number was 97.6% since the 5% display was rounded, but I think you get the general idea.

A full list of how each team did compared to their probability distributions is below:

Team E(wins) Actual Diff Percentile
Kansas State 4.71 10 5.29 99.0%
Arkansas State 4.66 10 5.34 97.9%
Louisiana State 7.93 12 4.07 97.6%
Baylor 4.50 9 4.50 96.8%
Houston 8.47 12 3.53 96.1%
Iowa State 2.47 6 3.53 95.1%
Wyoming 4.67 8 3.33 94.3%
Michigan 6.24 10 3.76 93.5%
Georgia 6.25 10 3.75 93.5%
Louisiana Tech 4.94 8 3.06 90.1%
Louisiana-Lafayette 4.84 8 3.16 89.0%
Marshall 3.23 6 2.77 88.9%
Utah State 4.11 7 2.89 88.6%
Colorado 1.27 3 1.73 88.5%
Oklahoma State 8.48 11 2.52 88.4%
Rutgers 4.91 8 3.09 87.6%
Eastern Michigan 3.23 6 2.77 87.5%
Clemson 6.03 9 2.97 87.2%
Southern California 7.37 10 2.63 86.1%
Vanderbilt 3.82 6 2.18 85.5%
North Texas 2.73 5 2.27 85.1%
Toledo 5.54 8 2.46 83.7%
Michigan State 8.14 10 1.86 83.3%
Western Kentucky 4.95 7 2.05 82.0%
Temple 5.81 8 2.19 79.8%
California 5.18 7 1.82 79.1%
New Mexico State 2.58 4 1.42 78.3%
Virginia 6.11 8 1.89 77.7%
Stanford 9.26 11 1.74 77.1%
Florida International 6.00 8 2.00 76.7%
Wake Forest 4.46 6 1.54 75.8%
Wisconsin 8.36 10 1.64 75.8%
San Diego State 6.42 8 1.58 75.5%
Tulsa 6.50 8 1.50 74.2%
Northern Illinois 7.26 9 1.74 74.1%
Washington 5.62 7 1.38 74.1%
Virginia Tech 9.45 11 1.55 73.6%
Cincinnati 7.57 9 1.43 71.6%
Arkansas 8.66 10 1.34 70.7%
Ball State 4.67 6 1.33 70.5%
Southern Mississippi 8.65 10 1.35 68.8%
Alabama 9.95 11 1.05 66.3%
Louisville 6.06 7 0.94 66.1%
Georgia Tech 6.86 8 1.14 64.7%
Kent 4.26 5 0.74 63.2%
Oregon 10.16 11 0.84 62.5%
Iowa 6.25 7 0.75 60.9%
San Jose State 4.56 5 0.44 59.2%
Texas-El Paso 4.71 5 0.29 57.4%
Penn State 8.49 9 0.51 56.8%
East Carolina 4.82 5 0.18 54.6%
Ohio 8.40 9 0.60 54.1%
Northwestern 5.83 6 0.17 53.1%
Miami (Ohio) 4.07 4 -0.07 51.3%
Purdue 6.00 6 0.00 50.2%
Florida 6.16 6 -0.16 49.8%
Buffalo 3.32 3 -0.32 49.7%
Texas 7.09 7 -0.09 47.3%
Miami (Florida) 6.22 6 -0.22 47.1%
Boise State 10.76 11 0.24 46.6%
Nevada-Las Vegas 2.42 2 -0.42 45.7%
Nebraska 9.01 9 -0.01 45.2%
Western Michigan 7.15 7 -0.15 45.2%
Illinois 6.33 6 -0.33 45.2%
Bowling Green State 5.43 5 -0.43 44.4%
South Carolina 9.92 10 0.08 44.1%
Texas Christian 10.04 10 -0.04 43.4%
Syracuse 5.59 5 -0.59 42.5%
Brigham Young 9.22 9 -0.22 41.1%
Memphis 2.86 2 -0.86 39.5%
Connecticut 5.68 5 -0.68 39.4%
Auburn 7.62 7 -0.62 38.1%
Alabama-Birmingham 3.97 3 -0.97 37.7%
Southern Methodist 7.61 7 -0.61 37.6%
Pittsburgh 6.78 6 -0.78 37.5%
Rice 4.93 4 -0.93 37.1%
Mississippi State 6.78 6 -0.78 36.8%
Washington State 5.02 4 -1.02 34.6%
West Virginia 9.56 9 -0.56 34.5%
North Carolina State 7.88 7 -0.88 34.0%
Utah 8.13 7 -1.13 30.9%
Oklahoma 9.72 9 -0.72 29.8%
UCLA 7.30 6 -1.30 29.2%
Missouri 8.19 7 -1.19 28.0%
Nevada 7.96 7 -0.96 27.5%
North Carolina 8.43 7 -1.43 25.9%
Tulane 3.79 2 -1.79 25.5%
Arizona 5.91 4 -1.91 22.2%
Notre Dame 9.38 8 -1.38 22.0%
Colorado State 5.05 3 -2.05 22.0%
Akron 2.99 1 -1.99 20.9%
Boston College 6.05 4 -2.05 20.6%
Kentucky 6.91 5 -1.91 20.5%
Minnesota 4.90 3 -1.90 19.2%
Oregon State 5.31 3 -2.31 19.0%
Arizona State 8.01 6 -2.01 18.3%
Florida Atlantic 3.16 1 -2.16 17.8%
Louisiana-Monroe 6.28 4 -2.28 16.4%
Duke 5.37 3 -2.37 15.7%
Army 5.64 3 -2.64 15.6%
Texas Tech 7.32 5 -2.32 15.2%
Air Force 8.71 7 -1.71 14.6%
Fresno State 6.75 4 -2.75 14.4%
Navy 7.60 5 -2.60 12.7%
Idaho 4.61 2 -2.61 12.6%
Central Florida 7.68 5 -2.68 11.7%
Florida State 10.19 8 -2.19 11.4%
Texas A&M 8.73 6 -2.73 11.1%
Kansas 4.62 2 -2.62 10.9%
South Florida 8.02 5 -3.02 10.8%
Hawaii 8.95 6 -2.95 10.5%
Tennessee 7.85 5 -2.85 9.0%
Maryland 5.15 2 -3.15 8.7%
New Mexico 3.53 1 -2.53 8.3%
Middle Tennessee State 5.46 2 -3.46 8.2%
Central Michigan 6.64 3 -3.64 7.9%
Indiana 4.00 1 -3.00 7.5%
Mississippi 5.79 2 -3.79 5.1%
Troy 7.56 3 -4.56 3.8%
Ohio State 10.20 6 -4.20 2.7%

You may have noticed that the win distribution probabilities don't exactly correlate to the difference between actual and expected win totals. As an example of why, consider two teams (team A and team B), each projected at exactly six wins. Team A has every game rated an absolute toss-up, while team B has six games rated at a 90% chance of winning and another six games rated at only a 10% chance of winning. In each case, the win expectation is six, but the probability distributions are much different. For team A, nothing between 10-2 and 2-10 would really be outside the realm of reasonable likelihood, while for team B, 8-4 or 4-8 would be extremely surprising. So in that case a 9-3 record for team A would actually be considered a less extreme result than an 8-4 record for team B. Similar types of situations occur in the below table, though not to the extreme example as just described.

I've spent a bit of time looking through the teams who finished especially well or poorly and haven't found any particularly meaningful tendencies on either end. If anyone has any good ideas I can dig into them a bit, but for now at least I haven't seen anything obvious that I'm missing, other than "replacing a top ten coach with a guy who has zero coaching experience" variable, and I'm guessing that situation isn't going to come up often enough to be useful going forward.

One other thing that I'm going to be spending some time trying to do is to tighten the win probability distribution bands. Right now the model is accurately reflecting what it sees as "noise", but at least some of that is data that it either hasn't seen or doesn't understand. My hope is that next season I can get a similarly accurate depiction of the limits of its knowledge, but with bands that are at least a bit less wide than I had to use in 2011.

There are a few important notes and caveats I need to make about this model:

1) Compu-Picks does not endorse implicitly or explicitly any form of illegal gambling. Compu-Picks is intended to be used for entertainment purposes only.

2) No guarantee or warranty is offered or implied by Compu-Picks for any information provided and/or predictions made.

2011 Compu-Picks Blog

Questions, comments or suggestions? Email me at cfn_ms@hotmail.com

Follow cfn_ms on Twitter