The BCS System: Rate Not, Lest Ye Be Rated

Post-dicting, indeed…..

The plethora of numbers and graphs required for you to make your point here is evidence enough as to the lunacy of the BCS.

Heh, yeah I included all the numbers to be thorough, but the interesting part is that all the BCS computers behave quite similarly.

To sum up: the BCS computer rankings:
Successfully predict ~70% of the next week’s games… but barely over 50% of games where the spread is a touchdown or less
-“Post-dict” (or “retrodict”) ~84% of the entire season-meaning ~15% of games “don’t fit the pattern” of the full season
-Also, the season as a whole is quite predictable; after just a week, Sagarin’s rankings can predict 67% of the remaining games. Unsurprising given the large number of on-paper mismatches, but I never would have guessed such a high figure.

So the cliffnotes numbers are 70%, 52%, 84%, 67%.

Imagine a team plays a Top 10 schedule, beats 7 teams with winning records and wins its conference. Then that same team is 3rd in the BCS and misses the MNC game to a team with a 40ish ranked SOS, only beats 3 winning teams and fails to even win its division. Now, imagine that team is Alabama.

I think the interesting number for me is the ~15% retrodict (love it!) outlier. That, to me, lends support to the idea that comparing losses should not be the metric for an AP voter to use when deciding whether to put Alabama over Okie State.

Radical.

O wait. You said “imagine.”

I’m trying. I can’t. Truth IS apparently stranger than fiction.

Ratings won’t be appreciably larger for this BCSCG than they were for the first LSU-Bama game. I am willing to bet they will be lower.

That will push us down the road toward plus-one.

The thing I like about the computers is that they don’t tell us one year that a team that didn’t win its conference shouldn’t play for the MNC, and then do a 180 3 years later. They recognize that a home loss to ole miss is worse than a road loss to tech, and can actually remember it a few games later.

Vegas isn’t picking games and does not measure itself by the outcomes you are using. They are strictly market makers and their only objective is to have a number that encourages, as close as they can possibly get, the betting public to bet half the action on either side. They measure by total “handle”, then do what tinkering they can (adjusting lines, laying off, etc) to balance the action.

If they do this, they know statistically what they can expect to earn as a percentage of the handle, less bad debts.

If you want to say they are predicting something, then it is the number that splits the action and not the outcome of the game. If there is any credit to be given to how close the betting lines are to the final score of the contest it is attributable to the wisdom of crowds more than anything else.

I actually disagree with that thought process. Vegas is that stock market for sports. Its final line is where buyer and sellers are equal, just like a stock. All the information is in those lines. Both buyers and sellers are acting in their best interest, vegas itself does not predict but its “investors” do

Re: Vegas

It’s a good point that “Vegas” itself doesn’t pick the lines but I think it should be a pretty efficient market. I don’t know what the breakdown is in sports betting between recreational and serious bettors, but I imagine it’s the more expert analyses (through combinations of algorithms and human assessment) that are driving the point spreads, otherwise someone would be cleaning up.

Basically the net out is that the point spread should be ~the median of the expert bettors’ predictions.

Awesome Zoolander references! I love that movie! (I feel like I’m taking crazy pills!). anyway, I can’t help but think that we would never have gotten computer ratings included in the original system in’98 if the AP had voted for Nebraska in ‘97 instead of insisting that if “number 1” doesn’t lose, then they can’t move down.

“Heh, yeah I included all the numbers to be thorough, but the interesting part is that all the BCS computers behave quite similarly.”

Not throwing stones here. Your analysis is peach. Just the monstrous amount of calculations required to justify keeping a crony bowl system is laughable.

Speaking of predictions, who would win between OKSt and Alabama in a neutral field? what percent of the time? Yeah, it’s irrelevant, but I want to know, but I’m not smart enough to do the math.

I think the Olympics should start using the BCS for the medal rounds.

Re: A Bama-OSU prediction.

Sagarin’s Predictor has Bama at 2 and OSU at 3, as does Massey’s margin-of-victory inclusive algorithm. I’m not sure how sophisticated either of these are in terms of analyzing relevant offensive and defensive stats versus just the final score margin. Massey’s, at least, appear to include some sort of sophisticated analysis of O & D.

BC’s own Huckleberry is probably the man to ask on how these more nuanced predictions are generated (i.e. those factoring yards-per-play and the like). Other sites like www.thepowerrank.com, which rates O & D separately, also has Bama ahead of OSU.

Unrelated, but on the Colley Matrix “football god” tool, switching the outcome of the OU-Baylor game results in OU jumping up from 6 to 3, comfortably ahead of Bama.

Now my head hurts. But not as badly as when I think about another Alabama/LSU game.

I think you kind of touch on the fundamental problem with the BCS, or even more generally, with any system which determines championships off the field. Computers are really good at measuring things, people less so, but at the end of the day someone has to define what exactly is being measured.

And that’s exactly what we do not have.

Are we supposed to be determining who the best/most talented team is? Or who has had the better year – and whether that should be determined by who’s got the best wins or who’s got the worst loss? Or who would win (or at least be favored) if they actually did play the games? Or some arcane combination of some or all of the above?

There truly is no established criteria, and so you have computer programs designed according to someone’s notion of what we should be seeking to measure, along with actual human voters who establish their own criteria (and often change that criteria on a year-by-year, or even week-by-week basis) and rank accordingly. And with the human vote, that’s not even taking account whether they’ve even input enough data to accurately determine whether their perceptions of which teams “win” based on their criteria are remotely accurate…

In the end, that’s the main reason a playoff is so desirable. If you set the field large enough to actually include all the teams with legitimate claims to a championship, it all comes down to actual results. There is no “best” team, there’s just a champion.

Tom, the link where you have the computer picks for the bowls, are they versus the spread or straight up? I could see the Vegas lines and the Sagarin predictor had the lines in them, but wasn’t so sure about the other choices. Nice writeup.

“(*Relevant evidence that there is some diversity among the computers: median gap between high and low rank for teams is 12—Tennessee and Texas Tech both vary by 32 positions!)”

While this does prove diversity among the computer formulas, it also emphasizes that we force stack rank a large chunk of teams that are largely equal in quality. Are Tennessee and Texas Tech actually 32 teams different? They’re likely nearly the same with the other 30 teams in that span. Rank gets forced from statistically insignificant numbers.

For the bowl “picks” I just used the higher ranking to select the teams, straight up. Only Sagarin’s Predictor has an easily extractable point margin for comparing teams, so I included that as well in its column.

Re: the variance of teams in the middle of the rankings, yeah I imagine that is strongly dependent upon how strong the computer weighs a team’s conference, and especially with Texas Tech, how it evaluates flukish but impressive wins. I’ve often thought about making plots of normalized ratings to see roughly where “the middle” begins—also a good way to compare the dominance of the top teams across different seasons.

It seems important to note that the only system included that is intended as an effort to predict future matchups is Sagarin’s Predictor rating. Every other system is either entirely “accomplishment-based” (e.g.,Wolfe and Sagarin ELO-Chess), or includes a Bayesian correction reflecting accomplishments to date (e.g., Massey MOV and Sagarin overall rating). Accomplishment in this case merely meaning the W/L results of each team against the schedule faced.

Therefore, judging the ratings sets based on a pre-concieved notion that they should be able to “assess the ability of teams” is possibly invalid because the systems are not designed to assess the ability of teams, but rather to rank the teams based on their accomplishments.

Because I’m interested, I may run this study using my system for this season later today.

I used the Sagarin Predictor for most of my bowl picks, so I’ll see how it ends up performing in my office pool.

Is there going to be a stat-off between Huck and the newcomer, Tom?

Re: Huckleberry, yeah, that’s why I was surprised how the BCS computers actually a decent prediction job, despite not being designed to do so. Do you know if most prediction-geared algorithms (like your own) give a single rating output wherein any prediction is determined by looking at this singe rating (e.g. as in the Sagarin ratings) or do they generate predictions for individual matchups using offensive & defensive metrics?

It’s interesting how much more hamstrung the computers are in evaluating teams relative to voters. Voters can consider who they think would win in a matchup whereas computers can only measure past wins & losses (without margin of victory, a prediction element is essentially impossible—although as I show, interpreting them as predictions works out okay) and are further limited in their post-game/body of work analysis by not looking at the score. For this reason I really think the BCS should hire some stats guys to design an array of computer ratings that weight all these things to different extent. Would be highly transparent, and weightings could be agreed upon beforehand. Would take out biases created by hype and bowing to Lord Saban’s every demand.

I agree on your end goal but I also understand it would never happen.

My power rating is a predictive rating when combined with standard homefield advantage. I also, of course, have stat-based predictive ratings. My most fervent agreement, though, is that the models used by the BCS should be transparent. It would be fairly easy for the BCS to get a group of guys, obviously including me, to publish a set of transparent ratings that all hit on the various options in publishing a computer rating set under the no-MOV constraint. I addressed that a little in this post from 2008.

“ut-06 said: December 16th, 2011 at 8:55 am

Is there going to be a stat-off between Huck and the newcomer, Tom?"

Possibly a more exciting matchup than the Sugar Bowl rematch, IMO.

TheBobs is correct: the fundamental problem with the football polling construct is that there’s no agreement on what is being measured. At least in the basketball tournament selection process, the selection committee says, ex ante, what factors are good (say, wins on road or at neutral sites), bad (low SOS), and not material (say, not winning conference). On the one hand, I find it ridiculous when football poll voters explain they’ve raised their ranking of Team X because player Y has become healthy again; on the other hand, who am I to say that the absolute quality of the roster, rather than on-field-results, shouldn’t be considered?

I will make the extraordinarily iconoclastic, never before heard in this forum point that the BCS regime, whether using human polls or computers, is fatally flawed. As long as you have some small, but greater-than-two, number of teams that have comparable credentials and haven’t played each other head-to-head, you will end up with credible teams shut out of the opportunity to play for a title. Move to a “+1” format and make the discussion a less high stakes one about “who’s #5 and on the outside looking in” rather than “who’s #3 or #4?” Move to an 8-team playoff and the controversy diminishes further. Move to 16-teams and #17 might whine, but you won’t have the whole system called into question. You’ll also have bowl games that have real consequence.

No real championship contestants should ever be chosen by voters or computer rankings. Only the outcome of real games within the confines of a real play off system, where no participant is awarded a spot in the play offs through any type of ranking or vote, should determine the champion. Until that happens, all Div 1 champions in college football are mythical and unearned.