/cdn.vox-cdn.com/uploads/chorus_image/image/23858009/gyi0062627221.0.jpg)
Some of you may recall RedmondLonghorn's fascinating two-part series on Football Study Hall before the season:
Apparent Talent, Team Quality and Coaching Effects
Apparent Talent, Team Quality & Coaching Effects: Part Deux
Since we're now on the cusp of a new coaching search, and since Big(g)Ern started the discussion in earnest here, I thought I would re-introduce RL's math to the party, with my own twist.
This will be a two-part series. Today I'll dig into RL's numbers a bit, give them a few basic tweaks, and present some interesting observations from the resulting dataset. In Part II I'll update the data to include this year's stats-to-date and present the complete results with some additional analysis.
The Source
For those of you too lazy to click the links above, RL has put together a quietly amazing dataset containing coach names, 4-year recruiting star averages for each team compiled from both Scout and Rivals, Football Outsider's F/+ and S&P+, and Sagarin Predictor scores for each individual season at every school in a BCS conference from 2006 to 2012. That's 466 observations (66 schools x 7 years, plus 2 for Utah, 1 for TCU and 1 for Temple) times 6 categories of data = 2796 datapoints.
(That's a lot.)
Then RL plays with the data to suss out coaching effects. Basically, he's trying to provide evidence for the general notion that by accounting for player talent separately, good coaching can be identified and its impact measured with statistics.
But I'm not interested in general notions. I want to know which specific coaches might be an adequate replacement for Mack Brown. Fortunately RL has blessed us with the vast majority of data entry needed to get such an analysis off the ground, in a format easily cut-n-pasteable into Excel. If you're out there, RL: holy crap, man. Thank you.
If you don't want to understand how it all works in detail, that's fine. Might want to skip this section in blockquote:
The Problem
Conceptually, success in any team sport (hence "quality") can be explained as the intersection of two factors: the ability of the players ("talent") and the value added by (or subtracted from) their environment, including stuff like strategy, game technique, strength and agility training, cooperative team relationships, etc. We'll call that "coaching", since the coach is accountable for most of those factors.
The fundamental relationship between talent and coaching appears to be best described as multiplicative, not additive. Meaning, the best teams almost invariably have both. You can't win FBS championships with South Dakota State talent nor can you win championships with a drunk idiot mule at the helm, Barry Switzer notwithstanding.
So far we have this basic equation: C(oaching) x T(alent) = Q(uality).
RichmondLonghorn has given us good proxy data for T (Rivals, Scout data) and Q (F/+, S&P+, Sagarin). What I'd like to do is plug that data into the equation and solve for C.
Problem is, we can't do it THAT simply, because our proxy for Talent is denominated in "stars" while our proxies for Quality are denominated in...uh..."points" I guess.
We can still solve for Coaching but it will take a bit of simple calculus. First, we run a regression of Talent on Quality across the entire dataset and produce an equation that can be used to predict Quality based solely on Talent score:
(Wonk note: I went with an exponential regression analysis because I want the line to be slightly curved upward, i.e., to set the bar disproportionately higher for coaches who already have good recruits. Call it an Un-Kiffening of the data. The R-squared isn't quite as good of a fit as a linear approach but it's still fine. Also: this analysis is performed after the data adjustments noted below.)
Then for each school-year, we can compare predicted Quality scores to actual Quality scores, and the difference for each is our Coaching effect for that school-year.
However, that Coaching effect isn't exactly what I'm looking for. C will tell me that Bill Snyder was a fantastic coach in 2012. But I know he's a fantastic coach because he runs a coaching system that perfectly suits a school that doesn't have any of Texas' major advantages.
We don't want a coach who can deliver decent Quality from low Talent. Instead we want someone whose high Coaching skills can credibly be applied to high Talent and thus produce elite, championship-contending team Quality.
So ultimately we want to give each coach in the database a score that is based on our measured Coaching impact but we need to weight that impact by team Talent. Coaches with good talent who succeed should get extra credit for proving they can do the job, and coaches with great talent who suddenly find themselves getting mudholed by average teams should get penalized more.
The simplest way to accomplish this weighting is (C times T), so that will be our raw score (hence "CxT"). To make the score easy to understand we'll convert it into an index (hence "CxT Index") where 100 is the top ranked CxT performance (Saban, Alabama 2011) and 1 is the lowest ranked CxT performance (Embree, Colorado 2012).
The Data
Before starting, I needed to make two major adjustments to RL's dataset.
First, after running a few tests on the data I noticed that RL's proxy for Talent - the four-year rolling average of Rivals/Scout stars - has actually been inflating over the last few years. The reason is simple: Scout and Rivals have been scouting more kids every year. That means they're handing out more 3, 4, and 5 stars rankings every year, which means there's a lot of kids out there getting three stars today who would've only gotten two in 2006.
This is problematic because in recent years talent-poor teams have seen almost uniformly rising talent scores, peaking circa 2008. Without adjusting the data, these leaps and bounds would soil the analysis, scattering average coaches at crappy schools across the top half of the results, all because scouting services finally got around to grading their crappy recruits. Here are the mean star averages for BCS conference teams from 2006-2012:
Fortunately most of the grade inflation is on the bottom end of the Talent scores and the effect appears to be very linear. So I used a formula to subtract that from the raw Talent scores. Hard to explain but it's a simple fix, and without this adjustment you get Jeff Jagodzinski and Steve Kragthorpe in your overall Top 20, so it's worthwhile.
Second, I didn't want to include S&P+ in my Quality Composite because S&P+ is already part of F/+. Plus, the S&P+ index is skewed and pear-shaped, while both Sagarin Predictor and F/+ are much more normally distributed. The resulting Quality composite is a damn fine dataset, as these things go.
Here's the descriptive statistics:
The Coaching results also come in very normal-ish. All of the indexes are somewhat fat-tailed. The Talent scale is also somewhat fat-bottomed and therefore so is the CxT Index, which is just C times T and indexed to 100.
The Results
Some tips when looking at the following numbers:
For both Quality and Coaching, stats the middle third of the range (33-67) will hold about two-third of the scores. 70 is a Top 10 score in a typical year, 80+ is a Top 5 score, and 90 is championship-level performance.
For the Talent rank, a 3.00 will typically put you on the cusp of the top 25 in Talent score in an average year, and a 3.5 puts you around the top 10. The highest Talent score in a year tends to be very close to 4.
For the CxT Index, scores over 50 are respectable. Scores over 60 typically produce Top Ten results, 70+ is a Top 5 score, and 85 is championship-level performance.
You'll notice that some coach names have an exclamation point in front of the name (e.g., "!Saban"). This is my method of marking seasons in which the coach on his "honeymoon" - i.e., in his first or second year, and thus the majority players on the team were not recruited by him.
This will allow me to distinguish "system coaches" - that is, coaches who need certain kinds of players to succeed and/or time to implement their system - and "fundamentals coaches" who can quickly coach up a sloppy roster someone else cobbled together. "System coaches" who recruit downward to fill particular needs and "fundamentals coaches" with no system are not well-positioned to take advantage of Texas' strengths.
Incidentally, the arrows signify the quartile in which a result resides. From top to bottom: green, yellow, grey, then red.
Most Talented
Note the sheer domination by USC, Florida, Ohio State, Texas and Alabama. Also note that only three of these 25 teams won national championships - and not coincidentally, those three teams are the only ones with Coaching scores above 80. It's not hard to see why some people think the biggest schools in recruiting hotbeds tend to see their recruits' grades inflated.
In any case, the notion that elite recruiting can compensate for bad coaching should be put to bed with benzo. When it comes to championships, elite recruiting doesn't even compensate for average coaching. Champions in college football almost all come from a small handful of schools that can instantly summon elite recruits if the coach has a reputation for winning. Texas is one of those schools. We should expect the next Texas HC to continue to put numbers up on this list but they won't come with championships unless he knows what to do with his blue chip recruits.
Least Talented
Note that despite a number of solid coaching performances, the only good CxT score out of this bunch is Bill Snyder's 2012 campaign. That's what you get for going to the Fiesta Bowl with one of the least-heralded rosters of recruits in America: recognition that you might be such an incredible coach that you merit consideration by CxT's elitist standards even though you only recruit unknown JUCOs and zero-profile high schoolers. A tip of the hat, sir.
Highest Quality
These are the schools that punched the hardest according to Sagarin and F/+. The C x T = Q relationship really stands out here; every yellow C score is offset by a high T score and vice versa.
Lowest Quality
These are the least fearsome teams over the last seven years of football per Sagarin and F/+. Note that every last one of them features a bottom-quartile coaching performance.
Best Coaching
Here are the best coaching jobs according to the numbers. Note that every time one of these coaching performances meets with top-quartile talent, the team contends for a championship. Also note the high score for Chizik in 2010, who otherwise has never had a good coaching score. This is probably because "Coaching" is in fact code for "environmental effects", one of which is "having a transcendental QB who impacts the game far more than his star ranking would suggest". This is why people should take Sumlin's CxT score (and Mack Brown's scores from 2008-9) with a grain of salt.
Worst Coaching
Why hello, Mack Brown 2010. Nice to see you've gotten cozy with the likes of RichRod 2008, Chizik 2012, Willingham 2008, Nutt 2011, Weis 2007 and abundant vintages of Neuheisel and Gill.
Best CxT Scores
This should be our first major hint that finding the next Nick Saban may not be THAT easy. There are so many usual suspects on this list I should probably change my handle to Verbal Kint. Once you've performed the discount for Heismanesque QB's - Newton, Manziel, Russell, Colt, Bradford, Luck - the only remaining name that registers as even a mild upset is Gundy 2011. Perhaps the best lesson here is how many of these top scores are accompanied by sublime QB play.
Worst CxT Scores
Hello again Mack Brown 2010. Not listed but close: Pete Carroll 2009 and Urban Meyer 2010. Mack isn't the only elite recruiter who drove the van plumb into the ditch.
Cumulative CxT Averages
Here are the top cumulative CxT scores for coaches active in 2012. Of the potential Texas coaching candidates, well, it's Nick Saban and everyone else. Shaw and Gundy look great by this metric though.
Best "Honeymoon" Performers
The green labels are for coaches still on their "honeymoon" in 2012. Saban makes a solid appearance despite the fact that he's a system coach. Shaw is off to an excellent start and appears to be in great shape to have similar career arc to Chip Kelly. Meanwhile one starts to wonder whether Brady Hoke is another Houston Nutt: a guy who can come in and make an immediate coaching impact on a sloppy team, but ultimately disappoint.
Best Established Performers
These numbers only include seasons for which the coach has been at the same school for 3 or more years. Green labels are for coaches active in 2012.
Jimbo Fisher looks like he'll be elite. One wonders whether Petrino or Tressel will ever coach FBS football again, but numbers don't lie: they had effective systems. Gundy and Bielema still looking good, while Pinkel and Dantonio are in Spurrier territory: above average but not setting the world on fire.
Team Analysis
I've posted these stats mostly to give y'all a good feel for the dataset. However, numbers DO sometimes lie, particularly when compiled into Best and Worst lists. The superior way to use this information is to pair it up with your subjective knowledge, to examine the evolution of coaching performance, to observe how numbers fit with one's prior knowledge, to recognize that some numbers can be explained away and others can't. The best way to do that is to look at performance by school or by coach. See, for example:
Here's Brian Kelly. What I see through these numbers is a fantastic system coach who recruits down to find people who fit in his system. His coaching value-added is through the roof once he has his players. Problem is, "his players" don't seem to be elite athletes. Despite several years of success at Cincy he didn't visibly improve the talent base, and through three years at Notre Dame the school recruiting rankings actually went down, despite the fact that Kelly is a bona fide college coach and Charlie Weis isn't. Could Brian Kelly do well at Texas? Certainly. But he's better-suited to coach at a school that has a major recruiting constraints. He'd rather have a three-star kid who fits his system than a blue-chipper who doesn't.
Conversely, here's Alabama:
THIS is what Texas needs: a system guy who can write his own legends with elite talent. This might work as well:
Here are two system guys who recruit ambitiously and produce elite results with average talent.
Here's what we want to avoid:
RichRod is a system coach too, but at Michigan he didn't recruit at all nor could he coach up the Carr leftovers. The bird barely got off the ground before it dove into the runway. Also we should avoid this:
Whoever thought Randy Shannon would be a good fit at Miami should've consulted with Luther. I'm just sayin'. Awful recruiting and a horrible, no good "honeymoon" that sent the program into a tailspin from which it has not yet recovered. I'll give him this much: as a fire extinguisher for an out-of-control program, he was effective. Too effective IMO.
And for reference, here's Texas:
Two great years featuring elite talent and solid-not-great coaching...and five (about to be six) subpar coaching seasons, including three consecutive bottom-quartile coaching seasons and one of the most epic wipeout seasons in the entire dataset. However, the recruiting reputation of Texas players is still insanely high - 2nd in the nation overall in 2012 - and team quality is not considerably lower than it was in 2006-7. A skilled coach with a sound plan should be able to walk right in and make hay....as long as he brings a QB with him.
What's Next
I'll post a few other schools/coaches in comments. I'll be happy to post requests as well, if there's a particular coach or school you want to see. I'm not going to post the whole dataset yet; I'm not finished with it. I want to update the data for 2013 and possibly weight the recruiting data for class size, but both of those will require some extensive manual data entry. Once it's done I'll post an update, with a copy of the full unlocked spreadsheet for whoever wants it.
