NERD FIGHT
As I think everyone knows by now, I adore Football Outsiders and wish to have Mike Tanier’s babies. And earlier this year, I took exception to an article on Sports Illustrated by Kerry Byrne of Cold Hard Football Facts because he’s an idiot. Now, they fight!
On Tuesday, Kerry Byrne posted this article about how great Tim Tebow was and how great it was that they were the only ones who knew why – that it was because Tim Tebow was great.
There’s no doubt that Tebow’s passing accuracy has been spotty at times. At the end of the day, though, he has consistently outplayed the other team’s quarterbacks. The problem is that most analysts are limited in their ability to analyze and compare quarterbacks with anything more concrete than the old eye test. Or they look at stats that simply do not matter at the end of the day, such as passing yards, and can’t figure out how Tebow is winning games.
Smarter analysts might know to look at critical measures of passing success, such as yards per attempt or passer rating – indicators that traditionally have a very high correlation to victory. But even those indicators fail to tell the whole story of Tim Tebow.
Enter Cold, Hard Football Facts.com’s Real Quarterback Rating, which we introduced over the summer and which has quickly proven itself the most important indicator in football outside of final score.
CHFF Real Quarterback Rating measures all aspects of quarterback play, passing, rushing, sacks, fumbles, etc., and spits it out in a number substantially similar to passer rating and that uses the same formula as passer rating. (Passer rating, while extraordinarily useful in its own right, measures only passing and nothing else – even if many fans and analysts erroneously refer to it as “quarterback rating.”)
In other words, Tebow is no statistical circus freak winning in spite of himself. Tebow’s Broncos are winning because he consistently outperforms the opposing quarterback when you take into account all aspects of production: passing, running, sacks, total touchdowns, interceptions and fumbles. In fact, he consistently outperforms them by a wide margin.
Denver is 5-0 when Tebow produces a higher Real QB Rating than the opposing quarterback and 0-1 when the other team has the advantage. And those results are no coincidence.
After all, it turns out that no stat in football outside final score – indeed, maybe no stat in North American sports, period – is more important than Real QB Rating this season when it comes time to separate winners and losers.
Emphasis added by me.
Mike Tanier’s Walkthrough column on Football Outsiders included a breakdown of the “Super Duper Quarterback Rating.”
Now, let’s look at the Super Duper Stat. I like the idea of calling sacks passing attempts and sack yardage lost passing yardage. I can live with the idea of adding fumbles to interceptions, assuming that someone took the time to remove aborted snaps and other events that have little to do with a quarterback’s true performance. And combining rushing stats with passing stats is laudable in theory.
But there is a huge problem, which I bet has readers are screaming at their monitors right now: The Super Duper Stat adds rushing attempts to pass attempts. And it adds rushing attempts to pass completions. So every time a quarterback runs, the system treats it like a completed pass.
And completion percentage is the one stat that the old efficiency rating grossly over-rewards.
I have tinkered with many failed statistical ideas over the years, and Football Outsiders has piloted, adjusted, fiddled with, and a few times thrown out whole models and methods. A statistical method can look good in the spreadsheet and spit out hundreds of encouraging results before revealing some terrible flaw. There is nothing wrong with being wrong, for having to do a headslap and an apology, or with pointing to some anomalous result and saying “this is odd, and it may be a sign of a bigger problem, but it’s the best result we can currently provide.”
But there is a major problem with taking a strange, anomalous result, ignoring all warning signs, and trumpeting it as some hidden truth in the name of making money on an article because it happens to say what you want it to say about a currently popular player.
The problems with this method are glaringly obvious to anyone who has ever worked with the old efficiency rating, and I find it impossible to believe that the designers didn’t see red flags when they were setting up their spreadsheets. If they calculated completion percentage in a separate cell, as most of us would, they saw completion rates shooting up from 50 to 77.5 and 45 to 60.7 (the Jets game). They must have noticed that these changes were extreme, that they affected one player more than others, and that they represented essentially zero value added to the quarterback’s performance (remember, the yards themselves are separate). They could have realized the obvious flaw in their method, or at least said something to the effect that “Tebow is blowing up our system.” Instead, they spun this critical conceptual error into “Only our stats can see how well Tebow is doing, but yours can’t.”
That was lazy, irresponsible, dishonest, and frankly embarrassing to my entire profession.
The best thing I can say about this Super Duper rating is that whoever designed it could not be any more backward in their thinking. They took an ancient statistic and tried to put fins on it. We have been using play-by-play for eight years, and we now have a database going back another decade before that. The Total QBR guys are working with play-by-play and charting. There are other sites using variations on charting. I don’t agree with a lot of their results, and I compete head-to-head with them for cash money at times, but I have to respect that their digging may yield different, perhaps more interesting information than our digging on occasion. Data like third-down splits is not hard to find if you want to create your own Triple Deluxe Efficiency Rating by awarding a third-down bonus or something.
The future lies in more granular data, whether it is separating out yards after catch, differentiating passes by type, or going to the game tape with a compass and protractor and figuring accuracy down to inches (a bad idea, but an example of what’s possible). Grabbing stats off the back of a football card and twiddling with them is not amateurish, because amateurism implies effort and love. It’s just weak and juvenile. And I am guessing this Super Duper Rating will disappear once it starts spitting out results its designers don’t like or cannot profit from.
I love stats, I love statistical analysis, and I love it when people get righteous. And my team is 4-7 so shut up this is all I have this year.
