Bruce Allen over at BSMW ran a long piece today about PFF and their grading system. It's an interesting read and touches on quite a few points we've brought up here. It's well worth checking out the entire thing, then coming back here.
Some quotes:
Allen then touches on his key question: how can they grade each player on the field if they don't know what his assignement was or where coverage/protection/route running broke down?
Another problem is that the NFL just recently added the coaches film to Game Rewind, so before that, the PFF graders could not even see the entire field. I don’t know if they currently even utilize the overhead game film, or just rely on the standard HD game telecasts.
There HAS to be a subjective element in the grading process. They have to be making conclusions based on conjecture and assumption or what they “think” the player was attempting to do or was assigned to do on any given play.
It's long so I'm not going to quote the whole thing. Allen raises some very worthwhile questions (qualifications of the reviewers, surety of their conclusions, etc). Go check it out, I'd like to hear people's thoughts on this, because it's come up here before.
Some quotes:
It is important to note what Pro Football Focus is. Actually, first we’ll define what they are NOT. They are not taking raw numbers and data and crunching them into new and exotic formulas to provide a different sort of insight into player performance. This is not sabermetrics for football.
No, their methods are different. They are a UK-based company, who obtain games through NFL Rewind and sit and watch and grade each player on each play. Their dedication to this is admirable, as I can’t imagine sitting down and doing this kind of deep grading for every play, every game week after week.
Allen then touches on his key question: how can they grade each player on the field if they don't know what his assignement was or where coverage/protection/route running broke down?
Bill Belichick doesn’t have it figured out. But Pro Football Focus does? They can provide a grade on every play?
So sometimes even the team itself doesn’t know exactly where things broke down and who did what wrong. Belichick then went on to talk about watching opposing team’s game films and the impossibilities of knowing what happened:
But believe me, I’ve watched plenty of preseason games this time of year and you’re looking at all the other teams in the league and you try to evaluate players and you’re watching the teams that we’re going to play early in the season and there are plenty of plays where I have no idea what went wrong. Something’s wrong but I don’t…these two guys made a mistake but I don’t know which guy it was or if it was both of them. You just don’t know that. I don’t know how you can know that unless you’re really part of the team and know exactly what was supposed to happen on that play. I know there are a lot of experts out there that have it all figured out but I definitely don’t. This time of year, sometimes it’s hard to figure that out, exactly what they’re trying to do. When somebody makes a mistake, whose mistake is it?
Another problem is that the NFL just recently added the coaches film to Game Rewind, so before that, the PFF graders could not even see the entire field. I don’t know if they currently even utilize the overhead game film, or just rely on the standard HD game telecasts.
There HAS to be a subjective element in the grading process. They have to be making conclusions based on conjecture and assumption or what they “think” the player was attempting to do or was assigned to do on any given play.
Again, this is not taking actual numbers and using them to come up with new stats to use in analytics. This is not taking passes complete and passes attempted and breaking it down into the various lengths of throws and spots on the field. This is sitting down in front of the monitor, forming an opinion and making up their own stats based on what they think is happening on each play.
It's long so I'm not going to quote the whole thing. Allen raises some very worthwhile questions (qualifications of the reviewers, surety of their conclusions, etc). Go check it out, I'd like to hear people's thoughts on this, because it's come up here before.