Analysing the Analysists: the ProFootballFocus Discussion Thread

[crystalline]

 
Tom Brady in 2010 had a 36/4 TD/INT ratio, was the first unanimous NFL MVP, and was ranked by PFF as the 20th best player in the league that season. It's nice that they set out to try and develop a relatively intuitive ratings system, but without proper knowledge of play calls and assignments it's pretty hard to make the kind of objective evaluations they seem to strive for. There are some other things that bother me, for instance, the way things like WR blocking and CB run defending are too heavily weighed, to the point where a CB can have a pretty high overall rating while being a subpar pass defender.

Yeah. PFF's scoring each play is cool, but their weakness is how they combine the scores- by merely adding them together. For example a CB might absolutely blanket his man every single play, except for 3 plays where he gives up three TDs and 21 pts. It's clear that is a bad CB performance: much worse than a guy who gives up a lot of 5 yd catches but makes the tackle every time. PFF would score them the opposite way.

Same for OL: a tackle might throw his opposing DL on the ground every single play but three, on which he gives up 3 bad strip sacks. That's a bad OL performance. He'd end up positive on PFF.

Football outsiders has tried to adjust for this via nonlinear transforms such as success rates- without too much success.

Football's tough to assign number grades to. PFF is the best 'scouting' out there but you need to drill down beyond their aggregate game scores.

 

[nazz45]

Yeah. PFF's scoring each play is cool, but their weakness is how they combine the scores- by merely adding them together. For example a CB might absolutely blanket his man every single play, except for 3 plays where he gives up three TDs and 21 pts. It's clear that is a bad CB performance: much worse than a guy who gives up a lot of 5 yd catches but makes the tackle every time. PFF would score them the opposite way.

Same for OL: a tackle might throw his opposing DL on the ground every single play but three, on which he gives up 3 bad strip sacks. That's a bad OL performance. He'd end up positive on PFF.

Football outsiders has tried to adjust for this via nonlinear transforms such as success rates- without too much success.

Football's tough to assign number grades to. PFF is the best 'scouting' out there but you need to drill down beyond their aggregate game scores.

I generally believe their offensive and defensive line grades pass the smell test. The linebackers to a lesser extent. I take the rest for what it is... there are just too many factors to account for on a given play (even including the line play for that matter).
 
For example, what if the defense is in zone and the linebacker underneath is supposed to limit the cushion by re-routing the inside receiver on a seam route towards the safety/deep defender on the back end, only the linebacker fails to do this for whatever reason (out of position, poor technique, etc.). The safety/deep defender is now "late" to his zone and cannot defend the play. Who gets the negative grade? Maybe they do account for stuff like this, but it's difficult to do either way because we don't always know the inner workings of a certain defensive call.

 

[MentalDisabldLst]

Every week I see these ratings and at least one of them so thoroughly fails the smell test that it's hard to take any of it seriously. This week it's Edelman. Of the three targets he didn't pull in, one was an errant pass and one was a DPI in the endzone (I don't recall the 3rd off-hand). He had, at worst, a good game.

 
As Bill James says, with any good statistic, 80% of its output will confirm what you already believe, and 20% will - and should - surprise you.
 
That doesn't mean that the 20% is necessarily correct and you're wrong about Edelman, just that a hallmark of good statistics is their ability to surprise your conventional wisdom.  Perhaps Edelman's contributions to other parts of the game were lacking, I can't say, but that "surprise" should be an impetus to look deeper, not to discard the statistic as being worthless.
 
edit: Crystalline's analysis in post #62 is a pretty solid move in that direction.

 

[Super Nomario]

As Bill James says, with any good statistic, 80% of its output will confirm what you already believe, and 20% will - and should - surprise you.

That doesn't mean that the 20% is necessarily correct and you're wrong about Edelman, just that a hallmark of good statistics is their ability to surprise your conventional wisdom.  Perhaps Edelman's contributions to other parts of the game were lacking, I can't say, but that "surprise" should be an impetus to look deeper, not to discard the statistic as being worthless.

I understand the abstract point you're trying to make, but the problem is that the PFF ratings are so much of a black box that it thwarts this kind of further analysis. There's no play-by-play log that shows which plays a guy gets a minus for and which a plus. They're semi-helpfully grouped into categories like "pass," but we don't know to what extent "pass" measures Edelman's ability to get open, his catching ability, or his run after catch ability. It would be great if the -1.2 was a jumping off point for further analysis, but PFF doesn't provide the tools to do that.

 

[StupendousMan]

Here's one way to evaluate rankings. 
 
First, assume that some players really are better than others. 
 
Second, given that assumption, we can expect that a "good" rating system will end up rating some players consistently higher than others.  If a rating system provides essentially random scores for players on a week-to-week or season-to-season basis, it isn't accurately reflecting the true talent difference between players (see item 1).
 
So, pick some stretch of time; perhaps one whole season, perhaps a stretch of six games.  Gather together the ratings for each player, and examine the variance of those values.   If the variance is small ("Good - Excellent - Good - Good - Good - Excellent"), then the rating system might be a good one.  If the variance is large ("Good - Bad - Excellent - Execrable - Good - Execrable"), then the rating system is probably not a good one.
 
Of course, any real player will have good days and bad days.   But most people would agree that a "good" player has considerably more good days than bad ones, and a "bad" player has considerably more bad days.
 
What's the variance in PFF player ratings?

 

[Shelterdog]

 
As Bill James says, with any good statistic, 80% of its output will confirm what you already believe, and 20% will - and should - surprise you.
 
That doesn't mean that the 20% is necessarily correct and you're wrong about Edelman, just that a hallmark of good statistics is their ability to surprise your conventional wisdom.  Perhaps Edelman's contributions to other parts of the game were lacking, I can't say, but that "surprise" should be an impetus to look deeper, not to discard the statistic as being worthless.
 
edit: Crystalline's analysis in post #62 is a pretty solid move in that direction.

 
Why 20% and not 13%  or 56% or 98%? This seems like a Gladwellian shortcut that sounds clever but actually has not legitimacy behind it.
 
Anyhow, besides being a black box PFF has another problem in that it doesn't adequately weight the importance of plays by position.  Janoris Jenkins shitty coverage last night is a lot more harmful than a DT who gets pancaked and gives up 7 yards as a result.  Evan Mathis may have been a better offensive lineman than anyone else in the league last year, but it's laughable to think that an interior lineman would have more of an impact than every QB other than Manning.  

 

[Devizier]

Of all the major professional sports, football may have the fewest "independent events". Almost everything that a player does on the field depends on the performance of every other player. How do you parse these contributions effectively? This is what I would consider the major problem with the PFF ratings, although I applaud the effort.

 

[MentalDisabldLst]

Why 20% and not 13%  or 56% or 98%? This seems like a Gladwellian shortcut that sounds clever but actually has not legitimacy behind it.
 
Anyhow, besides being a black box PFF has another problem in that it doesn't adequately weight the importance of plays by position.  Janoris Jenkins shitty coverage last night is a lot more harmful than a DT who gets pancaked and gives up 7 yards as a result.  Evan Mathis may have been a better offensive lineman than anyone else in the league last year, but it's laughable to think that an interior lineman would have more of an impact than every QB other than Manning.  

 
It's a heuristic, not a scientific measurement, and it's by someone whose head is so full of numbers that he'll use them to make a qualitative point sound more accurate.  The important part is the principle: some minority of the time, a good statistic or piece of analysis should surprise you and upset conventional thinking.  How frequently that occurs, obviously, will vary based on the context.
 
I hear you all on the black-box nature of the scoring.  DVOA frustrates me similarly, but at least I can track that against similar measures.  What annoys me is, the DVOA differential in a game does not predict the victor of a game (much less the margin of victory) at a higher rate than you could get just comparing the teams' respective point differentials per game (PF - PA / games played).  Because of this, I can't do either of the two things that I would most like statistics to do for me:
 
1) Help amalgamate what we know about what's happened in the past into a way that lets me better understand the meaning of those past events (because it's a black box too, it purports to sum up a team's quality, but such an assertion can't be falsified)
2) Help me predict the future of who's going to win
 
I'd really like to know whether PFF's ratings can be useful in either of those two respects before I defend them further or toss them aside.  That leads me to next-level-of-detail questions such as:
 
- How well do the sum total of a team's players' ratings in a given game describe the overall team performance in that game?  Is there some correlation between Offensive or Defensive net PFF ratings, and points scored or allowed?
- Do players' ratings across a season or group of seasons predict their Free Agency contract value relative to their position peers?  i.e., within a position, does performance correlate to compensation?
- Do the totals of a team's players' performance ratings in a season's previous games predict the game's outcome better than Point Diff Per Game?
 
Those seem like questions answerable with the data we've got today, even with the generator of that data being a black box, no?

 

[StupendousMan]

I'd really like to know whether PFF's ratings can be useful in either of those two respects before I defend them further or toss them aside.  That leads me to next-level-of-detail questions such as:
 
- How well do the sum total of a team's players' ratings in a given game describe the overall team performance in that game?  Is there some correlation between Offensive or Defensive net PFF ratings, and points scored or allowed?

 
Tell us how we can access PFF's ratings, and we might be able to help you.

 

[Super Nomario]

In the PFFLOL department:
 
Rob Ninkovich was initially graded as a -2.1 for last week. At present, he is graded a +1.4. It's not uncommon to see grades shift by half a point or a point, but 3.5 is a ton.
 
Also the Patriots offense graded mildly negative against Buffalo and very negative last night.

 

[rodderick]

Of all the major professional sports, football may have the fewest "independent events". Almost everything that a player does on the field depends on the performance of every other player. How do you parse these contributions effectively? This is what I would consider the major problem with the PFF ratings, although I applaud the effort.

 
This is a great point. It's like people looked at baseball and thought "hey, let's try something like this out in other sports" without realizing that baseball, by its nature, provides a very specific framework that makes statistical analysis not only possible, but easily applicable and even falsifiable to an extent. In football it's extremely hard to isolate player performance, but even when you attempt to do so (i.e., analyzing every snap a single player takes in the whole game), there are situations in which the context dictates  his role on a given play, or even if he has any active participation in it. If you're not fully aware of that context, you can't judge fairly. Simply looking at All-22 tape, while helpful, doesn't give them the necessary insight to evaluate how the player performed vs. how he was supposed to perform, in order to assign a concrete grade.
 
It's a noble effort, but I honestly can't even begin to comprehend how statistical analysis in football can try to rate individual performance without full access to playbooks and coaching tapes. Until then, it will bother me when people offer up these numbers as if they represent a significantly greater knowledge than a fan's viewpoint, or your average "breakdown" of the game that happens in TV shows and blogs everywhere. It's another source to instigate further thought, and to challenge some preconceptions, sure, but not much more than that to me.

 

[dynomite]

Also the Patriots offense graded mildly negative against Buffalo and very negative last night.

You made a good point elsewhere about whether something passes the "smell test." That doesn't.

The Patriots offense had issues last night, unquestionably. They also scored 27 points against one of the best defensive fronts in the NFL.

For the record, that's the 2nd most offensive points anyone has scored on the Jets this season, 3 more than the Broncos and only 4 fewer than the Chargers, who had the benefit of 2 turnovers (one of which gave them possession at the Jets' 20 yard line).

 

[Stitch01]

I think summing their numbers across positions as they do creates a nonsense number.
 
I didn't know they regraded later in the week, do they use the all-22 to do that at least?

 

[Super Nomario]

PFFLOLs: Pats O grades out at -7.1 last night, with Brady and Gronk the only players above 0.1.

 

[RIrooter09]

Incredible

 

[lexrageorge]

Seems like PFF should get the "Game Goat" award.  

 

[rodderick]

PFFLOLs: Pats O grades out at -7.1 last night, with Brady and Gronk the only players above 0.1.

 
Wait, Edelman graded out negatively on this game? PFF is the worst. 

 

[SeoulSoxFan]

PFFLOLs: Pats O grades out at -7.1 last night, with Brady and Gronk the only players above 0.1.

 
There were a few hurries but that really seems stingy to me.

 

[Ed Hillel]

 
Wait, Edelman graded out negatively on this game? PFF is the worst. 

 
Well, offensive and ST grades are separate (Edelman was 1.4 on ST), but still...
 
Another lol is that the Patriots are currently 50 points behind the Bears as a team on the season, and over 60 behind the Saints. They're also behind the Redskins...
 
...

 

[mascho]

PFFLOLs: Pats O grades out at -7.1 last night, with Brady and Gronk the only players above 0.1.

 
We'll see if/how they adjust their numbers when the All-22 film is available, but that seems patently absurd.

 

[williams_482]

I feel like adding some meaningful units to their numbers would be a huge help. For example, on every play they could take the number of expected points generated and divvied out credit for that set number of points based off of how each individual player was graded in their +/- 2.0 scale. At the very least that would give the numbers some "real world" meaning (because who really knows what the difference between a +1 and a +5 player actually is) and would more appropriately credit players for performing well in more important aspects of the game. 

 

[amarshal2]

 
We'll see if/how they adjust their numbers when the All-22 film is available, but that seems patently absurd.

 
It is patently absurd and yet you can totally see how they would come to that conclusion.  They way over penalize players for mistakes...many of those guys had some big drops.  Additionally, 21 points were off of turn overs or special teams.  But at the end of the day when a team puts up 40+ on one of the best defenses in the NFL everyone can't be negative.

 

[Smiling Joe Hesketh]

PFFLOLs: Pats O grades out at -7.1 last night, with Brady and Gronk the only players above 0.1.

It's patent bullshit.

How many times do I have to say it: THEY'RE JUST MAKING IT UP AS THEY GO ALONG.

Use DVOA from FO. Or almost anything else. But PFF is selling snake oil. They always have.

 

[Phragle]

PFFLOLs: Pats O grades out at -7.1 last night, with Brady and Gronk the only players above 0.1.

 
What is so wrong with their ratings? Connolly, Wendell, and Stork account for -8.4 by themselves, and I didn't think they played very well. Granted they had a lot to deal with, but PFF doesn't account for the opponent talent level. Overall it might be a little harsh, but this is why I suggested a 10 times multiplier for the QB in RFP last offseason. PFF doesn't weight anything.

 

[Jnai]

Anyone have a data dump from pfp? I'm curious what it actually correlates with on a team level.

 

[Super Nomario]

 
What is so wrong with their ratings? Connolly, Wendell, and Stork account for -8.4 by themselves, and I didn't think they played very well. Granted they had a lot to deal with, but PFF doesn't account for the opponent talent level. Overall it might be a little harsh, but this is why I suggested a 10 times multiplier for the QB in RFP last offseason. PFF doesn't weight anything.

Not weighting anything or opponent-adjusting is one thing, but do you really find it plausible that the Patriots hung 30+ points on Denver with only two players having above-average performances?

 

[bowiac]

Not weighting anything or opponent-adjusting is one thing, but do you really find it plausible that the Patriots hung 30+ points on Denver with only two players having above-average performances?

Kind of?
 
First, one of those two players (it was actually three - Vereen was +0.1), one was a Quarterback, which with the outsized performance of the position, will take you a long way. Second, the Patriots offense was not actually especially explosive. They ran the ball 25 times at 2.6 yards per carry (reflecting pretty poorly on o-line). Brady threw for a lot of yards, but also threw 53 times. He only got 6.3 yards per attempt. 
 
The Patriots had 13 offensive drives. Five ended in punts. One was an interception. They had two scoring drives of ten yards or less, one resulting in a field goal. And they had five substantial drives. That's about a 53% success rate (five punts + an eight yard drive which only netted a field goal). That's good obviously, but isn't some kind of "wow" performance. They were working with good field position consistently (three scoring drives starting in Denver territory).
 
Now some of this is unfair, and you can do a lot to minimize any good offensive performance. But compare that to the game vs. the Bears (an asskicking according to PFF). In that Bears game, Brady threw for 50% more yards per attempt (10.1 yards), ran 50% more yards per carry (3.8 yards), and scored on eight of nine real drives, with only one "short drive, settle for a field goal". That's a 78% success rate by the same standard.
 
I dunno. It's surprising, but I can kind of see it when I look at the numbers. The rushing average in particular is pretty damning to the O-line.

 

[crystalline]

Kind of?
 
First, one of those two players (it was actually three - Vereen was +0.1), one was a Quarterback, which with the outsized performance of the position, will take you a long way. Second, the Patriots offense was not actually especially explosive. They ran the ball 25 times at 2.6 yards per carry (reflecting pretty poorly on o-line). Brady threw for a lot of yards, but also threw 53 times. He only got 6.3 yards per attempt. 
 
The Patriots had 13 offensive drives. Five ended in punts. One was an interception. They had two scoring drives of ten yards or less, one resulting in a field goal. And they had five substantial drives. That's about a 53% success rate (five punts + an eight yard drive which only netted a field goal). That's good obviously, but isn't some kind of "wow" performance. They were working with good field position consistently (three scoring drives starting in Denver territory).
 
Now some of this is unfair, and you can do a lot to minimize any good offensive performance. But compare that to the game vs. the Bears (an asskicking according to PFF). In that Bears game, Brady threw for 50% more yards per attempt (10.1 yards), ran 50% more yards per carry (3.8 yards), and scored on eight of nine real drives, with only one "short drive, settle for a field goal". That's a 78% success rate by the same standard.
 
I dunno. It's surprising, but I can kind of see it when I look at the numbers. The rushing average in particular is pretty damning to the O-line.

When a team beats the consensus best team in football and does it by scoring a lot of points in a shootout, that means their offense is good. (Subtract the ST score and adjust for field position and you come to the same conclusion.)

When you argue that PFF's numbers are reasonable because the running YPA and Brady's YPA are low,
that means that you should probably evaluate whether YPA is a good measure of football success too. For example if the Pats have a very consistent offense low YPA might be beneficial. Or if the Broncos were giving up the short route to limit big plays. Or if you start using the bubble screen to replace running plays (and the Pats definitely did this).

The offense played well yesterday. I'd take that performance from the offense in any playoff game. I thought your post was interesting as it put the offense in context but wow, those numbers make me want to figure out why YPA doesn't describe what happened.

Edithrasing

 

[Phragle]

Not weighting anything or opponent-adjusting is one thing, but do you really find it plausible that the Patriots hung 30+ points on Denver with only two players having above-average performances?

 
I don't think much of single game ratings - especially the ones close to zero. Consider that Denver's defense is well above average and they were in the negative too. I think the very good Denver defense is more to blame for the -7.1 than the Patriots offense playing poorly.  Every offense Denver has faced posted a negative PFF rating FWIW.  Hypothetically, if Joe Thomas battled JJ Watt all game and Thomas ended up with a -0.5 PFF rating, would you say Thomas had a below average performance or that he was pretty good but had to deal with the best defensive player in the league?
 
I find it plausible that if a QB like Brady plays great, 30+ is pretty easy with an average surrounding offense. The only rating that surprised me is Edelman's, but I haven't studied every play like they have, it's a positive rating if you don't count his blocking, his drop is probably enough to make his rating negative, and the all-22 will improve the accuracy of his rating. It may make it worse but rating a WR without all-22 is flawed IMO.

 

[Super Nomario]

Now some of this is unfair, and you can do a lot to minimize any good offensive performance. But compare that to the game vs. the Bears (an asskicking according to PFF). In that Bears game, Brady threw for 50% more yards per attempt (10.1 yards), ran 50% more yards per carry (3.8 yards), and scored on eight of nine real drives, with only one "short drive, settle for a field goal". That's a 78% success rate by the same standard.
 
I dunno. It's surprising, but I can kind of see it when I look at the numbers. The rushing average in particular is pretty damning to the O-line.

The run blocking was bad, sure, but PFF graded all five OL as -0.9 or worse in pass blocking, too.
 
PFF graded the Jets' offense and their 10 points better as a team yesterday, at 0.6.

 

[bowiac]

When a team beats the consensus best team in football and does it by scoring a lot of points in a shootout, that means their offense is good. (Subtract the ST score and adjust for field position and you come to the same conclusion.)

When you argue that PFF's numbers are reasonable because the running YPA and Brady's YPA are low,
*that means that you should evaluate whether YPA is a good measure of football success* too.

The offense played well yesterday. I'd take that performance from the offense in any playoff game. If some stats look bad, that means the stats are wrong.

Okay, I mean, your attitude here is fair - that's your prerogative. On the other hand, it also basically means you're not especially interested in stats period (you disregard them when they disagree with you). Again, that's fair, but probably a sign that you're not really the target audience for something like this.
 
I will probably have post up on Football Central eventually about the virtue of yards per play. Suffice to say, if you were to me one "basic" stat about a team, I'd pick yards per play as the most meaningful. It has more predictive value than just about anything else (success rate may be close last I looked).

 

[bowiac]

The run blocking was bad, sure, but PFF graded all five OL as -0.9 or worse in pass blocking, too.
 
PFF graded the Jets' offense and their 10 points better as a team yesterday, at 0.6.

I want to preface this by emphasizing how little of a film junkie I am, but I didn't get the sense the o-line had a good day in pass protection either. Anecdotal, it seemed Brady was facing a lot of pressure, and either successfully avoided the pass rushers enough to throw the ball away, or get a pass off.
 
With respect to the Jets thing, that's mostly a function of what's been mentioned here before a bunch. You can't just add up the ratings - QB performance is obviously worth far more than other position. I don't know the correct weighting by position, but that's what's going on here. Brady had a great PFF game, and Vick was average. In real life, a great QB game and an average RT game is way better than a great RT game and an average QB game. When you simply add up PFF ratings, that is lost.
 
I didn't watch the Jets game, but it's not crazy to me to suggest that their offensive line had a better than the Patriots did. I didn't come away from the Pats game thinking "wow" with respect to the line.

 

[crystalline]

That was a little strident, apologies. Thanks for staying civil in response. I had edited my post.

I do think your post helped me understand the Pats O on Sun. The mismatch between YPA and my perception of their performance, though, makes me want to think about what YPA is missing. I'd start with looking at the use of bubble screens to replace the run.


Edit: on weighting PFF scores when summing- in baseball stats you'll often see weights so numbers give you runs or wins. Those are real units with obvious interpretation. In football I'd suggest win probability added, or expected change in points scored. I.e. at the opponents 30 on second down teams on average score 4.5 pts. After a sack given up the average team scores 3.5 pts, so the sack is worth -1.0 points. I don't think it would be impossible to estimate the expected points values.

 

[Super Nomario]

 I find it plausible that if a QB like Brady plays great, 30+ is pretty easy with an average surrounding offense.

Sure, but PFF didn't say Brady had an average surrounding offense; they said he had a terrible one. Apart from Brady the Pats were -10.4 offensively per PFF.
 

The only rating that surprised me is Edelman's, but I haven't studied every play like they have, it's a positive rating if you don't count his blocking, his drop is probably enough to make his rating negative, and the all-22 will improve the accuracy of his rating. It may make it worse but rating a WR without all-22 is flawed IMO.

I don't see what value PFF ratings have for players where we have statistics, to be honest. Most of the time they square up pretty well anyway, and in the cases where a receiver scores poorly by PFF and well by PBP-based metrics (like Edelman, who they currently grade out as the sixth-worst WR in football) or vice versa, I don't see any reason to think the PFF score is more valid than more conventional stats. And without more granular ratings, I can't even see whether it's providing any value at all even as a dissenting opinion.

 

[Super Nomario]

I want to preface this by emphasizing how little of a film junkie I am, but I didn't get the sense the o-line had a good day in pass protection either. Anecdotal, it seemed Brady was facing a lot of pressure, and either successfully avoided the pass rushers enough to throw the ball away, or get a pass off.

I don't think the OL was great either, but PFF didn't rate it "not great," they rated it abysmal. The Pats were -7.1 in pass blocking yesterday, which would pro-rate to -63.9 if they were that bad in all of their nine games. The worst pass blocking team in the league per PFF is Carolina at -46.6. So PFF thought the Patriots were 50% worse at pass blocking than the worst team in football. I agree Brady was a significant part of the story versus pressure, but there's no way he would have been sacked just once in 58 drop-backs if the OL was really that bad.
 
Part of the issue is calibration. PFF grades plays/players relative to "average" but they only calibrate periodically. Right now just 10 teams in the NFL grade above 0.0 in pass blocking, with 22 at -3.0 or worse. Additionally, there is no team better than +17.0 in pass blocking, while 15 grade worse than -17.0. PFF really only grades negative plays for linemen, but there's a mild positive default to balance that out (see "Normalization and What the Grades Mean"). Which makes sense - pass blocking is a sufficiency skill, where you can't really do a good job or a bad job - you either did the job or you failed. But obviously the calibration isn't working this year. So it's no surprise that a team that threw 58 times like the Patriots did yesterday is going to grade out negatively in pass blocking, because pretty much everyone is grading out lousy in pass blocking, and the Pats threw a lot, so there are were a lot of opportunities to grade out lousy. So that makes sense, in a way, but it doesn't make me any more confident in PFF's methods.

 

[Phragle]

Sure, but PFF didn't say Brady had an average surrounding offense; they said he had a terrible one. Apart from Brady the Pats were -10.4 offensively per PFF.

 
I don't think a -10.4 rating from 15 players means that much. Certainly less than you do. Maybe if it was vs a lesser team and after all-22 review it would mean more, but I think Denver's defense is nasty and it had a big impact on the ratings. Even Seattle - who also beat Denver - rated -11.5 on offense and that was with a QB performance that wasn't close to Brady's.

 

[PedroKsBambino]

I don't think the OL was great either, but PFF didn't rate it "not great," they rated it abysmal. The Pats were -7.1 in pass blocking yesterday, which would pro-rate to -63.9 if they were that bad in all of their nine games. The worst pass blocking team in the league per PFF is Carolina at -46.6. So PFF thought the Patriots were 50% worse at pass blocking than the worst team in football. I agree Brady was a significant part of the story versus pressure, but there's no way he would have been sacked just once in 58 drop-backs if the OL was really that bad.
 
Part of the issue is calibration. PFF grades plays/players relative to "average" but they only calibrate periodically. Right now just 10 teams in the NFL grade above 0.0 in pass blocking, with 22 at -3.0 or worse. Additionally, there is no team better than +17.0 in pass blocking, while 15 grade worse than -17.0. PFF really only grades negative plays for linemen, but there's a mild positive default to balance that out (see "Normalization and What the Grades Mean"). Which makes sense - pass blocking is a sufficiency skill, where you can't really do a good job or a bad job - you either did the job or you failed. But obviously the calibration isn't working this year. So it's no surprise that a team that threw 58 times like the Patriots did yesterday is going to grade out negatively in pass blocking, because pretty much everyone is grading out lousy in pass blocking, and the Pats threw a lot, so there are were a lot of opportunities to grade out lousy. So that makes sense, in a way, but it doesn't make me any more confident in PFF's methods.

 
Agreed.  PFF is trying to so something very difficult, and they (to my eyes) are making progress.  But just as baseball defensive metrics had embarassingly bad periods while they were learning, and NBA metrics still omit significant parts of the game, PFF will have these struggles too.   Some posters start from the premise "can we prove they are wrong?"  and this approach isn't a realistic way to assess metrics which are themselves unproven.

 

[Smiling Joe Hesketh]

 
Agreed.  PFF is trying to so something very difficult, and they (to my eyes) are making progress.  But just as baseball defensive metrics had embarassingly bad periods while they were learning, and NBA metrics still omit significant parts of the game, PFF will have these struggles too.   Some posters start from the premise "can we prove they are wrong?"  and this approach isn't a realistic way to assess metrics which are themselves unproven.

 
It's worth linking again to the long BSMW article that ran on PFF a while ago, because it's good to see exactly who they are and what they're doing:
 

 
What Pro Football Focus Is (And What They Aren’t)
 
It is important to note what Pro Football Focus is. Actually, first we’ll define what they are NOT.  They are not taking raw numbers and data and crunching them into new and exotic formulas to provide a different sort of insight into player performance. This is not sabermetrics for football.
 
No, their methods are different. They are a UK-based company, who obtain games through NFL Rewind and sit and watch and grade each player on each play. Their dedication to this is admirable, as I can’t imagine sitting down and doing this kind of deep grading for every play, every game week after week.

 

 
. In my interview linked above, founder Neil Hornsby said that PFF’s value is this:
 

• Who was on the field – in 2010 this was 99.83% accurate but we didn’t double hand most games then – this year we do so I’m predicting well in excess of 99.9%

• What position they played (at a level which allows us to provide formation as well as package information)

• What they generically did (block, pass route, cover, pass rush etc.)

• A measure of how well they achieved what they attempted to do (obviously we don’t know their assignments so this is what we use)

 
Check out that last line very, very carefully.
 
They're a bunch of guys from the UK using Game Rewind for their initial grades (the all-22 doesn't come out until the middle of the week so they're not using that for their first numbers) and assigning grades to players despite not knowing either the scheme the teams are using nor each player's role.
 
They aren't football experts. What are the football backgrounds of the guys looking at the film? They have no knowledge of the assigned roles in each play. They have no way of knowing where mistakes happen in, say coverage since they don't know if a safety was supposed to come over to help or if a DB had one man the whole way. They simply don't know. They're guessing.
 

 
Again, this is not taking actual numbers and using them to come up with new stats to use in analytics. This is not taking passes complete and passes attempted and breaking it down into the various lengths of throws and spots on the field. This is sitting down in front of the monitor, forming an opinion and making up their own stats and advanced formulas based on stats garnered from what they think is happening on each play.

 
I've said this a lot, but to me it bears repeating: I have no confidence in PFF's grades because their foundation is fundamentally unsolid. They're simply making a guess. And to me their grades have never, ever stood up to the laugh test. Never mind not accounting for opponent talent, their conclusions don't make a damn bit of common sense. Only 2 players on the offense had above-average games on Sunday? The Pats are way behind the Bears and behind the Redskins on the season? Come on. When you read that you have to assume that there's something fundamentally broken in PFF's process.

 

[lexrageorge]

Okay, I mean, your attitude here is fair - that's your prerogative. On the other hand, it also basically means you're not especially interested in stats period (you disregard them when they disagree with you). Again, that's fair, but probably a sign that you're not really the target audience for something like this.
 
I will probably have post up on Football Central eventually about the virtue of yards per play. Suffice to say, if you were to me one "basic" stat about a team, I'd pick yards per play as the most meaningful. It has more predictive value than just about anything else (success rate may be close last I looked).

I'd like to address a couple of points being made upthread.  First, with regards to the yard-per-play, I took a closer look at this past weekend's 13 games to see how well a couple of metrics predicted the outcome:
 
Net passing yards (passing yards - sacked) per attempt (YPA):   Leader won 10 of 13 games (exceptions were Washington/Minnesota, Rams/49'ers, and Broncos/Pats)
 
Adjusted net yards per attempt (net passing yards +TD*20 - INT*45) (AYPA):  Leader won 9 of 13 games (exceptions were Jaguars/Bengals, Washington/Minn, Houston/Philly, and Rams/49'ers) 
 
Yards per play (YPP): Leader won 9 of 13 games (exceptions were Tampa/Cleveland, Redskins/Vikings, Rams/49'ers, & Broncos/Pats)
 
Obviously, a small sample size, and nothing really unexpected.  All 3 stats matter to some degree.  Minnesota won by 3 partly because they were able to score a TD on their 4 scoring drives while Washington settled for 2 FG's and 2 TD's.  The Rams/49'ers was also very close, with St. Louis benefiting from 2 49'er fumbles (lost fumbles are not considered in any of the above stats).  The difference in the Bucs/Browns game was one blocked FG and one missed FG.  Manning's 2 picks hurt his AYPA rating, while Brady's 4 TD's helped his.  As pure stats go, the winner in any of the above categories is more likely to win the game.  
 
However, my quibble with PFF is not the fact they use statistics.  It's that I don't believe their rating method is robust.  YPA, AYPA, YPP are stats that can be calculated by anyone.  Similarly, with enough data, one can calculate the cost of an INT in field position and win probability.  PFF doesn't do this.  Instead, someone, somewhere arbitrarily decides how much a play is worth to each player.  That is not a statistic, and without any details as to how that determination is made, PFF rightfully lends itself to criticism when its results fail both the eye test and the smell test.  When the results are so contradictory, an explanation is in order, but none seems to be forthcoming. 

 

[Smiling Joe Hesketh]

However, my quibble with PFF is not the fact they use statistics.  It's that I don't believe their rating method is robust.  YPA, AYPA, YPP are stats that can be calculated by anyone.  Similarly, with enough data, one can calculate the cost of an INT in field position and win probability.  PFF doesn't do this.  Instead, someone, somewhere arbitrarily decides how much a play is worth to each player.  That is not a statistic, and without any details as to how that determination is made, PFF rightfully lends itself to criticism when its results fail both the eye test and the smell test.  When the results are so contradictory, an explanation is in order, but none seems to be forthcoming. 

 
Exactly. That's not a quibble, that's a howitzer that aims right for the heart of PFF's business. This isn't football sabremetrics, this is non-expertise scouting being done and passed off as objective analysis.

 

[DJnVa]

 
 
 
They're a bunch of guys from the UK using Game Rewind for their initial grades (the all-22 doesn't come out until the middle of the week so they're not using that for their first numbers) and assigning grades to players despite not knowing either the scheme the teams are using nor each player's role.
 
They aren't football experts. What are the football backgrounds of the guys looking at the film? They have no knowledge of the assigned roles in each play. They have no way of knowing where mistakes happen in, say coverage since they don't know if a safety was supposed to come over to help or if a DB had one man the whole way. They simply don't know. They're guessing.
 
 

 

This is important because we've heard Belichick say numerous times that it's hard to say who messed up on a certain play without knowing what they were supposed to do. If BB can look at the All-22 and not know for certain, we need to take PFF's number with a huge grain of salt.

 

[rodderick]

This is important because we've heard Belichick say numerous times that it's hard to say who messed up on a certain play without knowing what they were supposed to do. If BB can look at the All-22 and not know for certain, we need to take PFF's number with a huge grain of salt.

 
Not to completely make a left turn in the conversation, but this post reminds me of this Belichick quote from the same interview:
 

But believe me, I’ve watched plenty of preseason games this time of year and you’re looking at all the other teams in the league and you try to evaluate players and you’re watching the teams that we’re going to play early in the season and there are plenty of plays where I have no idea what went wrong. Something’s wrong but I don’t…these two guys made a mistake but I don’t know which guy it was or if it was both of them. You just don’t know that. I don’t know how you can know that unless you’re really part of the team and know exactly what was supposed to happen on that play. I know there are a lot of experts out there that have it all figured out but I definitely don’t. This time of year, sometimes it’s hard to figure that out, exactly what they’re trying to do. When somebody makes a mistake, whose mistake is it?

 
And it just boggles my mind how people still think "Spygate" was anything other than Belichick trying to extract more information about opposing teams' schemes and reactions in certain situations, so that his scouting of them would be more precise and he could identify tendencies more easily. There are people out there who truly believe filming some hand signals gives an offense the entire knowledge of the opposition playbook and what they were trying to accomplish is a given play. 

 

[bowiac]

However, my quibble with PFF is not the fact they use statistics.  It's that I don't believe their rating method is robust.  YPA, AYPA, YPP are stats that can be calculated by anyone.  Similarly, with enough data, one can calculate the cost of an INT in field position and win probability.  PFF doesn't do this.  Instead, someone, somewhere arbitrarily decides how much a play is worth to each player.  That is not a statistic, and without any details as to how that determination is made, PFF rightfully lends itself to criticism when its results fail both the eye test and the smell test.  When the results are so contradictory, an explanation is in order, but none seems to be forthcoming. 

I don't agree they're deciding "arbitrarily", but I agree its not a statistic. I also disagree that it doesn't pass the smell test.
 

I don't see what value PFF ratings have for players where we have statistics, to be honest. Most of the time they square up pretty well anyway, and in the cases where a receiver scores poorly by PFF and well by PBP-based metrics (like Edelman, who they currently grade out as the sixth-worst WR in football) or vice versa, I don't see any reason to think the PFF score is more valid than more conventional stats. And without more granular ratings, I can't even see whether it's providing any value at all even as a dissenting opinion.

I agree the main value of PFF ratings is for players where we don't have statistics. That's half or more players on the field however - (five linemen, plus any receivers/backs who stay in to block). I'm not a very good scout, so for me, if PFF is shedding any light on which linemen are doing their jobs and which aren't, that's a real value added.
 
With respect to players where we do have statistics (sort of - since PBP metrics don't track blocking to my knowledge), I agree there's no reason to think PFF is more valid than conventional stats. However, it's not crazy to me that it can also shed some light on something, although I don't yet know what that light is. We see this happen in other fields - why not here?

 

[Super Nomario]

With respect to players where we do have statistics (sort of - since PBP metrics don't track blocking to my knowledge), I agree there's no reason to think PFF is more valid than conventional stats. However, it's not crazy to me that it can also shed some light on something, although I don't yet know what that light is. We see this happen in other fields - why not here?

I agree in principle, I just think in theory that PFF doesn't provide the tools for us to do that. Taking Edelman as an example, they grade him as -5.1 in the passing game. Why? Are they not impressed with his ability to get open short? Deep? Is it his hands? His ability to make contested catches? Do they think he lacks run-after-catch ability? Do they think his production is the product of scheme or Brady and that anyone can do what he does? If PFF provided answers to these questions, it might shed light on something, as you say. But they don't, so that's just a number, divorced of context. It's not a statistic; it's not even information.
 
And this is true for their ratings for OL and defensive players, too, only we don't have PBP statistics to sanity check PFF numbers against. That doesn't make those grades any more valid of course, and we should be sure to take their ratings of OL and defenders with a grain (or a shaker) of salt even though we don't have anything better.

 

[bowiac]

I agree in principle, I just think in theory that PFF doesn't provide the tools for us to do that. Taking Edelman as an example, they grade him as -5.1 in the passing game. Why? Are they not impressed with his ability to get open short? Deep? Is it his hands? His ability to make contested catches? Do they think he lacks run-after-catch ability? Do they think his production is the product of scheme or Brady and that anyone can do what he does? If PFF provided answers to these questions, it might shed light on something, as you say. But they don't, so that's just a number, divorced of context. It's not a statistic; it's not even information.
 
And this is true for their ratings for OL and defensive players, too, only we don't have PBP statistics to sanity check PFF numbers against. That doesn't make those grades any more valid of course, and we should be sure to take their ratings of OL and defenders with a grain (or a shaker) of salt even though we don't have anything better.

If you're complaining it's a black box, then sure. But that also doesn't mean they're wrong. DVOA is largely black-boxish. The best basketball stat, RPM, is close to a black box. The test, with all these things, is how well they help you predict future events. My limited research suggests PFF data is helpful in that regard, providing useful information not contained in other data sources. 

 

[Morgan's Magic Snowplow]

If you're complaining it's a black box, then sure. But that also doesn't mean they're wrong. DVOA is largely black-boxish. The best basketball stat, RPM, is close to a black box. The test, with all these things, is how well they help you predict future events. My limited research suggests PFF data is helpful in that regard, providing useful information not contained in other data sources. 

 
Out of curiosity, what sort of research have you done in this regard?

 

[Super Nomario]

If you're complaining it's a black box, then sure. But that also doesn't mean they're wrong. DVOA is largely black-boxish. The best basketball stat, RPM, is close to a black box. The test, with all these things, is how well they help you predict future events. My limited research suggests PFF data is helpful in that regard, providing useful information not contained in other data sources. 

I am complaining it's a black box, and I also have that complaint about DVOA. I think it's very unfortunate that FO and PFF are the two leaders in football "sabermetrics" and both have decided to withhold information that would help us understanding what they're tracking and how it ties into our evaluation of teams and players. I find value in the work PFF does with some of their signature stats (trackers pressure / hits, yards per route run, coverage statistics), but their ratings (and FO's DVOA) tend to forestall discussion rather than invite it. I think that's a serious barrier to a better understanding of football and what helps teams win games.
 
EDIT: And you're right that it "doesn't mean they're wrong," but we don't have any tools for answering whether they are or not. I can look at stuff like the pass blocking for the league as a whole and see that they've calibrated it improperly, but on the micro level (like the Edelman rating) it's basically an article of faith whether their score is nonsense or not. How does something like that move us forward?

 

[bowiac]

Out of curiosity, what sort of research have you done in this regard?

I built a model for predicting season win totals (against the Vegas lines), using a combination of Football Outsiders and Football-reference data. The results were fine, but I was able to significantly strengthen the out of sample R^2 by adding in Pro Football Focus data to the regression as well. While the total ratings were not a significant predictor, the run and pass blocking ratings were.
 
I'm going to try and do something for Football Central looking at this more in depth. Outside of my win total research, I haven't looked at the PFF data much.

 

[bowiac]

I am complaining it's a black box, and I also have that complaint about DVOA. I think it's very unfortunate that FO and PFF are the two leaders in football "sabermetrics" and both have decided to withhold information that would help us understanding what they're tracking and how it ties into our evaluation of teams and players. I find value in the work PFF does with some of their signature stats (trackers pressure / hits, yards per route run, coverage statistics), but their ratings (and FO's DVOA) tend to forestall discussion rather than invite it. I think that's a serious barrier to a better understanding of football and what helps teams win games.
 
EDIT: And you're right that it "doesn't mean they're wrong," but we don't have any tools for answering whether they are or not. I can look at stuff like the pass blocking for the league as a whole and see that they've calibrated it improperly, but on the micro level (like the Edelman rating) it's basically an article of faith whether their score is nonsense or not. How does something like that move us forward?

I'm not sure I agree PFF has "withheld" anything really. They're pretty open about what they're doing. It's black box in that you need to take them on faith that they're doing it right.
 
If, as I suspect, their run/pass blocking ratings help you predict future outcomes, then they absolutely help move us forward. 

 

[Morgan's Magic Snowplow]

I built a model for predicting season win totals (against the Vegas lines), using a combination of Football Outsiders and Football-reference data. The results were fine, but I was able to significantly strengthen the out of sample R^2 by adding in Pro Football Focus data to the regression as well. While the total ratings were not a significant predictor, the run and pass blocking ratings were.
 
I'm going to try and do something for Football Central looking at this more in depth. Outside of my win total research, I haven't looked at the PFF data much.

 
Interesting.  I'm generally a PFF critic but its not surprising to me that the OL ratings in particular might be useful - basically, its all about the utility of those ratings compared to other publicly available information about the quality of OL and the latter is so bad that improving on it is not that hard.