Analysing the Analysists: the ProFootballFocus Discussion Thread

[rodderick]

I don't agree they're deciding "arbitrarily", but I agree its not a statistic. I also disagree that it doesn't pass the smell test.
 
I agree the main value of PFF ratings is for players where we don't have statistics. That's half or more players on the field however - (five linemen, plus any receivers/backs who stay in to block). I'm not a very good scout, so for me, if PFF is shedding any light on which linemen are doing their jobs and which aren't, that's a real value added.
 
With respect to players where we do have statistics (sort of - since PBP metrics don't track blocking to my knowledge), I agree there's no reason to think PFF is more valid than conventional stats. However, it's not crazy to me that it can also shed some light on something, although I don't yet know what that light is. We see this happen in other fields - why not here?

 
That would be awesome if PFF had the slightest fucking clue about what exactly an O-lineman's job is in any given play, because they sure aren't scouts either. If you don't know assignments, you can't accurately tell who screwed up, who looked bad because they were compensating for someone else's failure, who did their job correctly but ended up in a terrible situation because the QB made the wrong adjustment at the line and etc. Aside from the very obvious screw ups/good plays that any reasonably knowledgeable fan can pick up, I can't see their judgment of offensive line play as anything other than gut feeling and general observation. I may very well be wrong about this, but I'd really like someone to tell me why I should trust PFF in their analysis of the OL over your average reporter's observation, provided he was present at the game/has access to NFL Game Rewind.

 

[bowiac]

I may very well be wrong about this, but I'd really like someone to tell me why I should trust PFF in their analysis of the OL over your average reporter's observation, provided he was present at the game/has access to NFL Game Rewind.

Well, the core reason is that PFF is going back and charting every play, while the "average reporter" in question is usually not. Just to pick on Mike Reiss, while he's at the game, and certainly has access to Rewind, is he actually sitting down and spending a few hours watching every play a few times grading each lineman on each play? If he is, then I'd love to see those ratings. I don't read much post-game coverage, but I haven't seen him put out such ratings.
 
I have no reason to think the PFF guys are actually any good, but they're (allegedly) actually taking the time to do all that. Mike Reiss is not. He's rewatching the games, but he's picking out the key parts, not trying to do any kind of granular analysis. That's valuable too, but it's also a very different task than what the PFF guys are doing.

 

[tims4wins]

Well, the core reason is that PFF is going back and charting every play, while the "average reporter" in question is usually not. Just to pick on Mike Reiss, while he's at the game, and certainly has access to Rewind, is he actually sitting down and spending a few hours watching every play a few times grading each lineman on each play? If he is, then I'd love to see those ratings. I don't read much post-game coverage, but I haven't seen him put out such ratings.
 
I have no reason to think the PFF guys are actually any good, but they're (allegedly) actually taking the time to do all that. Mike Reiss is not. He's rewatching the games, but he's picking out the key parts, not trying to do any kind of granular analysis. That's valuable too, but it's also a very different task than what the PFF guys are doing.

 
Reiss probably spends more time reviewing film than most reporters, but even he doesn't grade every play. I do know that he re-watches every play to A) chart personnel / groupings and B) make observations. Generally he does a blog post for each quarter of each game with roughly ten bullet points per quarter of "picked up pieces". I really enjoy those posts.

 

[rodderick]

Well, the core reason is that PFF is going back and charting every play, while the "average reporter" in question is usually not. Just to pick on Mike Reiss, while he's at the game, and certainly has access to Rewind, is he actually sitting down and spending a few hours watching every play a few times grading each lineman on each play? If he is, then I'd love to see those ratings. I don't read much post-game coverage, but I haven't seen him put out such ratings.
 
I have no reason to think the PFF guys are actually any good, but they're (allegedly) actually taking the time to do all that. Mike Reiss is not. He's rewatching the games, but he's picking out the key parts, not trying to do any kind of granular analysis. That's valuable too, but it's also a very different task than what the PFF guys are doing.

 
So they chart every play, but by their own guidelines when they aren't sure of how to grade a player in a given snap, they are supposed to assign a grade of 0.0, right? Are you telling me that these guys are capable of being 95% sure about how to grade an individual offensive lineman in the majority of plays? Because if they don't, a more general analysis that focuses on the more visible good/bad plays is pretty much just as effective, but way less pretentious about presenting objective truth about player performance. I honestly feel I have more knowledge about what happened in the game reading some articles on SoSH's football central, than reading the surprisingly generic performance reviews that accompany PFF's (almost entirely subjective) number ratings.
 
This:
 

Eli Manning, QB: -4.2
Breakdown: Manning’s performances have yo-yoed either side of the Giants’ bye week as he produced a performance reminiscent of the defeat in Philadelphia rather than the valiant effort in Dallas. The volume of yards and lack of touchdowns hide a performance marred by inaccuracy and forced throws that only really came together with the game all but out of reach in the final quarter.

 
Isn't exactly hard hitting analysis. I honestly don't know jack shit about how Eli played if I didn't watch the game.
 
I don't even know why I'm so annoyed by this subject, so please tell me if I'm repeating myself too much and fucking up the discussion. 

 

[bowiac]

So they chart every play, but by their own guidelines when they aren't sure of how to grade a player in a given snap, they are supposed to assign a grade of 0.0, right? Are you telling me that these guys are capable of being 95% sure about how to grade an individual offensive lineman in the majority of plays? Because if they don't, a more general analysis that focuses on the more visible good/bad plays is pretty much just as effective, but way less pretentious about presenting objective truth about player performance.
 
I don't even know why I'm so annoyed by this subject, so please tell me if I'm repeating myself too much and fucking up the discussion. 

Where is 95% coming from?
 
I think generally the PFF critics are holding them to too high a standard. The standard isn't perfection, or 95%, or whatever - it's something above the baseline we're working with otherwise. With line play, that baseline is exceptionally low, so its reasonably easily for PFF to add value. With QB play, that's a pretty high baseline meanwhile. I would be surprised if PFF data had anything substantive to tell us yet about QB play. With WR/RBs, it's probably somewhere in between.

 

[rodderick]

Where is 95% coming from?
 
I think generally the PFF critics are holding them to too high a standard. The standard isn't perfection, or 95%, or whatever - it's something above the baseline we're working with otherwise. With line play, that baseline is exceptionally low, so its reasonably easily for PFF to add value. With QB play, that's a pretty high baseline meanwhile. I would be surprised if PFF data had anything substantive to tell us yet about QB play. With WR/RBs, it's probably somewhere in between.

 
Apparently, this excerpt was recently on their grading page:
 

 DON’T GUESS — If you’re not 95 percent sure what’s gone on then don’t grade the player for that play. The grades must stand up to scrutiny and criticism, and it’s far better to say you’re not sure than be wrong.
It is, however, crucial that this is not seen as an excuse to shy away from making a judgement. What we definitely do not do is raise or lower the grading because we’re not sure. Giving a grade of -0.5 rather than -1.5 for a player on an individual play because you’re unsure is the wrong grade to give. If the grader is 95 percent sure of the severe fault on the play, the grade is -1.5. If, however, the grader is unsure of his judgment, the correct grade is 0.

 
But I can't find it anymore. It's here, though, so at the very least 95% was a hard guideline as of June, 2014.

 

[lexrageorge]

I'm not sure I agree PFF has "withheld" anything really. They're pretty open about what they're doing. It's black box in that you need to take them on faith that they're doing it right.
 
If, as I suspect, their run/pass blocking ratings help you predict future outcomes, then they absolutely help move us forward. 

And here's my fundamental problem.  Let's take Julian Edelman.  We know PFF does not rate him highly.  
 
We also know from his basic stats that he is 6th in the league in receptions, 9th in targets, 20th in yards, 93rd in yards per catch, 48th in yards after the catch, 19th in first down catches.  We look at the eye test and we see him make quick moves to get open, and we see him make some difficult catches in traffic as a slot receiver should.  We also see some drops, and we know he had bad games against the Bears and the Chiefs.  
 
But I have no idea how or if PFF weights any of these factors when they rank Edelman.  All we know is that there's some reason they don't like him, and some of that is based on a very subjective judgment of whether his play contributed positively or negatively at any given time.  There is some explanation, but we never hear it.  So I have trouble trusting such a system without some solid proof that it is indeed accurate.  Instead all I hear is "Trust us, we know what we're doing".  
 
Perhaps we should ignore their rankings for individual players but instead look at the Pats WR corps in aggregate.  But then why do they bother rating individual players?  
 
Transparency shouldn't be overlooked when evaluating the predictive capability of a model.  

 

[Super Nomario]

I'm not sure I agree PFF has "withheld" anything really. They're pretty open about what they're doing. It's black box in that you need to take them on faith that they're doing it right.
 
If, as I suspect, their run/pass blocking ratings help you predict future outcomes, then they absolutely help move us forward.

In both cases, though, we're stuck with the crumbs that FO and PFF deign to give us, so even to the extent there is some value in the exercises you describe we're stuck there. We can't see why they work, break them down into elements that are predictive and descriptive, determine appropriate weightings to adjust, etc. We can't see what factors are traditionally underrated or overrated and use it to help us scout future players, project team fits, estimate which players might fit better in a different scheme or position, etc. That's great if it gives you better numbers to plug into a prediction model, but DVOA and PFF ratings don't really advance our understanding of the game at all. You can see that adding the pass blocking and run blocking numbers improves your correlation, but you don't know why - where do you go from there?

 

[Jnai]

What would you be satisfied with?

 

[williams_482]

What would you be satisfied with?

PBP logs which show how the player was rated for any given play would be a big help.

 

[rodderick]

What would you be satisfied with?

 
Honestly, if they're rating every single snap, they should offer a play by play breakdown with an explanation as to why a player was graded negatively or positively in each play. There would still be a huge amount of subjectivity involved, but at least that way it would make it more clear that this is only one person's opinion. Maybe something like this is available for premium members, but I can't tell.
 
My main problem with PFF is the amount of times I see their ratings used almost as a football version of WAR. It's fucking annoying in baseball when someone just offers up WAR without any context to try to settle an argument, but at least that's a well researched statistical concept that is used in the context of a sport that lends itself to individual statistical analysis. It seems like some view the PFF ratings as some objective measure, when that couldn't be further from the truth. 

 

[Shelterdog]

I built a model for predicting season win totals (against the Vegas lines), using a combination of Football Outsiders and Football-reference data. The results were fine, but I was able to significantly strengthen the out of sample R^2 by adding in Pro Football Focus data to the regression as well. While the total ratings were not a significant predictor, the run and pass blocking ratings were.
 
I'm going to try and do something for Football Central looking at this more in depth. Outside of my win total research, I haven't looked at the PFF data much.

Do the PFF online stats tell you more than, say, sacks plus hurries or tackles for loss against?

 

[Jnai]

 
Honestly, if they're rating every single snap, they should offer a play by play breakdown with an explanation as to why a player was graded negatively or positively in each play. There would still be a huge amount of subjectivity involved, but at least that way it would make it more clear that this is only one person's opinion. Maybe something like this is available for premium members, but I can't tell.

 
Of course, this would be an insane amount of work.

 

[DJnVa]

 
Of course, this would be an insane amount of work.

 
Yea, but they could start by just showing the grade for each play.

 

[rodderick]

 
Of course, this would be an insane amount of work.

 
Sure, but it's the only thing that would make their ratings model viable in my eyes. Just saying "Player X receives a 1.4 rating in this game. He had a solid, but unspectacular performance against Team Y" isn't enough. Merely showing the grade for each play would be a start, since then we could watch it on our own and compare to see how far we can trust the reviewer.

 

[EricFeczko]

One way to compare PFF's numbers is to use the "eye test". Another is to use other models of football performance to see whether PFF's numbers are consistent or not.

Brian Burke's EPA model for evaluating performance finds that the top performers for the Pats on offense (via expected points added: EPA) were Brady, Vereen, Gronk, Edelman and Amendola. The O-line, Lafell, and Gray detracted from the offense, but not by much. Gray was really the biggest detractor in general.
 
To me, this fits in line quite nicely with the "eye test" and suggests that PFF's numbers are not very reliable.

 

[Super Nomario]

 
Of course, this would be an insane amount of work.

Presumably they do this on their end, so it would just be making the information publicly available.
 
As a smaller step, having more granular categories is essential. As I noted above, "pass" can represent a half-dozen different things, and we ought to know which items are reflected in the rankings. Better splits would be useful - PFF keeps very detailed records on how players line up, but the only thing they really share is slot vs outside for receivers, slot vs tight for TEs, and slot vs outside for corners. But they could easily provide rushing stats from various spots (Phragle and I had a back-and-forth last year where Palazzolo gave us Chandler Jones' rushing numbers from 4-3 DE, 3-4 OLB, 3-4 DE, 4-3 DT, etc.), safety ratings by deep or in box, man vs zone statistics, press coverage and receivers vs press coverage, etc. They track all of this; they just don't share it, which is sad.

 

[PedroKsBambino]

 
Of course, this would be an insane amount of work.

 
If they are doing what they say they are, they are doing all the work already---it's only the visibility that would change.

 

[Jnai]

Well, there's a difference between providing a more granular representation and "an explanation as to why a player was graded negatively or positively in each play." I think a more granular representation is a completely reasonable thing to ask for, and is probably a very valuable feature of their dataset. Why they don't provide it is an interesting question. Do they sell this data to media or something?

 

[bowiac]

Transparency shouldn't be overlooked when evaluating the predictive capability of a model.  

Why? I'm serious - DVOA isn't very transparent, but I know as much about its predictive capability as I know about SRS (completely transparent and easy to calculate). Transparency is a good thing, but predictive capability isn't one of the the things it seems to be important for.
 

One way to compare PFF's numbers is to use the "eye test". Another is to use other models of football performance to see whether PFF's numbers are consistent or not.

I basically disagree with your entire premise. If we wanted something that confirmed EPA/the eye test, then we could just use EPA/the eye test. There's no reason to look for other sources of data if we just want to confirm what we already know. The value of PFF data comes from the places where it disagrees.
 
Now, lets say I've come up with a new system to measure baseball player valuation, and it tells us that Nick Punto is the best player in baseball. We should be rightly skeptical of that model because we know a lot about baseball stats these days, and the rest of them tell us Nick Punto stinks. It's not really plausible that we've been missing Nick Punto's greatness this whole time.
 
That's not really the case with football. We don't know nearly as much, so disagreement with our other data sources isn't as big a deal. Further, the sources we do have, like EPA, are often times not even trying to measure the same thing as PFF, comparing the two is especially unhelpful.

 

[lexrageorge]

Why? I'm serious - DVOA isn't very transparent, but I know as much about its predictive capability as I know about SRS (completely transparent and easy to calculate). Transparency is a good thing, but predictive capability isn't one of the the things it seems to be important for.
 

Here's my problem:  we know that PFF determines how each player should be graded for each play in which they are involved.  We know that the grade could be positive, negative, or 0, and that the zero grade should be given unless the grader is 95% certain of the grade.  But who double checks the grader?  How does the grader know if the player did the right thing on that play, when Belichick himself has gone on record as saying it's almost impossible to know that for players outside your team?  By not knowing any of these things, the assumption is that the grading is often subjective, which calls into question the model and its predictive ability.  Due to the lack of transparency, any discussion of the model becomes uninteresting.  And without such discussion, I cannot understand how the model will improve over time.  
 
When other statistical models fail the eye test, there is usually a ready explanation as to why and how the discrepancy exists.  That doesn't happen with PFF; at least the FO guys do sometimes provide some insights as to why DVOA departs from expectations.  
 
Maybe it's just personal preference; some folks are OK with "trust me" models, I get that.  I'm not.  And when the model blatantly departs from the eye test, as it did with Edelman in the Broncos game, and no further explanation is offered, then the model is, IMO, borderline worthless.  And without solid evidence that their model is actually predictive by itself, as opposed to being a simple curve fit, it is nothing more than a "trust me" model. 

 

[bowiac]

Maybe it's just personal preference; some folks are OK with "trust me" models, I get that.  I'm not.  And when the model blatantly departs from the eye test, as it did with Edelman in the Broncos game, and no further explanation is offered, then the model is, IMO, borderline worthless.  And without solid evidence that their model is actually predictive by itself, as opposed to being a simple curve fit, it is nothing more than a "trust me" model. 

I mean, I'd prefer if I were getting a long narrative writeup for every play, but I imagine that service would be much more expensive.
 
I didn't get the sense that Edelman had some kind of amazing receiving game, but like I said, I don't know what to watch for, especially with stuff like route running. He had a great punt return, which was obviously value added and PFF gives him credit for it. Is his receiving number really so far outside the eye test?
 
It seems to me that I just have lower expectations for value-added analysis than others. I don't need another stat like EPA that mostly just repackages that Edelman had 89 yards receiving. The box score gives me that already. Telling me his 89 yards were dramatically less impressive than Gronkowski's 105 yards on the other hand at least gives me something to think about. I don't know if it's accurate, but it's plausible to me at least, and worth investigating. That's what spurs discussion.
 
Now, if time and time again, you do a deep dive into PFF's numbers and find there's no plausible explanation, then it's fair to move on, but I don't get the sense that's where we're at. Right now, most of the complaints remains at the "C'mon, he had a great game!" stage.

 

[PedroKsBambino]

Why? I'm serious - DVOA isn't very transparent, but I know as much about its predictive capability as I know about SRS (completely transparent and easy to calculate). Transparency is a good thing, but predictive capability isn't one of the the things it seems to be important for.
 
I basically disagree with your entire premise. If we wanted something that confirmed EPA/the eye test, then we could just use EPA/the eye test. There's no reason to look for other sources of data if we just want to confirm what we already know. The value of PFF data comes from the places where it disagrees.
 
Now, lets say I've come up with a new system to measure baseball player valuation, and it tells us that Nick Punto is the best player in baseball. We should be
rightly skeptical of that model because we know a lot about baseball stats these days, and the rest of them tell us Nick Punto stinks. It's not really plausible that we've been missing Nick Punto's greatness this whole time.
 
That's not really the case with football. We don't know nearly as much, so disagreement with our other data sources isn't as big a deal. Further, the sources we do have, like EPA, are often times not even trying to measure the same thing as PFF, comparing the two is especially unhelpful.

 
The only note is while it is true we should carefully examine conclusions that are different than our existing biases (whether those biases are 'eye test' or 'existing metrics') we also do need to remain open to the possibility that there is something we've historically been missing. This is, really, the single most important principle of sabermetrics and it's important to reemphasize it.  Just because we've made progress on measurement does not mean we can't have a paradigm shift in the future, and it does not mean we should assume our existing metrics are right.

Do I expect we'll have one and learn about the greatness of Punto?  No...

 

[Shelterdog]

I mean, I'd prefer if I were getting a long narrative writeup for every play, but I imagine that service would be much more expensive.
 
I didn't get the sense that Edelman had some kind of amazing receiving game, but like I said, I don't know what to watch for, especially with stuff like route running. He had a great punt return, which was obviously value added and PFF gives him credit for it. Is his receiving number really so far outside the eye test?
 
It seems to me that I just have lower expectations for value-added analysis than others. I don't need another stat like EPA that mostly just repackages that Edelman had 89 yards receiving. The box score gives me that already. Telling me his 89 yards were dramatically less impressive than Gronkowski's 105 yards on the other hand at least gives me something to think about. I don't know if it's accurate, but it's plausible to me at least, and worth investigating. That's what spurs discussion.
 
Now, if time and time again, you do a deep dive into PFF's numbers and find there's no plausible explanation, then it's fair to move on, but I don't get the sense that's where we're at. Right now, most of the complaints remains at the "C'mon, he had a great game!" stage.

I'd flip the burden of proof. Why should we care about their system until they actually show it has predictive or descriptive value? The fact that it disagrees with the eye test doesn't matter unless it disagrees correctly.

 

[Smiling Joe Hesketh]

 
The only note is while it is true we should carefully examine conclusions that are different than our existing biases (whether those biases are 'eye test' or 'existing metrics') we also do need to remain open to the possibility that there is something we've historically been missing. This is, really, the single most important principle of sabermetrics and it's important to reemphasize it.  Just because we've made progress on measurement does not mean we can't have a paradigm shift in the future, and it does not mean we should assume our existing metrics are right.
Do I expect we'll have one and learn about the greatness of Punto?  No...

I agree with this, but I would argue that what PFF is doing is not sabermetrics at all.

 

[bowiac]

I'd flip the burden of proof. Why should we care about their system until they actually show it has predictive or descriptive value? The fact that it disagrees with the eye test doesn't matter unless it disagrees correctly.

This is fair. I give them the benefit of the doubt because: 1) It doesn't offend my "eye test" all that much; 2) I like the idea behind the project, even knowing its inherent limitations; 3) For some sources, like blocking, we don't have anything else of substance to go off of. Why do I think Logan Mankins was a great LG? Because everyone said so, and I don't have any reason to think otherwise.
 
To an extent however, I disagree that there is any burden of proof. Some of this is just an avenue for investigation. I don't subscribe to NFL game rewind, but I'll probably find a source to watch each of Edelman's targets, to see if there's anything going on there for instance. 

 

[PedroKsBambino]

I agree with this, but I would argue that what PFF is doing is not sabermetrics at all.

 
That's a very philosophical comment!  For me, and I don't know if it's a 'correct' definition in the minds of others, I'd say that trying to apply a structured set of standards to data for purposes of analysis falls within 'sabermetrics'   But I also agree with many of your concerns about the way they are (or might be) doing it, too.  I think that goes to validity of their conclusions, though understand one might say it goes to the nature of their inquiry too.   
 
To net it out, I like their idea, I consider their data and appreciate when it's posted as one of many inputs in my own assessment, and I also have enough concerns about their approach that I won't simply defer to their rankings.  
 
Personally, I find the argument of 'we don't have anything else' is a little disingenuous---we have opinions from others who watch, we have our own impressions, we have a very small set of statistics (noisy as they are, like sacks against), etc.   Just because someone comes up with a number doesn't mean the number is reliable, and we should always inquire whether it is.

 

[Phragle]

Well, there's a difference between providing a more granular representation and "an explanation as to why a player was graded negatively or positively in each play." I think a more granular representation is a completely reasonable thing to ask for, and is probably a very valuable feature of their dataset. Why they don't provide it is an interesting question. Do they sell this data to media or something?

 
They sell it to teams. I don't know about the media. I'm guessing they feel there's a low demand for even more in depth stats and it wouldn't be worth the time or additional clutter on the site.
 

How does the grader know if the player did the right thing on that play, when Belichick himself has gone on record as saying it's almost impossible to know that for players outside your team? 

 
Well how does anyone? Should we just abandon every type of film based evaluation because we don't know what the calls are? Fire the scouts and GMs and then use a roulette wheel instead?

 

[PedroKsBambino]

Well how does anyone? Should we just abandon every type of film based evaluation because we don't know what the calls are? Fire the scouts and GMs and then use a roulette wheel instead?

 
I don't think it means we throw it out completely and just say it's random numbers; I do think it's fair to say that it lowers our confidence that their rating is a fair reflection of the player's skills, though.

 

[Jnai]

 
They sell it to teams. I don't know about the media. I'm guessing they feel there's a low demand for even more in depth stats and it wouldn't be worth the time or additional clutter on the site.

 
They could also feel like they can't give away the entire product.

 

[williams_482]

Now, lets say I've come up with a new system to measure baseball player valuation, and it tells us that Nick Punto is the best player in baseball. We should be rightly skeptical of that model because we know a lot about baseball stats these days, and the rest of them tell us Nick Punto stinks. It's not really plausible that we've been missing Nick Punto's greatness this whole time.

I guess this is beyond the point, but out best available baseball metrics actually do say that Nick Punto was a pretty good player, averaging roughly 2.5 wins per 650 PAs according to both B-Ref and Fangraphs. 
 

 
They could also feel like they can't give away the entire product.

I get why FO keeps DVOA as a black box, as if they revealed it anyone could get exactly the same results with a solid understanding of R or Excel (or whatever they use). That isn't true with PFF: we all know what their methodology is, they publish it for all the world to see. The reason that PFF stats are worth the subscription price to some people is that they require a shit ton of charting time by (I assume) a fairly large group of people, and setting all that up would be expensive enough that you wouldn't expect to make much money trying to compete with them. 
 
Unless I am wildly off base here, it seems that being able to see how PFF rated a 6 yard catch on 2nd and 3 wouldn't affect their long term outlook one bit. 

 

[Jnai]

I guess this is beyond the point, but out best available baseball metrics actually do say that Nick Punto was a pretty good player, averaging roughly 2.5 wins per 650 PAs according to both B-Ref and Fangraphs. 
 
I get why FO keeps DVOA as a black box, as if they revealed it anyone could get exactly the same results with a solid understanding of R or Excel (or whatever they use). That isn't true with PFF: we all know what their methodology is, they publish it for all the world to see. The reason that PFF stats are worth the subscription price to some people is that they require a shit ton of charting time by (I assume) a fairly large group of people, and setting all that up would be expensive enough that you wouldn't expect to make much money trying to compete with them. 
 
Unless I am wildly off base here, it seems that being able to see how PFF rated a 6 yard catch on 2nd and 3 wouldn't affect their long term outlook one bit. 

 
Sure, but you also can't sell data if you're literally giving away all the data. There are plenty of websites and data collection companies that do similar things in baseball.

 

[Super Nomario]

Now, if time and time again, you do a deep dive into PFF's numbers and find there's no plausible explanation, then it's fair to move on, but I don't get the sense that's where we're at. Right now, most of the complaints remains at the "C'mon, he had a great game!" stage.

What constitutes a "deep dive," though? There's nowhere further to go with the PFF info unless you do your own film breakdown. It's a fundamental problem. It's pretty telling that discussion of individual rankings has immediately gone to a methodology discussion, because there's no further level of granularity to argue. If I think a particular Y/A figure is bullshit, we can have discussions about Y/A vs success rate, whether it's being disproportionately affected by large plays, how to weigh situations (red zone, 3rd down, garbage time), and the repeatability of turnovers. We can't go anywhere with the PFF numbers without going back to the film, which kind of defeats the point. 
 

Sure, but you also can't sell data if you're literally giving away all the data. There are plenty of websites and data collection companies that do similar things in baseball.

I don't think anyone's suggesting they give it away for free. But they don't give this information even to their subscribers, even though they have it.

 

[bowiac]

What constitutes a "deep dive," though? There's nowhere further to go with the PFF info unless you do your own film breakdown. It's a fundamental problem. It's pretty telling that discussion of individual rankings has immediately gone to a methodology discussion, because there's no further level of granularity to argue. If I think a particular Y/A figure is bullshit, we can have discussions about Y/A vs success rate, whether it's being disproportionately affected by large plays, how to weigh situations (red zone, 3rd down, garbage time), and the repeatability of turnovers. We can't go anywhere with the PFF numbers without going back to the film, which kind of defeats the point. 

By "deep dive", I meant doing the film breakdown.

 

[Phragle]

I don't think anyone's suggesting they give it away for free. But they don't give this information even to their subscribers, even though they have it.

 
He means they can't give the public all the data for $30 per year and then charge teams $20,000 for the same data. They have to keep some things away from the public to justify the price teams pay.

 

[Shelterdog]

By "deep dive", I meant doing the film breakdown.

 
Well one thing you could do to test it is to try and observe whether it predicts changes in player performance that track what coaches do. For example, BB is a pretty smart dude.  If he consistently picked up untouted players who had pretty high PFF ratings then you might say to yourself "hmm, they're looking at the same thing as BB".  Or if part time players who had high PFF ratings per play often got more playing time, or maybe if low PFF ratings correlated strongly with players losing their jobs.

 

[lexrageorge]

 
 
Well how does anyone? Should we just abandon every type of film based evaluation because we don't know what the calls are? Fire the scouts and GMs and then use a roulette wheel instead?

Of course not.  But I have no idea the credentials of the guys making the judgments on players.  And I have no idea how these guys weight various factors that go into their grades.    
 
When the results differ dramatically from the eye test, which they inarguably do with Edelman, are we then wrong for questioning their methodology?  Or are we supposed to just trust that those random guys watching film always get the grades correct?  All I do know is that I haven't learned anything about Edelman, nor has anyone else, other than "he said this, she said that".  

 

[bowiac]

Well one thing you could do to test it is to try and observe whether it predicts changes in player performance that track what coaches do. For example, BB is a pretty smart dude.  If he consistently picked up untouted players who had pretty high PFF ratings then you might say to yourself "hmm, they're looking at the same thing as BB".  Or if part time players who had high PFF ratings per play often got more playing time, or maybe if low PFF ratings correlated strongly with players losing their jobs.

My sense is there'd be too much noise here for this to be useful. Locker room, cap, and age are all going to be founding variables, to say nothing of the fact that even BB isn't perfect.

 

[PedroKsBambino]

By "deep dive", I meant doing the film breakdown.

 
This is a great example of imposing a burden of proof.

 

[bowiac]

This is a great example of imposing a burden of proof.

I don't know what this means.

 

[Phragle]

Of course not.  But I have no idea the credentials of the guys making the judgments on players.  And I have no idea how these guys weight various factors that go into their grades.    
 
When the results differ dramatically from the eye test, which they inarguably do with Edelman, are we then wrong for questioning their methodology?  Or are we supposed to just trust that those random guys watching film always get the grades correct?  All I do know is that I haven't learned anything about Edelman, nor has anyone else, other than "he said this, she said that".  

 
I no longer think Edelman's rating is surprising. It's only a -0.7 which mean very little IMO, -1.0 of that is blocking which - while weighted equally by PFF - is to me is significantly less important than his receiving, and it doesn't factor in his special teams rating which is a big part of Edelman's skill set and impact on this game. Count his ST rating instead of his blocking and he finishes with a +1.7 - his second best game of the year. And he only had 89 yards and a TD on 10 targets. It's not like we're talking about a Megatron type of game here.
 

My sense is there'd be too much noise here for this to be useful. Locker room, cap, and age are all going to be founding variables, to say nothing of the fact that even BB isn't perfect.

 
Free agent contracts?

 

[Dogman]

I don't know what this means.

 
 
PFF's rankings can only be taken as reliable when their methodology is proven credible. So far, it hasn't so it's not.

 

[bowiac]

PFF's rankings can only be taken as reliable when their methodology is proven credible. So far, it hasn't so it's not.

Sure. That "deep dive" I was talking about was one way of showing credibility. From playing around with their blocking data, I think that's also credible, but that's worth further analysis.

 

[bowiac]

In the "it's all garbage" camp, while searching for an blog post statistically analyzing the reliability or lack thereof of PFF, I came across a pretty thorough (and recent) haters apiece over at Field Gulls. Quoting the most scorched earth parts, but the it's actually a good piece and worth reading.
 

Like so much of the modern world, Pro Football Focus seems harmless but is in fact pernicious. It does not steal your money. It steals your time. It does not lie to you directly. It mystifies with numbers and hides the truth beneath layers of confusion. It is not innovative or informative. It is a new and fashionable way to con goodhearted people who have, maybe, not been burned by con-men or who don't care or who simply wish to con the con, so long as the latter con is victimizing someone else.
 
Pro Football Focus is a tiny, tiny thing and not at all likely to destroy the world. I hope people read irony into that. But its success is something else entirely. It relies on that same admixture of plausibility, cognitive ease and inborn biases which prop up such dangerous nonsense as conspiracy theories, pseudoscience and obscurantism. We live in a time warped by plausibility. Any two-bit theory can be supported and every consensus scientific opinion can be undermined.

 

[Super Nomario]

 
I no longer think Edelman's rating is surprising. It's only a -0.7 which mean very little IMO, -1.0 of that is blocking which - while weighted equally by PFF - is to me is significantly less important than his receiving, and it doesn't factor in his special teams rating which is a big part of Edelman's skill set and impact on this game. Count his ST rating instead of his blocking and he finishes with a +1.7 - his second best game of the year. And he only had 89 yards and a TD on 10 targets. It's not like we're talking about a Megatron type of game here.

This is fair. My issue with Edelman's rating is the whole season, where they have him as the 5th-worst WR with significant PT, 12th-worst if you ignore blocking.
 

By "deep dive", I meant doing the film breakdown.

That's a lot of work, though. I think that's damning that if we disagree with a PFF rating, the options we have for further investigation are limited to "eye test" and "hours of film breakdown."

 

[PedroKsBambino]

That's a lot of work, though. I think that's damning that if we disagree with a PFF rating, the options we have for further investigation are limited to "eye test" and "hours of film breakdown."

 
This was my point on imposing a burden of proof on questioning the methodology; suggesting the above is a rhetorical game to try and protect ones own preferences, not an analytical approach to trying to understand something.

 

[Smiling Joe Hesketh]

In the "it's all garbage" camp, while searching for an blog post statistically analyzing the reliability or lack thereof of PFF, I came across a pretty thorough (and recent) haters apiece over at Field Gulls. Quoting the most scorched earth parts, but the it's actually a good piece and worth reading.
 

 
Damn, that's as thorough a takedown as could possibly be imagined. I don't know how anyone could read that and come away still believing that PFF has the slightest bit of positive value to add. "Truthiness" indeed.
 

 
Analyzing one player and his grade for one game was more than sufficient to prove how hopelessly flawed this method is. No standard is given for how a player should perform against a double team. No adjustment for quality of opponent is accounted for. And, after watching every snap played by Mebane multiple times and in slow motion, I cannot even guess how Mebane's -3.8 against the run was reached. Somehow Mebane rated as the worst regular defensive tackle in the NFL in week 8, when despite injured/out-of-position/rookie linebackers and a hobbled strong safety, the Panthers averaged 3.3 yards per attempt on 17 rushes between left guard, center and right guard. That same week, the Texans rushed for 149 yards and two touchdowns in 20 rushes at the Titans left guard, center and right guard.

 
They're simply making it all up. We ought to be better than that here and not regard their work as trustworthy. As noted in the article, Arian Foster rushed for 151 yards and 2 TDs on 20 attempts against Tennessee and received a -0.2 grade for the game. THAT. MAKES. NO. GODDAMN. SENSE.
 
And as PkB and others have alluded to, the effort itself is noble enough: let's try to get a deeper understanding of what's going on in the game. I'm all for that. But in the development of sabermetrics, there were many, many false starts and dead ends even when working from actual objective data: does anyone remember range factors? Range factors turned out to be a garbage stat, yet they were at least based on actual statistical analysis. PFF's numbers aren't based on anything but some dude's uninformed opinion. Of course they're going to be garbage, they're not working from an educated POV.

 

[Jnai]

 
Damn, that's as thorough a takedown as could possibly be imagined. I don't know how anyone could read that and come away still believing that PFF has the slightest bit of positive value to add. "Truthiness" indeed.
 
They're simply making it all up. We ought to be better than that here and not regard their work as trustworthy.

 
PFF does come off as rather bad here. I'm not trying to defend them. But is a company like BIS just making it all up? What really differentiates BIS from PFF, except for even less data availability?

 

[Smiling Joe Hesketh]

 
PFF does come off as rather bad here. I'm not trying to defend them. But is a company like BIS just making it all up? What really differentiates BIS from PFF, except for even less data availability?

 
Forgive my ignorance, I don't know what BIS is.
 
I'd like to continue to focus on trashing PFF, if that's alright with everyone here. 

 

[Jnai]

 
Forgive my ignorance, I don't know what BIS is.
 
I'd like to continue to focus on trashing PFF, if that's alright with everyone here. 

Baseball info solutions