Evaluating the Front Office

SMU_Sox

queer eye for the next pats guy
SoSH Member
Jul 20, 2009
9,293
Philly
Now do it for Vollmer. Or Gronk. Or Matt Light. Etc. I get that it's not BB drafting but no one knows anything. With all due respect.
We do know something though. We have studies that show that reaching is not as bad as it looks but still produces worse expected results.

Was Matt Light or Gronk a reach? I would use Mankins personally as an example of it working out. The point isn't that you can find examples of it working out, because it can work out, but that it is harder for it to work.

I can play the game too though but it doesn't really help us understand this:

Jordan Richardson, Tyquan Thornton, Cole Strange, Anfernee Jennings, Devin Asiasi, Dalton Keene, Joejuan Williams, Duke Dawson, Geneo Grissom. All were considered reaches. While a couple of these guys were decent role players that's all they were (or flat our busts). Individual examples really don't make sense when we can see studies that analyzed this.

82313

That's 6 years of analysis so sample size is good.

Quick edit: Jury is still out on Strange. He is technically an average to below average starter right now. When and if healthy.
 

Dogman

Yukon Cornelius
Moderator
SoSH Member
Mar 19, 2004
15,360
Missoula, MT
Working on getting Arif's registration pushed through and we can get his thoughts and answer questions on the front office, Pats draft, valuations, reaches, steals, etc.
 

Dogman

Yukon Cornelius
Moderator
SoSH Member
Mar 19, 2004
15,360
Missoula, MT
Arif is good to go and post at any time. This thread is full of good convo and thoughts. A logical starting point for discussion.
 

ArifMHasan

Member
Lifetime Member
SoSH Member
May 8, 2024
24
Sorry that I didn't see this until just right now, I was kind of buried in the weeds on a piece. But happy to be here and answer any questions! I'll check back on this in the morning and make sure I hit all of the important notes.

Edited to add that I want to add my background a bit here: I've been writing about the NFL for a bit over a decade now and have taken an interest in evidence-based approaches to our understanding of the NFL. This led me early on in my career to delve deep into analytics, but math is not the only source of evidence we have -- data comes in all forms, so I made sure to brush up on technical and schematic play throughout my career as well, with the intent of writing Xs and Os breakdowns as often as I did statistical analysis. Recently, it looks much more like my skills are still better suited for statistical breakdowns but it's always been more important to me to focus on process more than numeracy. I have a background in debate and speech and spent much of my high school and college years involved in academic debate while also coaching it, and that's where I like to live -- teasing out certainty from uncertainty.

I started out writing for an SB Nation site (The Daily Norseman) and moved from there to the Bleacher Report while taking on a wide variety of freelance work before taking over ownership of a Vikings blog called VikingsTerritory (which still exists but in a much different format now). From there, I was recruited to a Minnesota sports-specific publication called Zone Coverage and at the time also had my work appear in the Star Tribune, 1500 ESPN, the LA Times, Business Insider and a few other places. From there, I was recruited to the Athletic, where I worked for several years as the Vikings beat writer with an offseason focus on the NFL draft. After that, I took on a national role with Pro Football Network to cover the NFL at large before parting ways and starting my own independent publication, Wide Left -- which you can find at wideleft.football.

At Wide Left, I've written articles on the Vikings, analytics, the sportswriting industry (including a piece that many people took note of titled The Rise and Fall of the Draft Network), culture, race, politics and more. I'm more than happy to answer any questions on those topics in addition to questions about the Consensus Big Board.
 
Last edited:

SMU_Sox

queer eye for the next pats guy
SoSH Member
Jul 20, 2009
9,293
Philly
First of all huge thank you for doing this. You and Ted Nguyen are the reasons I subscribed to the Athletic years ago.

I’ll start a thread for the Q&A today and give people a week or so to start formally posting questions.

The below is not for you to respond to now but to generate some discussion for questions or topics that might be pertinent. Please, SOSHers, steal anything below as well as coming up with your own stuff!

One of the topics I’ve always wanted to know more about from you, and I’ll probably butcher writing it out in layman’s terms, is the science behind why younger players can physically and mentally (and the combination of both) learn quicker/easier (for lack of better words) vs older players. Some franchises, like the Browns, tend to draft much younger players on average.

I think the big question here (paging @EL Jeffe) that started discussing your work again this year was how big a reach Caedan Wallace was and what that means//so-what. I have linked your Wide Left article and I’ll do so again when I am back at my laptop.

Then there is how the Patriots did overall aspect of this. I’ll note that the Patriots had, according to your metrics, one of the best value picks in Drake Maye but still ended up with a negative ROI because the rest of their draft was filled with reaches. (I will edit this later to get the verbiage and details in). It felt like the front office was determined to get an OT and a WR on day 2 and that resulted in some reaches for positional need. When you are evaluating the Patriots new front office how are you considering their draft given the context of the team going into it and that it is a limited sample size?

@pokey_reese we might want to ask Arif if he has a take on if he has a criteria for if a QB is a hit or a miss since stuff like making a pro bowl and getting a high value second contract have too many exceptions.

I have been mulling over why there is a disconnect between QB film X/Twitter and QB analytics X/Twitter. Herbert, for example, has average stats like NY/A or EPA/dropback but is universally considered a top 5 or 10 QB. Meanwhile Brock Purdy puts up huge numbers in a stacked system and gets little respect from the Ruiz’s of the world. Jared Goff has had more professional success in every imaginable way than Trevor Lawrence but couldn’t crack the top 15 QBs drafted in the latest Athletic pod. Do you think there are possible holes in one side or the other?
 

EL Jeffe

Member
SoSH Member
Aug 30, 2006
1,369
Oh wow, this is really cool! Thank you to Arif for lending your time and expertise. And thank you to John and everyone else who played a part in getting you here!

I have a million questions and could pick your brain all day, but I'll start with 2 if that's okay?

  • How do draft analysts account for someone like Giovanni Manu when they create and maintain their big boards? As far as I can tell, he wasn't on the radar until his measurables came out at his 3/29 B.C. pro day and he ends up being a 4th round pick (#126) a month later. I imagine NFL teams had access to his Canadian all-22 film and he was scouted live during the season, but I don't know how readily available his film would be for the draft analysts out there. When there's such limited information about a player but his measurables are extremely rare at a scarce position, what's a reasonable way to rank that player? (It makes me wonder how draft analysts would have rated a Matt Cassel or even an Eric Swann back in the day before online draft analyses exploded in such a meteoric way.)
  • When an NFL "insider" like Tom Pelissero tweets out: "A name to remember in next week’s draft: Penn State OL Caedan Wallace. He had nine “30” visits with teams, including six in the last 10 days. A potential Day 2 pick" a week before the draft, and he's ranked in the 180s on the consensus big board, is that something draft analysts do (or should) take into account? Do they ask "what is the NFL seeing that I'm not seeing?" Or do they stick to their proverbial guns and trust their evaluations? I'm sure it varies on the analyst but I wonder how much attention is paid to that type of insider reporting. When there is that sort of disconnect between the consensus big board and how at least some NFL teams view a player, it makes me question how or why that disconnect happens. Maybe OL coaches fall in love with a guy based on private workouts that analysts don't have access to and it becomes an artificial rise? Or is it a case where too many analysts just mis-evaluated the player for whatever reason? I know some like Thorn and Brugler had him more a 4th rounder (making him a reach but not a massive reach), but outside of the Pelissero tweet, I'm not aware of any analysts thinking he could go as high as he did.
Thanks again!
 

RG33

Certain Class of Poster
SoSH Member
Nov 28, 2005
7,668
CA

ArifMHasan

Member
Lifetime Member
SoSH Member
May 8, 2024
24
Oh wow, this is really cool! Thank you to Arif for lending your time and expertise. And thank you to John and everyone else who played a part in getting you here!

I have a million questions and could pick your brain all day, but I'll start with 2 if that's okay?
  • How do draft analysts account for someone like Giovanni Manu when they create and maintain their big boards? As far as I can tell, he wasn't on the radar until his measurables came out at his 3/29 B.C. pro day and he ends up being a 4th round pick (#126) a month later. I imagine NFL teams had access to his Canadian all-22 film and he was scouted live during the season, but I don't know how readily available his film would be for the draft analysts out there. When there's such limited information about a player but his measurables are extremely rare at a scarce position, what's a reasonable way to rank that player? (It makes me wonder how draft analysts would have rated a Matt Cassel or even an Eric Swann back in the day before online draft analyses exploded in such a meteoric way.)
Thanks again!
I'll be honest, I don't think that the current draft media environment does a great job with international prospects. While I suspect the NFL slightly overvalues them (and had even before Jordan Mailata succeeded) I think that analysts undervalue them to a far greater degree. I think this will eventually change but for now I think it's just not having these players on their radar. Film may not be easily available for Canadian football players, which obviously matters, but there are workarounds for that. For other international players, it's a bit different. Some played football (like Moritz Boehringer and Babatunde Aiyegbusi) and some did not (Jordan Mailata and Jarryd Hayne). The ones that did not just have to be incorporated as if they are pure athletes (the same way that some basketball players or track athletes get valued). It obviously would be up to each analyst how they want to evaluate pure athletes, but they should at least be open to them. Someone like Dan Chisena was still valuable for the Vikings despite playing about 20 snaps of football in college because of his pure track speed — because there was a special teams role for him. They would probably want to build a checklist for these kinds of players ("Have we seen them engage with physicality or contact? How do they react to it," "Are their movement patterns linear or dynamic?" "Do they have to handle a ball, and how did they do so?" etc) and slot them based on the answers to those questions along with athletic testing data.

This undervaluation of international prospects (and specialists) is one reason why I produced two draft grades this year -- one that used consensus big board data and compared it to every pick and one that excluded picks for international players and specialists.

  • When an NFL "insider" like Tom Pelissero tweets out: "A name to remember in next week’s draft: Penn State OL Caedan Wallace. He had nine “30” visits with teams, including six in the last 10 days. A potential Day 2 pick" a week before the draft, and he's ranked in the 180s on the consensus big board, is that something draft analysts do (or should) take into account? Do they ask "what is the NFL seeing that I'm not seeing?" Or do they stick to their proverbial guns and trust their evaluations? I'm sure it varies on the analyst but I wonder how much attention is paid to that type of insider reporting. When there is that sort of disconnect between the consensus big board and how at least some NFL teams view a player, it makes me question how or why that disconnect happens. Maybe OL coaches fall in love with a guy based on private workouts that analysts don't have access to and it becomes an artificial rise? Or is it a case where too many analysts just mis-evaluated the player for whatever reason? I know some like Thorn and Brugler had him more a 4th rounder (making him a reach but not a massive reach), but outside of the Pelissero tweet, I'm not aware of any analysts thinking he could go as high as he did.
Thanks again!
The data suggests that visits provide signal to eventual draft position beyond mock draft and big board data. Another way to put it -- any model that attempts to predict draft position will improve with the addition of visit data (presumably when excluding locals, which these reports sometimes do not do -- if Caedan Wallace visited the New York Giants, New York Jets and the Philadelphia Eagles, I probably would not count that). Nine visits is quite a bit for a player ranked outside of the Top 150 on the consensus board. Joe Milton, ranked right next to Wallace on the Consensus Board but at a much more valuable position, only had two visits! Walter Rouse, ranked next to Wallace but at the same position, seemingly had zero visits.

What I have not seen tested is whether or not a model that attempts to predict player outcomes improves with the input of team visits. My intuition is that it would be given how useful draft position is to predicting player outcomes, with or without the board. Maybe I should look into it a bit more -- do reaches who have had a lot of team visits outperform reaches without those team visits? If I did that, I would also want to compare that to the data produced by those who produce anonymous scout quotes on teams, because those are also revealed (if more selective/biased) preferences. Bob McGinn's survey of NFL personnel, for example, did not regard Wallace as one of the top 12 tackles in the draft. At least one other scout told Mike Giardi (you know more about his credibility than I do) that the pick didn't make much sense.

As for what analysts should do, it really depends on their own internal process. One thing I really like about this project is seeing the different approaches different analysts take. Some purport to only use film analysis, at the exclusion of production data, athletic workout data, off-field concerns (including arrests), press appearances/interviews and rumors. Some will include all or some of those factors in their evaluations. Some of these boards have different goals -- some will want to be a lot closer to the NFL while others want to be independent of the NFL. Predicting draft order will matter a different amount to different analysts. Were I to have the time and energy to put together my own big board, I would use high-visit count as a re-check flag and go back to re-watch any prospects that ranked low on my board but had a number of team visits.

It is perhaps relevant that two sources that have a bit of a better bead on what NFL teams think -- Lance Zierlein and the person now running Scouts, Inc (Steve Muench) were the highest and third-highest on Wallace, ranking him 80th and 106th respectively. Others who have some insider access include Jeff Legwold, who was close, ranking him 111th while Field Yates ranked him 114th. That's one reason the Forecaster Board (composed of people with more insider access) ranked him 157th while the Evaluator Board ranked him 191st.

Generally speaking, reaches have a much, much worse track record -- even outside of the Top 100 -- than even picks and steals. Substantial reaches that were nevertheless ranked much higher by forecasters still carry that poor track record. But were I to reach for a player it would be for a player with a high variance score (Wallace's variance was about average for his rank), who was identified by NFL executives as a potential sleeper (e.g. Darius Robinson, Mike Sainristil or Quinyon Mitchell) and for a player who took a number of team visits. Wallace only meets one of those three criteria to my knowledge, which suggests to me that his Consensus Board ranking is a little low on him but not extraordinarily so.
 

Jinhocho

Moderator
Moderator
SoSH Member
Jul 31, 2001
10,446
Durham, NC

lexrageorge

Member
SoSH Member
Jul 31, 2007
20,236
That article is using the authors obviously incorrect interpretation of the Rooney rule to make the claim:

In New England, there are two explanations. Either Eliot Wolf had final say and the Patriots violated the Rooney Rule or Wolf didn’t have final say and ownership did. It cannot be neither.
There is nothing in the Rooney Rule that would have prevented Eliot Wolf from making the final calls on draft day.
 

EL Jeffe

Member
SoSH Member
Aug 30, 2006
1,369
I'll be honest, I don't think that the current draft media environment does a great job with international prospects.
This was really insightful and I love the call out about 30 Visit volume and how it may (or or may not!) correlate to draft pick expectancy. I'd love to know what draws teams to bring in certain players versus others. Certainly sometimes it's injury or even character check-ins. I wonder in the case of Wallace, it was teams wanting to LT footwork and drills? Maybe dig in further as to why it took until his 5th year to real lock in?

I'd also love to get your tale at some point on RAS scores and In-Game Athleticism (IGA) scores. It seems like RAS has the most correlation with tight ends, and the agility/shuttle numbers with OL. I'm wondering how much teams should factor 40 time versus maybe in-game play speed data. Keon Coleman was obviously Exhibit A with that (poor 40, strong in-game data), but I also wonder if/how teams rationalize combine/pro day RAS numbers, versus IGA data, versus film evaluation of a player's athleticism. Are the in-game metrics (GPS mph, acceleration/deceleration times, change of direction times, separation distance, etc.) at a point where teams and evaluators can make good use of them? Or are we still a bit aways from really being able to leverage that data in a productive and meaningful way? Like, if you were an NFL decision maker, how would you go about quantifying a player's athletic ability and how much weight would you put into it?
 

ArifMHasan

Member
Lifetime Member
SoSH Member
May 8, 2024
24
I'd also love to get your tale at some point on RAS scores and In-Game Athleticism (IGA) scores. It seems like RAS has the most correlation with tight ends, and the agility/shuttle numbers with OL. I'm wondering how much teams should factor 40 time versus maybe in-game play speed data. Keon Coleman was obviously Exhibit A with that (poor 40, strong in-game data), but I also wonder if/how teams rationalize combine/pro day RAS numbers, versus IGA data, versus film evaluation of a player's athleticism. Are the in-game metrics (GPS mph, acceleration/deceleration times, change of direction times, separation distance, etc.) at a point where teams and evaluators can make good use of them? Or are we still a bit aways from really being able to leverage that data in a productive and meaningful way? Like, if you were an NFL decision maker, how would you go about quantifying a player's athletic ability and how much weight would you put into it?
Sorry, I didn't reply to this earlier. I don't mean to be dismissive when I cop out with a link behind a paywall, but here's a link behind a paywall: Athletic Scores and Why I Don't Use RAS. As for your bit on the use of in-game data, I'll refer you to my piece on why the Combine Might Be Dying.

For what it's worth, athleticism is dependent on position and role. I care more about athleticism at edge rusher and for smaller wide receivers than I do for nose tackles and bigger receivers. Generalized athleticism seems to be pretty good for evaluating tight ends, while position-specific athleticism matters much more for offensive linemen, with an emphasis on the short shuttle. For me, athleticism would be a knock-out factor at edge rusher and a significant factor on the offensive line. I might move a player up a full round over the traits-based report at edge rusher based on athletic ability.
 

Mystic Merlin

Member
SoSH Member
Sep 21, 2007
49,575
Hartford, CT
Dov Kleinman - and I say this with all due respect - might be the dumbest reporter in the history of the NFL. But your point about Florio is still valid.
Hear hear. I have a physical reaction when a Dov tweet pops into my field of vision, so I couldn’t agree more. I actually sort of regret that I scoffed at Gregg Easterbrook, and even Pete Prisco, for so many years because this isn’t a better life.

BTW, good to see you pop in - was wondering how Jason Kelce’s retirement was affecting you and the rest of the Philly faithful. He and Brian Dawkins are who I personally identify as the archetypal Iggles (granted, pre-98/99 I admittedly have almost no NFL memories).
 
Oct 12, 2023
1,273
Dov Kleinman - and I say this with all due respect - might be the dumbest reporter in the history of the NFL. But your point about Florio is still valid.
Kleiman had the same schtick 15 years ago before he was getting paid for it. He used to post on another forum under the name Billy Spikes. He would just obsessively follow Twitter and immediately post any breaking news to the forum. On cutdown day or day 1 of free agency half the forum would be his posts and he’d get salty if anyone beat him to it.

Hes just an aggregator/reposter. He has no connections or insight. He trawls social media and repackages posts/tweets as if he’s breaking news or has a scoop. Also has a really creepy fixation on Jennifer Lawrence.

Florio at least, ostensibly, has one or two connections in the industry by now and is just a hot take artist who leveraged his “rumors” (BS he made up) into a career.

They’re both good examples of people who made an entire career out of having no real information of their own though I think Florio actually does get a tiny bit more real insight now than he did 25 years ago when he ran his rumor site.
 

Rick Burlesons Yam Bag

Internet Cowboy, Turbo Accelerator, tOSU Denier
Lifetime Member
SoSH Member
Hear hear. I have a physical reaction when a Dov tweet pops into my field of vision, so I couldn’t agree more. I actually sort of regret that I scoffed at Gregg Easterbrook, and even Pete Prisco, for so many years because this isn’t a better life.

BTW, good to see you pop in - was wondering how Jason Kelce’s retirement was affecting you and the rest of the Philly faithful. He and Brian Dawkins are who I personally identify as the archetypal Iggles (granted, pre-98/99 I admittedly have almost no NFL memories).
I love Jason Kelce. He got a little overexposed over the last 2 seasons, but I would rather that than the fate that awaited the amazing Dan Koppen (who should have taken a huge shit on the podium at the Brady roast and just said "I'm like 9/11.....you should never forget me" or something comparably roasty) at the end of his career.

I also love Jason Kelce and am glad that he quit when he did. In truth....he may have played one season too long, he was tired in the last 4-5 games and you could see it. I don't want him to be in a wheelchair or unable to recognize Andy Reid 15 years from now. On the field, the line seems to be in good shape, but let's see.