Oh wow, this is really cool! Thank you to Arif for lending your time and expertise. And thank you to John and everyone else who played a part in getting you here!
I have a million questions and could pick your brain all day, but I'll start with 2 if that's okay?
- How do draft analysts account for someone like Giovanni Manu when they create and maintain their big boards? As far as I can tell, he wasn't on the radar until his measurables came out at his 3/29 B.C. pro day and he ends up being a 4th round pick (#126) a month later. I imagine NFL teams had access to his Canadian all-22 film and he was scouted live during the season, but I don't know how readily available his film would be for the draft analysts out there. When there's such limited information about a player but his measurables are extremely rare at a scarce position, what's a reasonable way to rank that player? (It makes me wonder how draft analysts would have rated a Matt Cassel or even an Eric Swann back in the day before online draft analyses exploded in such a meteoric way.)
Thanks again!
I'll be honest, I don't think that the current draft media environment does a great job with international prospects. While I suspect the NFL slightly overvalues them (and had even before Jordan Mailata succeeded) I think that analysts undervalue them to a far greater degree. I think this will eventually change but for now I think it's just not having these players on their radar. Film may not be easily available for Canadian football players, which obviously matters, but there are workarounds for that. For other international players, it's a bit different. Some played football (like Moritz Boehringer and Babatunde Aiyegbusi) and some did not (Jordan Mailata and Jarryd Hayne). The ones that did not just have to be incorporated as if they are pure athletes (the same way that some basketball players or track athletes get valued). It obviously would be up to each analyst how they want to evaluate pure athletes, but they should at least be open to them. Someone like Dan Chisena was still valuable for the Vikings despite playing about 20 snaps of football in college because of his pure track speed — because there was a special teams role for him. They would probably want to build a checklist for these kinds of players ("Have we seen them engage with physicality or contact? How do they react to it," "Are their movement patterns linear or dynamic?" "Do they have to handle a ball, and how did they do so?" etc) and slot them based on the answers to those questions along with athletic testing data.
This undervaluation of international prospects (and specialists) is one reason why I produced two draft grades this year -- one that used consensus big board data and compared it to every pick and one that excluded picks for international players and specialists.
- When an NFL "insider" like Tom Pelissero tweets out: "A name to remember in next week’s draft: Penn State OL Caedan Wallace. He had nine “30” visits with teams, including six in the last 10 days. A potential Day 2 pick" a week before the draft, and he's ranked in the 180s on the consensus big board, is that something draft analysts do (or should) take into account? Do they ask "what is the NFL seeing that I'm not seeing?" Or do they stick to their proverbial guns and trust their evaluations? I'm sure it varies on the analyst but I wonder how much attention is paid to that type of insider reporting. When there is that sort of disconnect between the consensus big board and how at least some NFL teams view a player, it makes me question how or why that disconnect happens. Maybe OL coaches fall in love with a guy based on private workouts that analysts don't have access to and it becomes an artificial rise? Or is it a case where too many analysts just mis-evaluated the player for whatever reason? I know some like Thorn and Brugler had him more a 4th rounder (making him a reach but not a massive reach), but outside of the Pelissero tweet, I'm not aware of any analysts thinking he could go as high as he did.
Thanks again!
The data suggests that visits provide signal to eventual draft position beyond mock draft and big board data. Another way to put it -- any model that attempts to predict draft position will improve with the addition of visit data (presumably when excluding locals, which these reports sometimes do not do -- if Caedan Wallace visited the New York Giants, New York Jets and the Philadelphia Eagles, I probably would not count that). Nine visits is quite a bit for a player ranked outside of the Top 150 on the consensus board. Joe Milton, ranked right next to Wallace on the Consensus Board but at a much more valuable position, only had two visits! Walter Rouse, ranked next to Wallace but at the same position, seemingly had zero visits.
What I have not seen tested is whether or not a model that attempts to predict player outcomes improves with the input of team visits. My intuition is that it would be given how useful draft position is to predicting player outcomes, with or without the board. Maybe I should look into it a bit more -- do reaches who have had a lot of team visits outperform reaches without those team visits? If I did that, I would also want to compare that to the data produced by those who produce anonymous scout quotes on teams, because those are also revealed (if more selective/biased) preferences. Bob McGinn's survey of NFL personnel, for example,
did not regard Wallace as one of the top 12 tackles in the draft. At least one other scout told Mike Giardi (you know more about his credibility than I do)
that the pick didn't make much sense.
As for what analysts should do, it really depends on their own internal process. One thing I really like about this project is seeing the different approaches different analysts take. Some purport to
only use film analysis, at the exclusion of production data, athletic workout data, off-field concerns (including arrests), press appearances/interviews and rumors. Some will include all or some of those factors in their evaluations. Some of these boards have different goals -- some will want to be a lot closer to the NFL while others want to be independent of the NFL. Predicting draft order will matter a different amount to different analysts. Were I to have the time and energy to put together my own big board, I would use high-visit count as a re-check flag and go back to re-watch any prospects that ranked low on my board but had a number of team visits.
It is perhaps relevant that two sources that have a bit of a better bead on what NFL teams think -- Lance Zierlein and the person now running Scouts, Inc (Steve Muench) were the highest and third-highest on Wallace, ranking him 80th and 106th respectively. Others who have some insider access include Jeff Legwold, who was close, ranking him 111th while Field Yates ranked him 114th. That's one reason the Forecaster Board (composed of people with more insider access) ranked him 157th while the Evaluator Board ranked him 191st.
Generally speaking, reaches have a much, much worse track record -- even outside of the Top 100 -- than even picks and steals. Substantial reaches that were nevertheless ranked much higher by forecasters still carry that poor track record. But were I to reach for a player it would be for a player with a high variance score (Wallace's variance was about average for his rank), who was identified by NFL executives as a potential sleeper (e.g.
Darius Robinson,
Mike Sainristil or
Quinyon Mitchell) and for a player who took a number of team visits. Wallace only meets one of those three criteria to my knowledge, which suggests to me that his Consensus Board ranking is a little low on him but not extraordinarily so.
