Super Nomario said:
I don't think we're too far apart on this. It's like the clutch hitting conundrum - maybe some have a difference in skill, but at the level we're talking about we've already selected out the vast mass of difference in skill and any remaining difference is going to be small, and the sample sizes are such that we might not ever have enough data to make conclusions.
I would say that there may be a difference in skill, and we may or may not be able to measure it given the sample sizes here (a real question, but one that is equally likely to hide real skill as hide the absence of skill). Whether we've designed the right test is different from whether the underlying skill exists.
I don't see how one can attempt to estimate the impact of a skill if they feel we can't yet measure it. The approach you describe above is what got all sorts of people egg on their face about (for example) catcher pitch-framing skills in the mid-2000s....people said it didn't exist, and then we got better and different data and learned it not only exists, but is impactful. My point is only that we need to be cognizant of the difference between what we can demonstrate based on a few studies and what might be demonstratable.
The related question (which I think is legit, and is different than where this started) is how much confidence we should have about a particular NFL executive based on drafting record, since we currently seem to have limited ability to assess it. That's somewhat different from saying it likely doesn't matter, though.
Stitch01 said:
I wouldn't and didn't say the data proves decisively there is no skill in drafting. I said the data supports the idea that there is very little skill in drafting, and I stand by that. That's why I said if you want to argue its 10% skill I wouldn't argue much, I have an opinion based on the data and literature Ive read but its certainly not definitive.
10% skill <> 10% edge. Being 10% better at drafting would be material. Being marginally better at something that's 10% skill, meh, its going to be swamped by luck and I doubt we can distinguish.
I personally don't think drafting is even 10% skill, so Im fine with calling it a crapshoot, but YMMV.
Im not saying I have a definitive answer here. I have an opinion based on the available data and literature Ive seen and read. Research showing more evidence there is skill to drafting would certainly change my mind.
The data cited in this thread simply does not say what you suggest---I don't know if you are thinking about other studies I have not seen and haven't been cited here, or you're missing what the authors of these are stating as limitations, or what. But as a couple of us have noted, your conclusion is going beyond the data here.
SuperNomario's point that we are looking at a very small subset of GMs who have been filtered also is relevant to this---if there truly were so little skill in drafting than it would be the case that anyone could do it almost equally. Is that really your view? I suspect not. If not, then the hypotheses must be that the NFL very effectively selects a group that is very closely bunched in possessing the skill (in spite of radically different backgrounds for owners and selection processes). While this is conceivable, I think chewing on whether that hypotheses makes sense is worthwhile in evaluating this one. Or, perhaps, that the NFL selects some distribution of people with the skill and we just have trouble separating out the 'drafting' ability versus the 'coaching and developing' ability, on top of the natural variance in how 21 year olds will develop at most anything.
I do not know how much skill is involved, or what the difference in skill amongst GMs might be...and just think we should be clear that the data doesn't suggest any of us do.
If you are asking my best guess---I'd guess that there are most likely some differences, and it would surprise me if there are not outliers within that group since that is the case in most performance-related things. But I'm just speculating; it could be teams just aren't any different at it. I do think a significant complicating variable here (beyond sample size, as noted above, and selection bias considerations) is that players are drafted for a scheme/roster situation and thus there is nothing close to equal opportunity for picks even if they are right next to each other in the draft.