Bleedred said:
Volin's article today claims that a Physicist replicated the Exponent methodology and confirmed its accuracy, and found a significant flaw in the AEI report (although didn't say what it was). Not linking to him b/c I don't think he deserves it.
This is a little humorous. In summary, Volin and the biostatistician (DeSarno) are most likely wrong - and AEI is right.
This is a very nerdy story. The dispute is over a very obscure corner of statistics, and it's at the intersection of two fields- economics (where people at AEI usually come from) and biology/psychology. The problem is those fields don't care to educate each other about statistics, so their approaches are different and their words confuse each other.
Volin's article says: (emphasis mine)
The AEI scientists wrote that Ted Wells and Exponent used an unorthodox statistical procedure at odds with the methodology the report describes. On a whim, DeSarno decided to check AEIs work. He plugged the football PSI data into a standard statistical analysis method a mixed model, repeated measures analysis of variance, or ANOVA.
On his first try, DeSarno replicated Exponents results exactly.
I totally expected to see that I couldnt replicate the Wells Report, either, DeSarno said. But I was kind of surprised that once I ran my standard analysis method, it matched Wells exactly.
The first model I tried, based on the usual model I use for this data, worked immediately. So I dont know why [AEI] didnt use that model.
To a biostatistician, the ANOVA approach is standard. But to an economist, it's "unorthodox".
As this article points out
http://andrewgelman.com/2005/01/25/why_i_dont_use/
that standard ANOVA model is ambiguous.
An economist would never use a mixed-effects ANOVA, they would use a particular specification of a linear regression. As the AEI report did.
Now, I haven't dug fully into the exact details of AEI's argument, but their point is mostly boring. It's basically snark from economists saying that other fields don't understand statistics as well as they do. From AEI:
"Although the text that prefaces table A-2 indicates that the values are 'adjusted for other effects,' implying the authors performed a multiple regression, they seem to be the result of simply including no other explanatory variables"
[Etc, etc.]
In general, ANOVA estimation is difficult, because it can require very complicated specification of error terms, and that requires an expert whenever you go outside a small set of cookie cutter examples. You can't just run "your standard analysis" and expect to get the same results as others. See my first link above, and look at the manuscript linked on that page for examples as to how ANOVA specification can go wrong (his Figure 1 shows how commonly errors can be made in SPSS). Not only that, terms like "mixed model" are often ambiguous.
The concrete, nonambiguous way to specify an analysis like this, regardless of which field you come from, is to give the equation you are fitting, and say how you fit it. That is what AEI does. DeSarno, on the other hand, uses the ambiguous terms.
Therefore, because they reason from the model equations, it is likely AEI is right and DeSarno is wrong. (I'm not inclined to make that statement ironclad by spending an hour or two to find DeSarno's email, figure out what he did, and vet the AEI regressions fully.)
But ultimately for Brady and the NFLPA, conflicts between two different fields over statistical conventions are irrelevant. Here's the only thing that matters about the Wells report, and DeSarno and AEI agree:
Now, just because DeSarno replicated Exponents results does not mean that the conclusions of the Wells Report are accurate.
DeSarno pointed out that the data set is incomplete and flawed what weve been saying for four months about referee Walt Anderson not recording the pregame data and all of the different gauge-switching scenarios and that Wells cherry-picked results to fit the Patriots are guilty conclusion.
I still dont place any faith in [Wellss] conclusions, DeSarno said.
Exactly. Garbage in, garbage out.