Kessler's point from the opening, on the standards for PED testing - chain of custody, valid methods, etc - would carry the day in a sane world.
#DFG: Canceling the Noise
- Thread starter nattysez
- Start date
The easiest answer is that Tom probably has no recollection of ever saying that. Which again depending on each of our own biases will either be incriminating and way too easy of an excuse OR perfectly reasonable and understanding for a guy that gives many interviews and who pretty clearly did not see ball pressure as a big fucking dealPeak Oil Can Boyd said:
For those of you that know how to make the memes, this needs to go viral. A big ass picture of Vincent and the NFL sheild with the quote "I didn't include science, no sir"ragnarok725 said:
( . ) ( . ) and (_!_) said:
I actually think it is a frame of reference thing. Tom Brady thinks of footballs in more elementary terms: 'soft', 'easy to grip,' 'no nubs.' He doesn't think about the feel of the ball as a relation to PSI or air pressure. Not unsimilar to how a sommelier may think of a wine as heavy in dark fruit, cherry, oak, and tannin but a restaurant patron may see it as 'big' and 'bold.' Neither is wrong. But when framing the question in terms of air pressure, it doesn't register with him because he literally has never thought about footballs that way.
“[W]hen Gronk scores – it was like his eighth touchdown of the year – he spikes the ball and he deflates the ball,” Brady said in November 2011. “I love that, because I like the deflated ball. But I feel bad for that football, because he puts everything he can into those spikes.”ScubaSteveAvery said:
I mean, "I like the deflated ball" is pretty unambiguous.
I still don't really know if this is a big deal, I just think he unnecessarily dodged this question in a way that is, IMO, troubling.
Edit: The NFL is clearly trying to get him to say "I like the footballs deflated as much as possible" so they can infer he likes them under the legal limit. So I'm fine with the dodging, I just don't get why he would say he never thought about it before the Jets game.
Peak Oil Can Boyd said:
If you can't tell that Brady is giving a somewhat humorous answer to a question about Gronk and how he spikes the ball, I don't know what to tell you.
Do you also think that Brady really feels bad for the football?
Because a fucking joking answer isn't really thinking about it.
( . ) ( . ) and (_!_) said:
The NFL went from direct evidence (PSI readings) in late January to claiming the evidence is the lack of evidence (cell phone).
For the sake of argument, regardless of whether you believe Brady, McNally, and JJ are innocent as they claim, how would they prove that to the NFL? It seems every question and piece of evidence is asking them to prove a negative.
Edit: Don't mean you specifically in the above question, just in general
Brady forgot an answer he gave to an interview question 3 1/2 years ago = he definitely conspired to deflate footballs in the AFCG
Peak Oil Can Boyd said:
I'm not disagreeing that it is unambiguous. Just that when they frame the question from a scientific perspective it doesn't register with him. "Deflate" is not a scientific term. This doesn't make him guilty or innocent, just that he's not wrong when he said he doesn't think about air pressure. He may not and still tell the equipment guys to make them soft as possible or to deflate them because they are too hard.
FBI Agent: Now when I say "Hello Mr. Thompson" and press down on your foot, you smile and nod.
Homer: No problem.
Homer: No problem.
I can't get over this point. Wells flat-out tells Brady that he doesn't require his cell phone. And that "I did not tell Mr. Brady at any time that he would be subject to punishment for not turning over the documents. I did not say anything like that."
At no point does the CBA or any league rule explain that a player will be subject to punishment for not turning over his cell phone. The only precedent for any penalty whatsoever wasn't a suspension, but a fine - given to Favre in 2010.
In the bounty gate ruling, Tagliabue said, “In my forty years of association with the NFL, I am aware of many instances of denials in disciplinary proceedings that proved to be false, but I cannot recall any suspension for such fabrication. This is no evidence of a record of past suspensions based purely on obstructing a League investigation.”
So no warning from Wells to Brady. In fact, the opposite. Nothing but reassurance that he didn't need Brady's phone, and no hint that not turning over his phone could lead to penalties. Nothing in the CBA. Nothing in the rulebook. The only precedent being a $50k fine.
Yet Brady gets a 4 game suspension and loss of nearly $2 million?
At no point does the CBA or any league rule explain that a player will be subject to punishment for not turning over his cell phone. The only precedent for any penalty whatsoever wasn't a suspension, but a fine - given to Favre in 2010.
In the bounty gate ruling, Tagliabue said, “In my forty years of association with the NFL, I am aware of many instances of denials in disciplinary proceedings that proved to be false, but I cannot recall any suspension for such fabrication. This is no evidence of a record of past suspensions based purely on obstructing a League investigation.”
So no warning from Wells to Brady. In fact, the opposite. Nothing but reassurance that he didn't need Brady's phone, and no hint that not turning over his phone could lead to penalties. Nothing in the CBA. Nothing in the rulebook. The only precedent being a $50k fine.
Yet Brady gets a 4 game suspension and loss of nearly $2 million?
https://twitter.com/TomPelissero/status/628673864046899200RedOctober3829 said:
link to tweet
I thought the NFL was aware of the allegations before the game?
Haven't read the whole transcript yet - where does this point get addressed - when was the issue first raised and who was aware?
General takeaway from Brady Appeal remains same: what a colossal waste of time since NFL has no idea if footballs were deflated
— Dan Wetzel (@DanWetzel) August 5, 2015
— Dan Wetzel (@DanWetzel) August 5, 2015
- I thought Brady's explanation for why he needed to talk to Jastremski so much after the AFCCG made sense: for the Super Bowl, they need to prepare ~100 balls rather than the typical 12-24, so he wanted to make sure Jastremski - who wasn't in his role in 2011, the last time the Pats played in the Super Bowl - stayed focused and knew what he was doing that week despite the deflategate distractions. I hadn't heard that tidbit before, and it provided a logical innocent explanation for what seemed like a fishy post-scandal pattern of communication
- Goodell asked if you have to change your phone number when you get a new phone
- Nash asks Brady about three texts to Jastremski on 2/7, but never asks him what those messages were about. It's like, he's just bringing them up to cast aspersions on Brady without getting specific.
- The lawyer (Reisner) that cross-exams the NFLPA's expert mostly seems focused on trying to make him look stupid because he can't recall exactly where in the Exponent report certain quotes and analyses are made. He does not really dispute his point (which is that the significance test didn't incorporate timing), he just throws a smokescreen over the whole thing
- The NFL clearly did tell the NFLPA / Brady that he had four hours (contradicting their contradiction of a Schefter Tweet), though magnanimously Goodell was willing to grant another hour and then be flexible at the end of that.
- Kessler was not allowed to ask about the delegation of authority from Goodell to Vincent
- Kessler starts right in on Ted Wells with what kind of comments Pash made on drafts of the report. Wells claims attorney-client privilege.
- It seems like once Brady declined to turn over texts, Wells decided everything he had to say was BS.
- Kessler volunteers to make the transcript public that day, but the NFL declines.
- The Exponent guy says the Colts balls weren't used as controls, but they were explicitly used to set temperature parameters on the transient test assuming the logo gauge was used pre-game. Kessler almost gets to this, but not quite. Oddly enough, Reisner goes over this passage in cross-cross.
- Exponent's statistician says that a statistical significance of p=.05 isn't a necessary standard here because they're talking "more probably than not," not a more stringent standard. He also poo-pooh's Synders (NFLPA's statistician) significance analysis by saying they're different variabilities even though they had determined the difference in variabilities was not statistically significant. He brings up the variability repeatedly in fact. Then Marlow brings it up.
- I think Kessler / NFLPA made a mistake bringing up a lot of things like the wet / dry differences that Exponent tested and don't make much of a difference. There are two or three serious problems with Exponent's work and he should have hammered those. Bringing up a bunch of different things didn't serve to create a boatload of doubt; it just made it seem like they were throwing mud to see what stuck. The Snyder thing was good and he did a good job hammering the Exponent stat guy on cross; I wish he'd stuck to that sort of thing.
- A lot of this report is lawyers trying to make scientists and mathematicians give black or white answers to non-black-and-white questions.
- Marlow said he studied the post-game data and wanted to put it in the report but was overruled. He says it would have made the variability of the Pats' footballs look bad (AGAIN, WHY ARE THEY HARPING ON VARIABILITY), but an analysis the raw numbers would have helped the Pats (they indicated the Pats the Pats played the second-half with over-inflated footballs because the NFL didn't understand the ideal gas law, and by basically the same amount that they were under in the first half, suggesting no tampering - but with the caveat that we're dealing with uncertain starting points and two sample sizes of four).
The principal from Billy Madison should've been in the appeal hearing
"Mr. Goodell, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul."
Would make a fun GIF
"Mr. Goodell, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul."
Would make a fun GIF
To me, Wells' comments are kind of a red herring. As an "independent investigator" is he even technically able to speak with any kind of authority about what punishments Brady may or may not face?ivanvamp said:
The overall point about precedence/notice stands and Brady's phonewalling(TM) clearly affected Wells' conclusions and Goodell's decisions, but I dont think there is or even should be any kind of burden on an investigator to deliver punishment warnings.
Of course, we all know at this point what "independent" really meant but unfortunately the Wells Report is not on trial here. TW's entire part in this is undoubtedly frustrating to no end, and it would be great if the NFL's contributions to the report end up burning them.
From Volin's tweet:
So the NFL crooks stealing game balls were in action for the game against the Ravens, and yet the Pats get the blame?
So the NFL crooks stealing game balls were in action for the game against the Ravens, and yet the Pats get the blame?
Not to nitpick on less relevant parts of the investigation but WTF:
This is just flat-out, factually incorrect. Significant findings in small sample sizes that are underpowered are, almost by definition, inaccurate and generally overestimated.
This is just flat-out, factually incorrect. Significant findings in small sample sizes that are underpowered are, almost by definition, inaccurate and generally overestimated.
I agree, a lot of the testimony focused on red herrings in the exponent report that didn't get at its major flaws.Super Nomario said:
As an aside, I'm still surprised at how everyone simply accepted a linear mixed effects model, where the distribution of PSI in the 12 Pats balls and 4 colts balls did not meet the assumptions of a linear mixed model in the first place.
I was just debating posting this. I've read that paragraph 5 times now and have no idea what he's saying.PhilPlantier said:
For the umpteenth time, reason #whatever why BB is a genius. Never would have made the "read the rule book" comment in a million years. Innocuous? Yes. Made in jest? Surely. But here we are.DrewDawg said:
Clesse expand on this, if you could and have time?EricFeczko said:
I've seen claims about the problems with and disagreements about the appropriateness of statistical models used. I have not seen a good explanation, though, and you have shown yourself in the past to have the knowledge and patience to make one, so I think you are the man for the job.
If you would be so kind.
EricFeczko said:
One of the things that's interesting is that the NFL presented its case second. I believe normally in trials, the defendant goes second, no? Presumably Kessler could have asked Snyder, the stats guy, to comment on this if the Exponent dude had testified first, but obviously he doesn't have the statistical background to get into this sort of thing.
Holy Shit that is ridiculous.EricFeczko said:
I assume exponent's "model" was "keep fiddling with things until we find the magic p<0.05."
To be fair, there's a number of SoSHers that could expand on this: Crystalline, kickerinthered(sic), Iayork. Hopefully, they'll correct the mistakes I make here.There is no Rev said:Clesse expand on this, if you could and have time?
I've seen claims about the problems with and disagreements about the appropriateness of statistical models used. I have not seen a good explanation, though, and you have shown yourself in the past to have the knowledge and patience to make one, so I think you are the man for the job.
If you would be so kind.
I think I posted a very long post on this problem in the locked thread somewhere. I'll find it and repost the link here. Just let me know if you want me to post it in full (it is long).
http://sonsofsamhorn.net/topic/88102-nfls-wells-report-the-fix-is-in/page-353#entry6032879
Among other things, a linear-mixed effects model assumes that data are distributed normally. A simple examination of the PSI in the 12 pats balls, 4 colts balls (assuming variance is different between the population of pats and the population of colts balls) or examination of the PSI across all 16 (assuming that pats and colts balls belong to the same population of infalted footballs) will show that it isn't. A simpler and more appropriate analysis to pursue, which only assumes that the measures made are independent of one another, would be a permutation test.
EDIT: Another thing that really gets me is the intentional switching of datapoints to make them look cleaner. You cannot decide that one rating should've been switched simply because it looks like it could have. The data are what the data are and any such switching would be considered grounds for retraction in a published peer-reviewed paper.
Eddie Jurak said:
I just flipped a coin three times, and got heads every time. So, I guess according to the NFL that means every time I flip I coin I will get heads from now on.
PhilPlantier said:
I think he says that with both gauges, the Pats balls were found non-compliant, whereas on one gauge all Colts balls were found in compliance.
EricFeczko said:
I am not a statistician, but I believe what the Exponent guy is referring to is the "power approach paradox". Here is a snippet of the description in Wikipedia:
"all post-hoc power analyses suffer from what is called the "power approach paradox" (PAP), in which a study with a null result is thought to show MORE evidence that the null hypothesis is actually true when the p-value is smaller, since the apparent power to detect an actual effect would be higher. In fact, a smaller p-value is properly understood to make the null hypothesis LESS likely to be true." Link
hoothehoo said:
Per Peter King's story of the pre-game ritual of the officials, the kicking balls are new out of the box something like 2 hours before the game, but each team gets to run off with a set and scrub them up for a short period of time before the game starts. See the Wells Report appendix about the NFL guys stealing balls. Ghost was upset because the K ball his guys had worked up the most was being taken out of the game after very limited use.
Harry Hooper said:
Aha. I've been misunderstanding and misremembering. Thank you for clarifying for me.
Of course, it still doesn't make sense that the Ravens would think the Patriots had anything to do with new kicking balls.
Then again. Nothing about any of this makes sense.
If this is what the statistician was attempting to refer to, then it is irrelevant here. A priori, theoretical, simulations of a comparison of means via ANOVAs or t-tests will show that with small sample sizes, when one is at five to ten percent power, a significant effect overestimates the true effect size by a factor of 10 or so. Linear mixed models are slightly different but will show similar effects.slamminsammya said:
In other words, underpowered analyses don't really tell you anything, and can show significant effects that aren't really there.
EDIT: I'm not disagreeing with the effect, by the way. The wiki-page is right as is the statistician; a statistical cut-off of 0.05 percent is arbitrary.
hoothehoo said:
It's like teams against Red Auerbach and the Celtics. No hot water in the locker room, that was Red's doing. No air conditioning in the Garden, Red again. Plane has to circle in the fog at Logan before landing, Red bribed the airline. People frustrated by losing let all sorts of crap into their heads.
Eh. Screw it. Here's both of my prior posts on the problem with using a linear mixed model approach for the stats. As I've said before, I don't think anyone really understood the statistical analysis, and I'm not confident that it played much of a role in Goodell's decision. Blame Rev for the posts, if any mod wants to delete it go ahead.
Ok. I've been waiting awhile before stepping in, and I probably shouldn't. But something does need to be stated about Exponent's poor statistical analysis of the data given.
There are two primary problems with the statistical analysis in Exponent's report. The first has been mentioned already; using parametric statistics on non-normal distributions leads to inaccurate tests of statistical significance. In other words, the model exponent used to answer the question, "Did the Patriots' balls drop in PSI more than the Colts?" assumes that the data are distributed like major league career OBP numbers:
Note how the most frequent values in the range of career OBP values fall in the center of the distribution. Also note that the dropoff in frequency is symmetrical on both sides. This is often referred to as a "bell-curve" distribution because it is shaped like a bell. The model used for analysis uses the standard deviation (technically the variance, but since standard deviation is a more familiar term that people might understand, I'm using that here) to assess whether the drop in PSI is greater for the Pats' balls, then what would be expected due to random chance. The problem is that the PSI-drop data looks like this:
This is a common problem when dealing with small sample sizes; distributions of data look weird and the standard deviation is inaccurate. As a result, the significance assessed from the standard deviation is also inaccurate. As a side note, I've looked at this small data in multiple ways and it doesn't really matter how you look at it, the distributions always look non-bell curvy.
One simple solution is to use a better model. The question is, how do we construct a better model? If the hypothesis is that the drop in PSI is greater for the pats than the colts, then the null hypothesis is that the drop in PSI is similar between the pats and the colts. Under this null assumption, if we mixed up the two sets of balls, there should be no difference between the average PSI drop. Therefore, we can construct a distribution of possible results by mixing up the groups in lots of different ways and calculating the difference of means. If our observed value when the groups are preserved is far from the center of the distribution, the probability that the null assumption is correct (a.k.a. the "p value") decreases. This is known as a permutation test.
For those in the know, I ran 10000 permutations, so the "p value" here is extremely accurate.
Here, we actually have two permutation tests, one for piroleau and one for blakeman (see below why). Below are the distributions for such a test assuming, like exponent, that the pats balls were at 12.5 and the colts at 13.0.
The red dot here denotes our "observed" value if we don't assume the null hypothesis. This result is highly significant (the probability of accepting the null hypothesis is 0.0012). However, we are dealing with a small sample size, and it could be that blakeman's measures are off. Here is what a test of Piroleau's ratings look like:
As you can see from these ratings, the differences between the pats and colts balls are less clear. While the observed value lies along the left tail of the distribution (where pats balls have a bigger PSI drop), its not on the tail itself. The the probability of accepting the null hypothesis is 0.0796, which is above the threshold normally used to reject the null hypothesis (0.05).
One limitation in using this approach is that we must assume that the measures here are independent, this limits our ability to permute the data from both sets of measures to increase power. Worse, there are potential confunds to the repeated measures that makes the differences between raters appear systematic, as opposed to random, here. This limits our ability to conclude anything about the data. In one set of ratings, the results are statistically significant, in the other, they are not.
One thing that hasn't been modeled is the variation in PSI pressure from the original data. We can simulate this by assuming, conservatively, that the standard deviation is 0.16 for all data (and that the 0.41 standard deviation for the pats was due to something). Making this assumption does not change the results much. Blakeman remains significant (p = 0.0026) while piroleau's are not (p = 0.0978).
The problem here is that our measure of standard deviation is inaccurate (due to the problems of small sample sizes discussed above). In fact, this is a valid explanation for why the pats balls vary more than the colts balls, as the variance of PSI drop is not stable. However, there is a way we can estimate the standard deviation of the PSI by using another sampling approach, called a bootstrap.
A bootstrap is a method for exploring the population underlying small sample sizes. The Pats footballs and colts footballs belong to a population called "inflated pre-game NFL footballs". One way we can simulate this population is by constructing a series of "average" inflated NFL footballs. Because of math (specifically the law of large numbers), any distribution of averages will be shaped like a bell-curve, which allows us to calculate the variance along the distribution. Specifically, the population standard deviation will be approximately four times the standard deviation of the bootstrap.
Now, one could argue that the colts and pats deviations should be handled separately because they may have been inflated to different values, so we can run the bootstrap on each population, to see what deviations we should plug in for the colts and pats. This is a conservative estimate, as it will reduce the standard devations if this assumption is true, however, this estimate may still be higher than the observed standard deviation. In this case the standard deviation is about 3.3 times the bootstrap deviation for the pats, and 2 times the bootstrap deviation for the colts.
Here, we start to see the problem with small sample sizes. Below are the pats distribution of balls:
You can easily see that the raters are significantly different from one another (suggesting that the raters are actually unreliable, which limits the interpretability of any data). However, the variance from the population means are really similar. The standard deviation of the bootstrap is approximately 0.39 and 0.4 for the two raters, which matches the standard deviation observed for our data (0.41 I believe). Unfortunately, the distributions for the colts look weird:
While piroleau's ratings looks somewhat normal. Blakeman's look uniform. Both have a much wider spread than observed for the patriots. I'm not suggesting that the officials did anything at all to obscure their measures for the colts. Rather, this is a problem of extrapolating standard deviation from a sample size of four footballs vs. eleven. In any case, the standard deviation here is about 0.31 for both ratings. The fact that this standard deviation is higher may explain why the permutation test was less significant than the parametric test; the colts sample may be underestimating the true standard deviation leading to an overestimation of significance.
Let's be conservative here and re-run our tests with 0.3 standard deviation for both groups, just for grins.
As you can see from the above, we still have the same result. Blakeman's measures show a significant difference in PSI drop (p = 0.0082), while piroleau's does not (p = 0.16).
Finally, let's say that the data are flipped, and that prioleau and blakeman switiched gauges between measuring the colts and the pats balls. In this case we will have significant effects for both groups (p = 0.0232, and p = 0.0438). However, the significance is not nearly as high as before, and the latter p value would not be significant if controlling for the fact that I ran two tests.
Regardless of whether you control for variance in the footballs, you have sets of measures that show a significant drop off and sets of measures that do not. The strongest conclusion we can draw is that we are underpowered to detect any differences in inflation between the two datasets without knowing two things:
1) The PSI values of all footballs at both halftime and the start of the game.
2) Who used what gauge at both halftime and the start of the game.
Such knowledge would enable better confidence in estimating the statistics we need to determine whether the difference in PSI drop is no different than what would be observed by chance.
In truth, Exponent shouldn't be blamed solely here; they were given a sloppy dataset to analyze, and chose to analyze it in a manner consistent with many peer-reviewed publications of sloppy data. In fact, many of their models maintain significance when using a better test. The larger problem here is that the refs were not paying attention when they measured the footballs at half-time or at the start of the game, which neither the Wells report nor the Exponent report explained.
----------
Here's the distributions for the permutation tests for the last two tests (where the ratings were switched for the colts):
From a statistics perspective, the 0.4 standard deviation value is likely more accurate, in which case the distributions look like this:
In this case, blakeman's ratings show a significant difference in PSI drop (p = 0.0383), but piroleau's does not (p = 0.0772).
One should probably keep in mind that nearly every test shows a difference between the pats and colts balls. However, due to the small sample size and sloppy recording, it is difficult to dissociate from random chance. The statistics, when done better, are inconclusive without more testing.
Ok. I've been waiting awhile before stepping in, and I probably shouldn't. But something does need to be stated about Exponent's poor statistical analysis of the data given.
There are two primary problems with the statistical analysis in Exponent's report. The first has been mentioned already; using parametric statistics on non-normal distributions leads to inaccurate tests of statistical significance. In other words, the model exponent used to answer the question, "Did the Patriots' balls drop in PSI more than the Colts?" assumes that the data are distributed like major league career OBP numbers:
Note how the most frequent values in the range of career OBP values fall in the center of the distribution. Also note that the dropoff in frequency is symmetrical on both sides. This is often referred to as a "bell-curve" distribution because it is shaped like a bell. The model used for analysis uses the standard deviation (technically the variance, but since standard deviation is a more familiar term that people might understand, I'm using that here) to assess whether the drop in PSI is greater for the Pats' balls, then what would be expected due to random chance. The problem is that the PSI-drop data looks like this:
This is a common problem when dealing with small sample sizes; distributions of data look weird and the standard deviation is inaccurate. As a result, the significance assessed from the standard deviation is also inaccurate. As a side note, I've looked at this small data in multiple ways and it doesn't really matter how you look at it, the distributions always look non-bell curvy.
One simple solution is to use a better model. The question is, how do we construct a better model? If the hypothesis is that the drop in PSI is greater for the pats than the colts, then the null hypothesis is that the drop in PSI is similar between the pats and the colts. Under this null assumption, if we mixed up the two sets of balls, there should be no difference between the average PSI drop. Therefore, we can construct a distribution of possible results by mixing up the groups in lots of different ways and calculating the difference of means. If our observed value when the groups are preserved is far from the center of the distribution, the probability that the null assumption is correct (a.k.a. the "p value") decreases. This is known as a permutation test.
For those in the know, I ran 10000 permutations, so the "p value" here is extremely accurate.
Here, we actually have two permutation tests, one for piroleau and one for blakeman (see below why). Below are the distributions for such a test assuming, like exponent, that the pats balls were at 12.5 and the colts at 13.0.
The red dot here denotes our "observed" value if we don't assume the null hypothesis. This result is highly significant (the probability of accepting the null hypothesis is 0.0012). However, we are dealing with a small sample size, and it could be that blakeman's measures are off. Here is what a test of Piroleau's ratings look like:
As you can see from these ratings, the differences between the pats and colts balls are less clear. While the observed value lies along the left tail of the distribution (where pats balls have a bigger PSI drop), its not on the tail itself. The the probability of accepting the null hypothesis is 0.0796, which is above the threshold normally used to reject the null hypothesis (0.05).
One limitation in using this approach is that we must assume that the measures here are independent, this limits our ability to permute the data from both sets of measures to increase power. Worse, there are potential confunds to the repeated measures that makes the differences between raters appear systematic, as opposed to random, here. This limits our ability to conclude anything about the data. In one set of ratings, the results are statistically significant, in the other, they are not.
One thing that hasn't been modeled is the variation in PSI pressure from the original data. We can simulate this by assuming, conservatively, that the standard deviation is 0.16 for all data (and that the 0.41 standard deviation for the pats was due to something). Making this assumption does not change the results much. Blakeman remains significant (p = 0.0026) while piroleau's are not (p = 0.0978).
The problem here is that our measure of standard deviation is inaccurate (due to the problems of small sample sizes discussed above). In fact, this is a valid explanation for why the pats balls vary more than the colts balls, as the variance of PSI drop is not stable. However, there is a way we can estimate the standard deviation of the PSI by using another sampling approach, called a bootstrap.
A bootstrap is a method for exploring the population underlying small sample sizes. The Pats footballs and colts footballs belong to a population called "inflated pre-game NFL footballs". One way we can simulate this population is by constructing a series of "average" inflated NFL footballs. Because of math (specifically the law of large numbers), any distribution of averages will be shaped like a bell-curve, which allows us to calculate the variance along the distribution. Specifically, the population standard deviation will be approximately four times the standard deviation of the bootstrap.
Now, one could argue that the colts and pats deviations should be handled separately because they may have been inflated to different values, so we can run the bootstrap on each population, to see what deviations we should plug in for the colts and pats. This is a conservative estimate, as it will reduce the standard devations if this assumption is true, however, this estimate may still be higher than the observed standard deviation. In this case the standard deviation is about 3.3 times the bootstrap deviation for the pats, and 2 times the bootstrap deviation for the colts.
Here, we start to see the problem with small sample sizes. Below are the pats distribution of balls:
You can easily see that the raters are significantly different from one another (suggesting that the raters are actually unreliable, which limits the interpretability of any data). However, the variance from the population means are really similar. The standard deviation of the bootstrap is approximately 0.39 and 0.4 for the two raters, which matches the standard deviation observed for our data (0.41 I believe). Unfortunately, the distributions for the colts look weird:
While piroleau's ratings looks somewhat normal. Blakeman's look uniform. Both have a much wider spread than observed for the patriots. I'm not suggesting that the officials did anything at all to obscure their measures for the colts. Rather, this is a problem of extrapolating standard deviation from a sample size of four footballs vs. eleven. In any case, the standard deviation here is about 0.31 for both ratings. The fact that this standard deviation is higher may explain why the permutation test was less significant than the parametric test; the colts sample may be underestimating the true standard deviation leading to an overestimation of significance.
Let's be conservative here and re-run our tests with 0.3 standard deviation for both groups, just for grins.
As you can see from the above, we still have the same result. Blakeman's measures show a significant difference in PSI drop (p = 0.0082), while piroleau's does not (p = 0.16).
Finally, let's say that the data are flipped, and that prioleau and blakeman switiched gauges between measuring the colts and the pats balls. In this case we will have significant effects for both groups (p = 0.0232, and p = 0.0438). However, the significance is not nearly as high as before, and the latter p value would not be significant if controlling for the fact that I ran two tests.
Regardless of whether you control for variance in the footballs, you have sets of measures that show a significant drop off and sets of measures that do not. The strongest conclusion we can draw is that we are underpowered to detect any differences in inflation between the two datasets without knowing two things:
1) The PSI values of all footballs at both halftime and the start of the game.
2) Who used what gauge at both halftime and the start of the game.
Such knowledge would enable better confidence in estimating the statistics we need to determine whether the difference in PSI drop is no different than what would be observed by chance.
In truth, Exponent shouldn't be blamed solely here; they were given a sloppy dataset to analyze, and chose to analyze it in a manner consistent with many peer-reviewed publications of sloppy data. In fact, many of their models maintain significance when using a better test. The larger problem here is that the refs were not paying attention when they measured the footballs at half-time or at the start of the game, which neither the Wells report nor the Exponent report explained.
----------
Here's the distributions for the permutation tests for the last two tests (where the ratings were switched for the colts):
From a statistics perspective, the 0.4 standard deviation value is likely more accurate, in which case the distributions look like this:
In this case, blakeman's ratings show a significant difference in PSI drop (p = 0.0383), but piroleau's does not (p = 0.0772).
One should probably keep in mind that nearly every test shows a difference between the pats and colts balls. However, due to the small sample size and sloppy recording, it is difficult to dissociate from random chance. The statistics, when done better, are inconclusive without more testing.
The last part of that, where I estimate the standard deviation from the bootstrap, looks inaccurate to me. I should've boostrapped the standard deviation instead of the mean. Oh well.
Yeah, you're cluttering this beautiful thread with your statistical rigor. More questions about things that were answered 15 pages ago and dumb theories, please!EricFeczko said:
We know there's a difference between the Patriots' and Colts' balls, because the Colts' balls were measured later and thus have a higher temperature / pressure (as they had more time to come up to room temperature). To be clear, you are not accounting for this, correct? This was Snyder's point, basically - the statistical significance tests aren't valid because they don't include the time.EricFeczko said:
You are absolutely correct, but let us forget about time for a second. The tests don't dissociate whether this difference is greater than the noise level (of which time, wetness, improper gauge usage, are all factors).Super Nomario said:
The critical thing here is that the data are sloppy and basically tell us nothing regarding the differences between halftime colts and pats balls. The only way you can acheive significant differences with both sets of measures to is if you switch the ratings between blakeman and piroleau, which is the definition of overt data manipulation.
By "switch the ratings," do you mean assuming that Blakeman and Prioleau switched who had the logo and non-logo gauge between measuring the Patriots' and Colts' balls?EricFeczko said:
Are you talking about the Mona Lisa Vito press conferences? Because Belichick was the one who went on television claiming that he had never thought about air pressure before, not Brady.Peak Oil Can Boyd said:
Brady went on television and talked about how those balls were perfect to him and that this isn't ISIS, no one is dying. Believing Brady was lying or whatever because of this has been part of the PR campaign, where they conflate small details until it gets to the point where nobody cares anymore. This was similarly done with implying that Brady made frantic phone calls to McNally, whom he had never talked to on the phone before, because of their conspiracy falling apart by combining the actions and identities of McNally (the bathroom guy, who Brady barely knew, and never talked to Brady) and Jastrzemski (who Brady actually did speak to and had shared his phone number with).
If you're a Patriots fan and can fall for this BS, think of how easy it must be for everyone else.
Yes. IIRC there was no evidence that they did so, apart from the ratings themselves. I could be wrong about it, it's been awhile since I've read the report.Super Nomario said: