The supervision component is also important i agree, but the SOLE Judge designation is unambiguous- even if the patriots provided balls out of specification, they are not the judge of compliance- ONLY the referee is.
By not being the sole judge, they are also not responsible for out of specification balls. The responsibility falls ENTIRELY on the referee. There has been absolutely no evidence that the Patriots deflated the balls post inspection/pre game, until this evidence is provided and vetted, i am extremely skeptical of any tampering. Given the magnitude of leaks already, direct evidence of ball tampering would have circled the world a million billion times already...