Evaluating Coaches

[Super Nomario]

In the Rex Ryan and Coaching Carousel threads (and in a coach ranking post on a friend's blog), there are a lot of thoughts on coaches' performance. I see the same logic thrown around here that I see in most conventional football analysis: it's pretty much based on in-game tactics, and whether teams "overachieve" or "underachieve" based on the perceived quality on their talent. But obviously there's more to it than that. I wanted to throw out some ways we could look at coaches and coaching performance and get some thoughts on what we can and can't measure.
 
Big picture:

• Overall won / loss record
• Performance relative to pythagorean W/L or point differential
• Scoring performance relative to yardage performance
• Success with different teams / QBs / coordinators / etc.

Tactical:

• Fourth-down decision making (punt vs field goal vs go for it)
• Clock / timeout management especially late in halves / games
• Situational playcalling

Strategic:

• Individual game plan - attacking opponents based on strengths and weaknesses
• Team coming out prepared at beginning of games
• Halftime adjustments
• Discipline - avoiding turnovers and penalties
• Preparedness - success in long weeks (after byes) or short weeks (Thursday games, games after Mondays)

Personnel:

• Developing young players
• Maximizing player value / getting value out of players other coaches wouldn't
• Being respected / liked by players
• Building an organizational culture

Staff:

• Picking good coordinators / position coaches
• Grooming young coordinators / assistant coaches

Not all of these are quantifiable, but many are.
 
The following items I think would be useful to track as well. They don't necessarily denote quality, but are part of the picture of what a coach is like:

• Run / pass preference
• Formation / personnel preference (3 WR, 3-4, pistol, etc.)
• Offensive scheme (Coryell, west coast, zone blocking, etc.)
• Defensive scheme (cover-2, zone blitz, man-1, etc.)
• Blitzing preferences
• How much do they value big play vs consistently moving the chains?
• Stick with the same players or substitute a lot?
• How much do they emphasize special teams (using ST-only players or starters on specials vs just backups)
• Hands-on vs delegating

What else can we look at? What else would reflect coaching?

 

[Reverend]

Organizational Culture/Philosophy stuff?

And I mean this as different from "chemistry."

 

[Super Nomario]

Organizational Culture/Philosophy stuff?

And I mean this as different from "chemistry."

OK, I added "building an organizational culture" under "Personnel." I have some "soft" stuff on there, but in general I don't have too much because it's not only hard to quantify, it's hard to even identify success at this sort of thing.

 

[DanoooME]

Not sure if it should be a separate element under Staff (and if it's even measurable), but what about delegation of duties?  I was thinking specifically of a situation like the Cowboys' Jason Garrett taking over play calling at times from his OC.  But I think there are other coaches that are probably very "hands off" when it comes to their staffs.

 

[Super Nomario]

Not sure if it should be a separate element under Staff (and if it's even measurable), but what about delegation of duties?  I was thinking specifically of a situation like the Cowboys' Jason Garrett taking over play calling at times from his OC.  But I think there are other coaches that are probably very "hands off" when it comes to their staffs.

I added "hands-on vs delegating" to the bottom (preferences section). It's not clear to me that the head coach involved in playcalling is inherently good or bad.

 

[wutang112878]

This is a great topic.  Not to oversimplify but I think what we focus on most is the year to year production for a coach, and we try to judge that based on the circumstances they had to deal with and thats what flaws our analysis.  We need to rationally quantify the subjective. 
 
For example, the Pats and 49ers both had 12 wins and both had challenges but by the end of the season we had lost our NT/DT, MLB, RT, TE all of whom are one of our 10 most talented players.  Our 12 wins should be looked at through a different lens than the 49ers 12 wins. 
 
In addition to injuries, there is also the strength of schedule issue, and the talent available issue see R. Ryan.  It becomes very difficult and we ultimately seem to argue about how difficult the circumstances were for that coach and generally dont circle back to 'in light of these circumstances that we all agree on, coach XYZ did a B+ job'.  Its the nature of the beast.
 
 
 
I wonder if we could look at things a bit differently.  Part of what what makes Dante so great is his ability to get replacement level production out of untalented players, the same can be said for Bill.  If we looked at DVOA for example for a coach ultimately its just a function of the DVOA for each individual player on the roster, which is going to be affected by star players and untalented players.  But if we start with the assumption 'good coaches make untalented players average' then perhaps we could judge a coach based on their ability to put players in a position such that they minimize below average play.  For example (with some back of a napkin math/analysis), the Rams lost Bradford, the league average QB rating this year was 84.1, Kellen Clemens had a 78.8 rating in 10 games and for his career he is a 68.6 rating guy.  I would argue that is a good coaching job and this approach is relatively easy to quantify.

 

[Super Nomario]

This is a great topic.  Not to oversimplify but I think what we focus on most is the year to year production for a coach, and we try to judge that based on the circumstances they had to deal with and thats what flaws our analysis.  We need to rationally quantify the subjective. 

Sure.
 

For example, the Pats and 49ers both had 12 wins and both had challenges but by the end of the season we had lost our NT/DT, MLB, RT, TE all of whom are one of our 10 most talented players.  Our 12 wins should be looked at through a different lens than the 49ers 12 wins.

I think the 49ers had to deal with a lot of adversity this year, to be honest. Chris Culliver and Michael Crabtree went down with injury before the season even started; those are two of their better young players on either side of the ball. Aldon Smith had the DUI distraction, suspension / leave, and had his worst season by far. Patrick Willis and Vernon Davis both missed some time with injury. And of course, Harbaugh has a talented but fairly raw young QB in Kaepernick while Belichick gets to lean on a Hall-of-Famer in Brady. But here we are doing the same thing you said we shouldn't do in the original post - judging based on subjective assessment of circumstance.
 

In addition to injuries, there is also the strength of schedule issue, and the talent available issue see R. Ryan.  It becomes very difficult and we ultimately seem to argue about how difficult the circumstances were for that coach and generally dont circle back to 'in light of these circumstances that we all agree on, coach XYZ did a B+ job'.  Its the nature of the beast.
 
I wonder if we could look at things a bit differently.  Part of what what makes Dante so great is his ability to get replacement level production out of untalented players, the same can be said for Bill.  If we looked at DVOA for example for a coach ultimately its just a function of the DVOA for each individual player on the roster, which is going to be affected by star players and untalented players.  But if we start with the assumption 'good coaches make untalented players average' then perhaps we could judge a coach based on their ability to put players in a position such that they minimize below average play.  For example (with some back of a napkin math/analysis), the Rams lost Bradford, the league average QB rating this year was 84.1, Kellen Clemens had a 78.8 rating in 10 games and for his career he is a 68.6 rating guy.  I would argue that is a good coaching job and this approach is relatively easy to quantify.

It's quantifiable at certain positions like QB ... sort of. For OL and many defensive positions, there aren't good stats. There are also a lot of other factors that go into this sort of thing. Did Matt Cassel have a better year in '08 than '09-'13 because the coaching was better, or because he got to throw to Moss / Welker?
 
And the nature of castoffs is you usually don't have a good sample size for apples-to-apples. Dante's helped turn Dan Connolly into a multi-year starter, but he never started a game before he got to New England, so there's nothing to compare his performance here to. You note above Clemens has had his best season, but much of his pre-2013 resume was a half-season he started for the Jets in 2007; is that really a relevant comparison?

 

[Saints Rest]

I fear that tyring to quantify such things will be enormously hard as the amount of noise in any data, not to mention SSS issues, will be enormous.  I think so much of htis will come down to so-called (and denigrated) "soft" issues that it will never really prove much more valuable than the current level of "my gut says . . ."
 
Adding to the mix in terms of discussion points, I would suggest that some level of excellence across other factors:

• Has he shown capabilities with different teams/ownerships/GM's?  Call this the Parcells Element.
• Has he shown capabilities with different QB's?  Call this the Andy Reid (or others).
• Has he been able to succeed with different systems (offense or defense)?  
• Has he been able to extend success over multiple coordinators?  Call this the Belichick effect.

 

[wutang112878]

I think the 49ers had to deal with a lot of adversity this year, to be honest. Chris Culliver and Michael Crabtree went down with injury before the season even started; those are two of their better young players on either side of the ball. Aldon Smith had the DUI distraction, suspension / leave, and had his worst season by far. Patrick Willis and Vernon Davis both missed some time with injury. And of course, Harbaugh has a talented but fairly raw young QB in Kaepernick while Belichick gets to lean on a Hall-of-Famer in Brady. But here we are doing the same thing you said we shouldn't do in the original post - judging based on subjective assessment of circumstance.

 
Its so hard to not get caught up in this 'how difficult' game. 
 
 

It's quantifiable at certain positions like QB ... sort of. For OL and many defensive positions, there aren't good stats. There are also a lot of other factors that go into this sort of thing. Did Matt Cassel have a better year in '08 than '09-'13 because the coaching was better, or because he got to throw to Moss / Welker?
 
And the nature of castoffs is you usually don't have a good sample size for apples-to-apples. Dante's helped turn Dan Connolly into a multi-year starter, but he never started a game before he got to New England, so there's nothing to compare his performance here to. You note above Clemens has had his best season, but much of his pre-2013 resume was a half-season he started for the Jets in 2007; is that really a relevant comparison?

 
Great point, looking at the performance of individual players is difficult because it doesnt address SSS and the talent around that player.  Maybe there is another approach that could eliminate the player specific analysis.
 
Imagine an NFL world where the salary cap hit for each player matched their actual production so good teams had much higher total cap numbers than bad teams and thus the salary cap hit per win was virtually equal for all teams.  Then we could measure total salary cap dollars lost to injury and the adjust salary cap hit per win for each coach, and that would be very quantifiable with an intelligent analytical approach.  Obviously that world doesnt exist, but I cant think of any good stat that could be used across the board.
 
Perhaps their is another way to quantify what talent a coach has to work with.  First develop a projection for each player and judge each position individually, one major problem would be 1st and 2nd year players.  For QBs perhaps QB rating is sufficient.  RBs could be judged mainly on yards per carry and slightly on yards per catch.  WRs on maybe production per snap or QB rating on/off?  DTs mainly on yards per rush against and slightly on passer rating against.  CBs are judged primarily on passer rating against.  With regression you could probably get a relatively accurate equation for total wins = f ( QB * Z + RB * Q + ...) where I am sure Z is the highest to highlight the QBs importance.  Then coaches could be measured by comparing the actual wins to the projected wins based on the talent available for each snap (so this year the Pats would have only had Vince's expected production for the roughly 25% of the snaps he played). 
 
This sounds amazingly complex but its really the only analytical way I can think of.  Its similar to the projected runs scored and against that are used in baseball which gets you to the pythag, and could be adjusted during the season to account for injuries.  But the stats and projections in baseball are infinitely easier than football.

 

[Super Nomario]

I fear that tyring to quantify such things will be enormously hard as the amount of noise in any data, not to mention SSS issues, will be enormous.  I think so much of htis will come down to so-called (and denigrated) "soft" issues that it will never really prove much more valuable than the current level of "my gut says . . ."

That's why I'm trying to break it down into smaller, potentially answerable questions. Most coaches (like most everyone else) aren't "good" or "bad"; they have different strengths and weaknesses. If we can say that coach X is good at being aggressive on 4th down and his teams tend to improve as the season goes on but they commit too many personal fouls and don't integrate rookies well, that's useful and interesting. How we weigh each of those factors into an overall framework is largely personal and subjective.
 

Adding to the mix in terms of discussion points, I would suggest that some level of excellence across other factors:

• Has he shown capabilities with different teams/ownerships/GM's?  Call this the Parcells Element.
• Has he shown capabilities with different QB's?  Call this the Andy Reid (or others).
• Has he been able to succeed with different systems (offense or defense)?  
• Has he been able to extend success over multiple coordinators?  Call this the Belichick effect.

Added this to the "big picture" section. I think the "different QBs" one should be the Gibbs effect.

 

[wutang112878]

 
I fear that tyring to quantify such things will be enormously hard as the amount of noise in any data, not to mention SSS issues, will be enormous.  I think so much of htis will come down to so-called (and denigrated) "soft" issues that it will never really prove much more valuable than the current level of "my gut says . . ."
 
Adding to the mix in terms of discussion points, I would suggest that some level of excellence across other factors:

• Has he shown capabilities with different teams/ownerships/GM's?  Call this the Parcells Element.
• Has he shown capabilities with different QB's?  Call this the Andy Reid (or others).
• Has he been able to succeed with different systems (offense or defense)?  
• Has he been able to extend success over multiple coordinators?  Call this the Belichick effect.

 

 
Are there any coaches besides Parcells and Bill that even satisfy 2 of these criteria?  Very few coaches actually use different systems, unless in some cases where they completely hand over one side of the ball to an assistant like Sean Paytons defense or Rex Ryans offense.  Very few coaches have the longevity to see 2 competent QBs, etc.  Its just so rare for an organization to be stable enough to give a coach the opportunity to meet the last 3 criteria

 

[Reverend]

OK, I added "building an organizational culture" under "Personnel." I have some "soft" stuff on there, but in general I don't have too much because it's not only hard to quantify, it's hard to even identify success at this sort of thing.

 
I agree it's difficult, especially on the margins. Sometimes, though, we get clear evidence that one organization just has its shit together more than another, like I'm having trouble finding it but was it Randy Moss on the Titans who said the place was weird and he was just keeping his head down? Or some receiver who went to Oakland? Also, the comments out of Houston after last year's game where they very candidly indicated that the Patriots were just a more professional football team than they were were striking.
 
I think a huge complication to this, though, would be separating the coach's influence on this organizational culture and the impact of the owner. Like, it's not at all clear that Belichick could do anything about the Washington Indigenous People, and we know how long it took him to bolt for NE given the choice of working for Kraft and working for Woody Johnson.

 

[Reverend]

 
I fear that tyring to quantify such things will be enormously hard as the amount of noise in any data, not to mention SSS issues, will be enormous.  I think so much of htis will come down to so-called (and denigrated) "soft" issues that it will never really prove much more valuable than the current level of "my gut says . . ."
 
Adding to the mix in terms of discussion points, I would suggest that some level of excellence across other factors:

• Has he shown capabilities with different teams/ownerships/GM's?  Call this the Parcells Element.
• Has he shown capabilities with different QB's?  Call this the Andy Reid (or others).
• Has he been able to succeed with different systems (offense or defense)?  
• Has he been able to extend success over multiple coordinators?  Call this the Belichick effect.

 
Would the bolded include adapting and evolving? Just using Belichick as an example again, it seems like he's moved across 3-4 and 4-3 stuff (or whatever more accurate concepts SN uses) and different types of personnel (which speaks to SN's consideration of tracking different sets) based on what's available/undervalued on the market. TEs. Realizing the possibility of slot receivers (who are cheap). Etc.