Real Plus Minus, good stat?

[luckiestman]

Thanks to Bowiac posting about it, I have taken to looking at RPM a bit and it seems interesting:
 
http://espn.go.com/nba/statistics/rpm/_/sort/RPM
 
Some things are amusing:
 
Celtics have 2 of the 12 best centers in the league (KO, Zeller 10/12)
 
Celtics have 2 of the top 9 PF in the league (Sully, Wright 5/9) Kevin Loves ranks 12 btw, so we have 2 guys better than Love.
 
Smart grades out as our best PG (24, Nelson is 36 Rondo the Mav surprisingly at 31)
 
Our SG do not show up in the top 50.
 
The most surprising given his scoring is that Jeff Green grades out at 52 among small forwards, not overall.
 
The outcomes look very strange, but I can't say wrong. Green has been putting up big numbers, but they look bigger in the box score than when you watch the games. 
 
 
I'm going to keep looking at this stat because it is the first stat I have found that fits my criteria of being counter intuitive yet plausible. 

 

[PedroKsBambino]

It's a good evolution; as several of us noted months ago when this came up, it attempts to determine the impact of quality of teammates and quality of opposition players on the floor, which is the right step to take.  It's unclear how WELL it does that, so that's a major caveat to remember when looking at RPM results.  To be clear, I have no particular reason to believe it's doing it poorly or with particular biases (e.g. favoring guys who play with great players; against bad players; who are efficient, etc.)  I just observe that advanced stats tend to take a while to calibrate on complex issues like that---the problem with teammate and opponent skill levels is tougher than UZR adjustments, for example, and we all know those haven't been fully cracked yet (though tons and tons of progress has definitely been made) and that until we have a lot more study, we shouldn't assume all those challenges have been figured out.
 
As you suggest with the note about the Celtics having two 'top centers' a metric like RPM is always going to be challenged by guys who play smaller minutes and/or who play only in certain situations or types of situations.  But that's part of the overall challenge with all-in metrics.
 
While far from perfect, I do think at a high level RPM does well on the 'eye test' where a review of the players near the top feels realistic.

 

[JCizzle]

I think it's going to be interesting to see how Zeller plays over the next few months without Rondo, to my eye they had great chemistry and Rondo was able to feed him good looks more frequently than Pressy/Smart.

 

[luckiestman]

I think it's going to be interesting to see how Zeller plays over the next few months without Rondo, to my eye they had great chemistry and Rondo was able to feed him good looks more frequently than Pressy/Smart.

 
 
I'm interested in this too. He didnt miss a beat last night (7/9). He has only taken more than 9 shots twice. He didn't get to the line at all last night, though. In the pre season, he looked good on the pnr with Turner. Zeller needs to get a little better defensively. He has the size and the strength, so I dont know what his problem is.

 

[bowiac]

As you suggest with the note about the Celtics having two 'top centers' a metric like RPM is always going to be challenged by guys who play smaller minutes and/or who play only in certain situations or types of situations.  But that's part of the overall challenge with all-in metrics.

I agree with the thrust of your post (RPM can struggle with overrating players who play with other good players, but does sorta okay in the end). With respect to Zeller/Olynyk however, as I've posted before, I'm not sure RPM is even struggling. RPM isn't saying the Celtics have had two of the most valuable centers in the league, just two of the most valuable per minute guys. They're only playing 49 minute between them. Further, as you can see from the RPM ratings, it's a pretty steep curve, and by the time you get to 9th and 10th, there's not too much distance between them and average.
 
Overall, it just combines to the Celtics getting around 1.5 points per game above average out of the position. It doesn't seem wrong to me to say that the Celtics have gotten above average production out of the position combined.

 

[repole]

My understanding of RPM is that it's essentially a huge system of equations, where each equation is a sum of 10 players (5 offense, 5 defense) set equal to the point differential that occurred while those 10 players were on the court together (also weighted for minutes of course). If you know the RPM values for 9 of those players, you can solve for the 10th.
Imagine Wright/Dirk/Crowder/Ellis/Nelson are on the court against Plumlee/Morris/Tucker/Green/Dragic, and in a given amount of time that Mavs lineup vs that Suns lineup is outscored by 5 points. Let's say we know the RPM values of the Suns players summed comes out to 0, and the RPM values of Dirk/Ellis/Crowder/Nelson sums to 0. That leaves Wright as the lone person responsible for the -5.
 
That's a very basic version of how those values are calculated. The whole point of this approach is to remove bias for who you're playing against, who you're playing with, and who you've replaced in the lineup.
 
Unfortunately this process misses a really key concept: role changes across lineups. RPM assumes a guy's value to a lineup is always the same, yet that's very often not the case. Take someone like Smart, who when on the court with Rondo plays mostly off ball, but without Rondo plays mostly on ball. You might as well be evaluation two entirely different players offensively. A lot of adjusted plus minus values are also "prior informed" and use multiple years of data. So imagine a guy like Gerald Green, who in Indiana was maybe the least productive player in the NBA, yet is very productive for Phoenix. Just his RPM value being skewed because of that can cause a big chain reaction of guys having their RPMs skewed.
 
Because of that, I'd be wary of putting too much stock into it. It's an interesting number to look at, a good conversation starter for trying to figure out if a guy is over or under valued, but it's not an end all be all number.

 

[bowiac]

While the thrust of your post is basically correct, the regularization for Gerald Green wouldn't have a particularly big impact. Regularization mostly impacts pairs of players who almost always play together. It impacts everyone, but it's not going to run into big issuse because of Gerald Green's rating changing dramatically.

Your point about role however is absolutely correct. It's something RPM can't ever hope to address really. I just don't think its an especially big deal - it doesn't need to be perfect to add value. RPM beats all comers in predicting wins and losses on the court, despite its flaws.

 

[repole]

That's fair. I haven't played around with it enough to know just how much role fluctuations impact it, just that conceptually there are some imperfections in the assumptions being made. But yeah, as far as single number evaluation things go, it's as viable or more than anything else out there.
 
Will add that single season data, especially less than half way in, has a lot of noise in it regardless, so it's difficult to trust that there's much value in how it grades someone like Smart right now.

 

[bowiac]

That's fair. I haven't played around with it enough to know just how much role fluctuations impact it, just that conceptually there are some imperfections in the assumptions being made. But yeah, as far as single number evaluation things go, it's as viable or more than anything else out there.
 
Will add that single season data, especially less than half way in, has a lot of noise in it regardless, so it's difficult to trust that there's much value in how it grades someone like Smart right now.

I'm told that currently RPM is actually being regularized solely towards box-score priors, which stabilize faster than plus/minus data. This will change at some point during the year, and isn't true of last year's RPM data. My current read is a proper weighting of RPM is to weight this year's offensive RPM data about 15 times heavier than last year's on a per minute basis, while the defense weights should given weight in equal proportion. I think this is because of how much better box score metrics are at capturing offensive value, so the box-score regularization is doing a much better job offensively than defensively presently.
 
This is another way of saying I'm not sure RPM at this point in the year is any better than BPM (box score only version of RPM).

 

[Kooz]

It's obviously fair to want to evaluate Marcus Smart, but it's unlikely to me that RPM will be sufficiently predictive in any relevant decision window. What's referred to as "noise" in the model is precisely the matter at hand. There's the roles question that repole points out, but there's many more (let's call them) variables (even though they are nowhere near discreet). Which roles, schemes, player combinations, substitution patterns, officiating biases, coaching techniques, training regimes and game situations will produce fit? By the time that is determined the need for the information is most likely gone. 
 
Said another way, in the absence of a stable evaluation of certain players is it fair to use RPM to measure Avery Bradley, especially if he is in a lineup with Smart and  Jae Crowder? To do that you have to assume that Smart and Crowder rise to league average, no? That's constructed and not "known." You say "yes" and I say "no." You analyze sensitivities and I pick at your assumptions. 
 
Again, time will tell..... How much time will tell? It depends......
 
Color me skeptical about "regularization." My beef isn't with the technique, exactly, but with frustration over what inevitably happens in the attempt to model complex interactions. You have to assume a model. It always strikes me that if I knew the model then I wouldn't need the model.
 
That's not to say that RPM doesn't provide an interesting "conversation starter" on certain decisions that face, say, a GM over time, but rather it's to argue that RPM isn't too relevant for the Celtics right now. There's not only no way out of the process of experimentation and development, I would argue that it's all essentially noise..... until it isn't. 
 
In other words, I agree completely with repole!

 

[Devizier]

Here's a question that I have; how many of these stats, from the box score variety (Dean Oliver and Kubatko stats) through the play-by-play numbers, have been tested against traded players? I mean, there are a lot of caveats with that kind of analysis, but at least you're comparing a player against himself in the same season, but with a different context.

 

[bowiac]

Here's a question that I have; how many of these stats, from the box score variety (Dean Oliver and Kubatko stats) through the play-by-play numbers, have been tested against traded players? I mean, there are a lot of caveats with that kind of analysis, but at least you're comparing a player against himself in the same season, but with a different context.

Yes, they've been tested for traded players, as well as free agents, to see which predicts the impact on the team best. When I say RPM is the gold standard here, it's precisely because RPM beats everyone else in this sort of testing.
 
Oddly, though much maligned, PER performs surprisingly well in such tests. Wins Produced is the most consistent loser meanwhile. I can track down an APBRmetrics link showing this testing (mostly by Neil Paine, now of 538).