Similarity Scores and Projections

As many visitors to this site probably know, the latest craze in projection system design is the use of similarity scores. The idea behind this, essentially, is that the performance of players similar to the one whom we are trying to project is somehow more predictive than simply his past statistics. First of all, I doubt this is really very true for batters. They are an overall much more predictable group, and they do not exhibit as many jumps all over the place. Moreover, I am unconvinced (though Chris from tells me that this is unquestionably true) that overall similarity scores will tell us anything more than category-based similarity scores, for hitters. In other words, I have a lot of doubts about how the fact that two batters have similar walk totals and batting averages will mean that their power should deveop along similar lines as well. While I am willing to believe that a 24 year-old, for example, with 10, 20, and 30 home runs in his first three seasons may not follow a Marcel projection (22 home runs) because of his young age and great progression, I think this fact (if it exists) would be reflected in a category-based similarity score, that is, by finding the average performance of 24 year-olds who also hit 10, 20, and 30 home runs from ages 21-23.
For pitchers, I am more inclined to believe that similarity scores might be a factor, and that in fact various components are more lkely to interact (making overall similarity scores more predictive than category-based similarity scores) though there too I would assume that in fact similarity scores are simply a substitution for what we really want: scouting information. Here’s my opinion on why similarity scores have shown to be predictive (especially for pitchers): They are a decent substitute for scouting information. Baseball Prospectus’ PECOTA system, for example, includes phenotypical attributes (height, weight) and things like strikeouts and walks give us a good idea of the quality (and “hardness”) of a pitcher’s stuff. They’re not always going to be spot-on, but generally, the more Ks and the more BBs, the harder a pitcher’s fastball, and if you get a guy with a lot of Ks and few BBs, you probably have a guy with great stuff. In reality, I think, a statistics-based similarity score system is just a substitute for a scouting-based similarity score system which would be just that much better. Which player is going to age better: The guy with who throws mostly fastballs or the curveball pitcher? Probably the first, but PECOTA (or any other similarity score system today) doesn’t know which pitcher is which. It can only guess based on their component stats. At some point, we’ll need to build up an extensive database of scouting information, and use that instead.
One last thing, on error ranges. One of the cooler features of PECOTA is that it gives us percentage chances for things like whether or not the player will breakout or collapse, and includes error ranges in its forecasts. My personal opinion is that those error ranges are done incorrectly (based on similarity scores); I believe they should be based on a player’s experience, with a normal distribution built off that. I doubt similarity scores give us better ranges than a simple normal distribution would. Finally, and this is a bit of a personal rant that should not be viewed as an attack on PECOTA, because I really, really admire what Nate has done with his system. I personally do not like how BP sells its error ranges. They say it acknowledges the inaccuracy of any projection (which is obviously good). However, what their error ranges really do is add a false sense of accuracy, like we know that player x has a 90% chance of being better than their 10th percentile projection or a 10% chance of being better than their 90th percentile projection. Until someone actually studies the accuracy of those percentiles, overall and among different age groups, we really have no idea of how important those percentiles are, and IMO should approach them with much, much caution.
EDIT: You can read more discussion of this here.


4 Responses to Similarity Scores and Projections

  1. MGL says:

    David, I think your opinion on this is spot on. Also, don’t forget that a Marcel-type system IS based on what other 22 yo who hit 10, 20, 30 HR’s have done in the past. The thing is that we find that their fourth year HR total is no different from a 23 yo who hits 30, 10, 20, or 20, 10, 30. That is the beauty of a Marcel and why I am skeptical of a system, for batters at least, that assumes that different categories of hitters (or categories of performance or performance “trends”) should have different projections even if their Marcel’s (basically a weighted average of the past) are the same. Is that true? Is there any research that indicates that it is? I have tried to find different “profiles” of hitters whose projections do not follow a Marcel and I can’t. Young players, old ones, banner years up, banner years down, a progression line such as 10, 20, 30. They all come out the same in that 4th year! So if that assumption is not true (that certain profiles have different progressions than others), that kind of shoots a gigantic whole in the whole theory and system.
    And I definitely agree with you that until someone (and Silver and BP have the burden) can show us that their confidence bands and intervals are better than a normal approximation around the projection mean, the width based on the amount of data used in the projection, they are a worthless bell and whistle.

  2. Robby Bonfire says:

    I just saw your Justin Verlander comment at “The Hardball TImes” to the effect that; “Add a couple strike outs per game and his ERA drops half-a-point.” FYI, the regression value of a strike out is 1/10 run. So that an increase of two K’s per nine innings represents an approximate 1/5 run ERA improvement, certainly not 1/2 a run.
    In fact, each walk allowed damages a pitcher’s ERA by 1/2 run, so that improved control is the fast track to ERA improvement for all pitchers, much more so than an increase in the K rate.

  3. Brent says:

    Robby, how did you find that a walk damanges a pitcher’s ERA by 1/2 a run? Is this theoretically 1/2 a run?

  4. David Gassko says:

    A walk is worth +.33 runs. A strikeout (for a pitcher) is worth -.22 runs. Half that value is the value of the out, and half the value of taking away a BIP (and thus a potential hit — remember DIPS theory). So two extra Ks are worth about half-a-run and an extra walk is worth quite a bit less.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: