On the reliability of defensive abilities, part 1

What part of defense is luck and what part is skill?  As I continue the development of my fielding system (OPA! or out probability added above average), I’m curious as to whether fielding is a repeatable skill.  Some time ago, DIPS theory was coined when it was discovered that certain things that pitchers do (give up walks, strike batters out) were more stable from year to year, while others (BABIP) were not.  The conclusion is that a pitcher has much more of a repeatable skill for events that do not involve the defense than those that do.  What of fielding skill?  Are there fielding events which seem to be more repeatable from year to year?
The nicest part about my OPA! system is that I’m able to take an isolated look at various components of fielding, especially on ground balls.  Add a few adjustments for the number of balls hit to the fielder and it’s easy to create a rating per ground ball or fly ball or popup for everyone in baseball.  Select out the guys who have played a decent amount of time at the position (450 innings, which is 50 full games in a season).  My statistical technique of choice is intraclass correlation, which is sort of like a year-to-year correlation, but with the ability to use multiple datapoints.  In this case, I’m using four years worth of data from 2004-2007.  The higher the correlation (the maximum is 1.0), the more stable the stat from year to year.  In a lot of cases, the results were in the .10 to .20 range, meaning that either those particular skills at that particular position are much more the result of luck than anything else, or there’s a lot of skill involved and my measure is completely off. 
I’ve detailed my method for getting the ratings on infield grounders here and here, discussed fly ball extra bases prevented (cutting the ball off) here, and my model of fly ball range here.  I looked at line drives and pop ups (for infielders) much the same way.  I pretty much stole John Walsh’s method for rating outfield arms.
I’ve hidden the numerical spaghetti behind the cut.  For those who are interested in the gory details, it’s all there in black and white.  For those who just want the conclusions, here they are:

  • Range is generally the most consistent thing about a fielder from year to year.  That means it’s something you’re either good at or not.  Sorry, Derek.
  • Middle infielders are most consistent in those stats that measure gross motor skills (range, throwing) than fine motor skills (catching, fielding).  Makes sense.  They have to cover the most ground, so coaches look for the guy who can consistently get to the ball.  We can worry about picking the ball off the ground when you get there.
  • Ratings involving catching the ball for first basemen are pretty consistent from year to year.  Part of it is that the first baseman is asked to catch a lot of throws during the game, so his “true” talent level is more likely to be exposed.  But, we also see this pattern when it comes to catching line drives and popups.  Then again, I suppose that a coach finds a good first baseman by looking for the guy who’s good at consistently catching the ball.
  • In fact, corner infielders seem to be much more consistent in their fine motor skills (fielding, catching) from year to year.  Of course, they also have less time to react to grounders (but don’t have to cover as much ground), since they play closer to the plate, so it makes sense that a player who was particularly sure handed would be steered to those positions.  It also means that since they don’t have a lot of reaction time, whether they get to the ball or not will be more of a matter of luck.
  • Sean Smith, former StatSpeak writer and inventor of TotalZone, one of the systems to which I owe a great deal of credit on OPA! once pointed out that pop ups tell us nothing about an infielder’s defensive abilities.  He’s right.
  • Line drives to the right side are mostly a matter of luck.  Line drives to the left side are slightly less a matter of luck.  I’m not entirely sure why.  It might be that there are more RH pull hitters who hit liners to the left side, so it’s just a matter of the left-sided fielders getting more chances.
  • Left fielder range is surprisingly more consistent year-to-year than CF or RF range.  Maybe it’s just that the guys who end up in LF are consistently slow.
  • In fact, most measurements in the OPA! system begin to become less consistent year-to-year as you move from left to right in the outfield.  Weird.
  • Arm ratings for outfielders showed an interesting pattern.  Left fielders were more consistent in their actual throwing runners out than their holding runners.  Center fielders showed the opposite pattern.  There was a lot of consistency in how many runners they held (presumably by the reputation of their arm), but little consistency in how many outs they recorded.  Seems that runners and third base coaches are afraid of center fielders’ arms on reputation despite the fact that a CF’s performance in the past has little bearing on his performance in the future.  Right fielders showed a good pattern of stability in both holding runners and throwing them out.
  • OPA! overall is a pretty consistent for infielders, but not so much for outfielders.  Again, either my measure is flawed (entirely possible) or outfield defense really is something of a crapshoot.  It’s weird because the list of outfielders that I get in terms of range at least passes the smell test.  Hmmm… 

Fielding positions are, of course, not arbitrarily assigned.  Unlike evaluating hitting, which everyone does, to evaluate second basemen or left fielders is to evaluate a very specific set of folks.  There are specific skill sets that come in handy in different positions on the field and it appears that managers (presumably at all levels) subcosciously or not-so-subconsciously put guys whom they know are consistent at that particular skill at those positions.
Tomorrow, I’ll look at the correlations between different fielding abilities, both within a position and among different positions (think utility infielders…)
Click below for the gory details.
The details: 
Intra-class correlations for infield ground balls (by position, per ground ball)
pos — range — field — throw — catch — DPs — total OPA/GB
1B  — .509    — .310 — .163   — .226   — xxx  — .423
2B  – .224    — .022 — .405  — .086  –  .385 — .543
SS  — .507    — .149 –  .277  — .044  – .151   — .418
3B  — .298   — .350 –  .322  — xxx   — xxx   — .510
DPs is turning the double play (the second throw), hence why 1B and 3B don’t get rated on this one.  Catches are based on the number of balls in which the fielder was called on to make a catch, not how many ground balls he fielded.
Intra-class correlations for infield line drives (by position, per ball)
pos — range — catch — total
1B  — .076   – .372    – .117
2B – .029   — .028   – .036
SS –  .222  — .218    — .239
3B — .208   — .046   — .193
Intra-class correlations for infield pop ups (by position, per ball)
pos — range — catch — total
1B — .053    — .258   — .037
2B — .045    — .059  — .123
SS – .173     — .032  — .182
3B — .073    — .090  — .050
Intra-class correlations for OF fly balls (by position, per ball) 
pos — range — catch — total — FB cut
LF  — .445   — .107   — .438 – .149
CF — .254    — .444   — .272  – .130
RF — .262   — .268   — .219   – .133
Intra-class correlations for other outfield events (by position, per ball)
pos — GB cut — LD range — LD cut
LF  — .319     — .162          — .187
CF  — .303    — .179           – .158
RF  — .284    — .046          — .005
Intra-class correlations for outfield arm events (by position, per chance)
pos — arm OPA — arm XBP
LF  — .226         — .028
CF  — .036         — .314
RF –  .257         – .291
XBP is extra bases prevented.
Total OPA per inning
1B: .340
2B: .552
SS: .387
3B: .547
LF: .362
CF: .166
RF: .149


3 Responses to On the reliability of defensive abilities, part 1

  1. dan says:

    I might have missed this…. Are you going to add up all the OPA!s for individuals and see if, and how well, they match up on a team level? It would be a sign that something is wrong if most of the players on a team come out below average but the team as a whole is above average in fielding.

  2. Pizza Cutter says:

    I hadn’t thought to do that, but I certainly can.

  3. […] are some very good research pieces out there on Evaluations of Defense. This one in particular was very […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: