How well does UZR predict UZR?

As I mentioned a few days ago, Fangraphs now has UZR data from 2002-2008. This is fantastic for two reasons. One, we now have a truly advanced and freely available defensive metric that’s easily accessible.

And we also get the chance to play around with the guts of UZR a bit and see how it works and what makes it tick. So, if you’re not afraid of trying to look a little deeper into UZR and defense in general, here goes nothing.

A simple little test. I simply looked at matched pairs of a player at a position, year one to year two. I wanted to see how well UZR (in this case, UZR/150) from year one agreed with year two.

I used two simple tests: average absolute error and correlation. No minimum cutoffs; instead I weighted each player by the smaller number of defensive games between the two seasons.

And here’s the results:

 AAE Correl. OF 13.2 0.27 IF 10.1 0.25 Total 11.3 0.26

Curiously, outfield UZR is better correlated with year two performance; and yet infield UZR has a lower average error year-to-year. I’ll be honest – I can’t explain it, although my feeling is that AAE is more useful than correlation in this case.

I shouldn’t have to tell you that a 10-run error band, given that UZR/150 should generally range from about -15/15 for a full-time player at a postion, is pretty big. That said, this includes all players, including a lot of part-timers. (If I cut it down to only players with > 90 DG, the AAE drops to about 8.)

So here’s the biggest caution I can give you with defensive (or ANY sample) data: use more than one season! Regress to the mean! Doing these two things will save you much grief later on.

8 Responses to How well does UZR predict UZR?

1. Pizza Cutter says:

I’d interpret that a little differently, Colin. Fielding performance is one of those “a lot of luck” involved things. It’s not DIPS-BABIP range, but it’s less reliable than we might think. Can you break that down by position?

2. jinaz says:

There’s clearly both a measurement-error problem (“video scouts” are apparently not very reliable, given the BIS vs. STATS Inc differences) and a “luckiness” issue. The first issue can be helped by using multiple data sources (e.g. still use a STATS Inc based system, and maybe TotalZone too, plus bUZR, and average ’em). It won’t eliminate the problem, but it’ll reduce it if you’re only using one year.
The luckiness issue is best resolved by using multiple years and regressing to the mean. Those help with the user error problem too, of course.
I really like Rally’s defensive projections, as they do all of this. I’m guessing he’ll use bUZR in his next update instead of RZR. -j

3. dan says:

Just curious…. If someone were to add up all the individual UZR numbers for a team, how would that compare to the team’s overall DER (or some team defense stat)? You would think they’d be close or the same, but if they’re not then that’s obviously a huge red flag.

4. Colin Wyers says:

It’s a good question, Dan. Unfortunately the way I gathered the UZR data off Fangraphs doesn’t work well for that – players who played for more than one team are only listed at their seasonal line. I could try and go back and recollect all the data the long way around, but it would be… tedious. I’m not ruling it out.

5. dan says:

I just did this quickly by hand for the Phillies. Addind up all the individual UZR data for players comes out to 48.3, and the team pages adds up to 48.5. I guess the team pages just automatically do what I just did. THT’s team fielding stats have the Phillies at +52, using RZR (I think ). Not sure if that means anything, but the individual UZRs add up to be pretty close to THT team data.

6. dan says:

And just to clarify, I realize they are from similar data sources.

7. Sky says:

Dan, team UZR and DER should have a decent correlation, but there still is the “luck” piece of DER to consider. Even over a full season, a team of pitchers might happen to give up batted balls that are easier or harder to field than average. UZR takes the difficulty of those batted balls as its inputs, so ignores the luck piece. DER still contains luck.

8. dan says:

I didn’t use DER when I did the thing for post #5, just to be clear.