Neyer discusses wOBA and some Sabermetric philosophy

From the “stuff we’re reading” file: Rob Neyer over at ESPN has an extended post about embracing wOBA (a creation of Tom Tango) and a few other issues worth commenting on.  Neyer’s one of the more Saber-savvy gentlemen over at ESPN and he’s worked extensively with Bill James, and it’s cool to see these concepts creeping into the mainstream. (h/t: The Book).

But don’t stop reading there.  Rob goes on to discuss another issue which are worth a little discussion.  The first, and more important, is the issue concerning Baseball Prospectus and the fact that while they have been the industry gold standards for a while in terms of player metrics, their flagship stuff like PECOTA, VORP, and WARP are proprietary.  They do have the right to keep whatever they so desire under their hat (and I, for one, pay my annual subscription dues to read it), but Neyer brings up an interesting hypothetical:

But that science-vs.-enterprise dynamic can be tricky. The methodology behind BP’s metrics is not, to my knowledge, peer-reviewed. If one or two people make a big mistake, would anyone else know? Now, let’s jump ahead and say that two or three years down the line, the big mistake was discovered internally. Would BP announce to the world that all those numbers over the previous three years had been wrong? Or would the guys running the show decide that the loss of credibility (and potentially, revenues) isn’t balanced by the loss of integrity?

One of the charming things about Sabermetrics is that it’s grown up as mostly a field of amateur hobyists who have too much free time.  If I screw up on something here at StatSpeak, I know.  (It’s the only time I ever get comments!)  But that’s the beauty of peer review, and I love that sort of dialogue for its own sake.  The luxury that I have is that I can screw up and it means nothing more to me than a red face over having blundered.  For what it’s worth, I do trust that the people at BP are smart, dilligent people who double-check their work and each other’s work, but they are humans and humans make mistakes.

Rob points out that “Science works best under blue skies, with little thought of green.”  And here’s a philosophical point that is becoming very real in Sabermetrics.  Sabermetrics has started to see that green influence creep into it.  BP started out as a buch of people throwing around ideas on Usenet.  Now, they’ve got a business model and some of those same folks have gone corporate.

In fairness to BP, they wouldn’t survive if they weren’t putting out something good, and while we may not know the exact machinations of how something like VORP works, we have a general idea through what they have said and the concept makes sense.  Plus their stuff generally passes the “smell test” and things like PECOTA projections have been shown to correlate pretty well with actual performance.  It’s not that I worry about their stuff being bad/wrong/misleading, it’s that I worry that the science part of Sabermetrics might stagnate.  When I developed OPA! (my fielding system), I was writing the syntax that did all the calculating at the same time that I was posting the articles on it.  I got some feedback through that process that I incorporated into the system.  OPA! is a better measure because of it.  Imagine for a moment that PECOTA were open-source.  Someone out there would probably figure out a way to make it better.  It’s not.  But, that’s the trade off that you make in a closed source system.  Nate Silver, who probably has poured hours on end into that system (as well as any of the BP folks who have helped him) certainly would see no return on his investment of time, at least financially.  Would he have created the system to begin with had that incentive not been there?  I don’t know Nate, and I don’t know what his answer would be.  Maybe he would.

This is a grown-up moment for the field.  This is a philosophical turning point and one that, like most philosophical issues, doesn’t have an easy answer, or even an real answer.  Do we want to be scientists in the strict sense or are we OK with the idea that some people will keep a few secrets and charge a few dollars to see them?


6 Responses to Neyer discusses wOBA and some Sabermetric philosophy

  1. Colin Wyers says:

    I tried to send Rob an e-mail on this topic, but was foiled by ESPN’s system. I may post it later for consumption.
    That said – we do know how VORP is constructed. And we do know there are flaws with it. VORP doesn’t properly value the home run or the walk.

  2. DanC says:

    Did you really just shamelessly ask for comments? Well, here you go. Hope it adds value.
    You numbers people get so tied up in digits past a decimal point that really don’t matter. If the numbers for something are 90% of the way to accurate, that’s usually plenty good to make a decision, since there is natural variance in most everything.
    Bottom line: if something says Jeter sucks, go with it. And, add bonus points for being short to every metric.

  3. Brian Cartwright says:

    Dan, I agree. If I can project someone’s BA within 10 pts, or SA within 20, I feel I’ve hit it.
    However, if a formula has a systematic flaw, that should be corrected. There’s been lots of discussion lately, here and at The Book blog, where several people, including Colin, have gone thru all the play by play, and rerun the numbers, and argued it out, to come to a consensus on what the propers run values are for each component. And VORP doesn’t have the right values.
    I don’t know how long ago BP did their calculations, or what there input data was, or how many people reviewed the results. I do know they’re not discussing it.

  4. dan says:

    You usually just leave us speechless.

  5. Pizza Cutter says:

    Me? Shameless? Nah.

  6. Colin Wyers says:

    Obviously there are limits to how well we can measure anything – baseball or otherwise. We’re only human, all too human.
    But. There is no reason to be wrong when there is no reason to be wrong. A fundamentally solid linear weights or BaseRuns-based system isn’t realistically any more difficult for a computer to implement than Basic RC or EqRAW. In edge cases – players with a lot of walks – you tend to see a very large disconnect between the way VORP values a player and a player’s actual run value.
    If that doesn’t bother you, I suppose I can’t argue with that. I just think that when it’s easy to not be wrong then you shouldn’t be wrong.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: