Neyer discusses wOBA and some Sabermetric philosophy
November 26, 2008 6 Comments
From the “stuff we’re reading” file: Rob Neyer over at ESPN has an extended post about embracing wOBA (a creation of Tom Tango) and a few other issues worth commenting on. Neyer’s one of the more Saber-savvy gentlemen over at ESPN and he’s worked extensively with Bill James, and it’s cool to see these concepts creeping into the mainstream. (h/t: The Book).
But don’t stop reading there. Rob goes on to discuss another issue which are worth a little discussion. The first, and more important, is the issue concerning Baseball Prospectus and the fact that while they have been the industry gold standards for a while in terms of player metrics, their flagship stuff like PECOTA, VORP, and WARP are proprietary. They do have the right to keep whatever they so desire under their hat (and I, for one, pay my annual subscription dues to read it), but Neyer brings up an interesting hypothetical:
But that science-vs.-enterprise dynamic can be tricky. The methodology behind BP’s metrics is not, to my knowledge, peer-reviewed. If one or two people make a big mistake, would anyone else know? Now, let’s jump ahead and say that two or three years down the line, the big mistake was discovered internally. Would BP announce to the world that all those numbers over the previous three years had been wrong? Or would the guys running the show decide that the loss of credibility (and potentially, revenues) isn’t balanced by the loss of integrity?
One of the charming things about Sabermetrics is that it’s grown up as mostly a field of amateur hobyists who have too much free time. If I screw up on something here at StatSpeak, I know. (It’s the only time I ever get comments!) But that’s the beauty of peer review, and I love that sort of dialogue for its own sake. The luxury that I have is that I can screw up and it means nothing more to me than a red face over having blundered. For what it’s worth, I do trust that the people at BP are smart, dilligent people who double-check their work and each other’s work, but they are humans and humans make mistakes.
Rob points out that “Science works best under blue skies, with little thought of green.” And here’s a philosophical point that is becoming very real in Sabermetrics. Sabermetrics has started to see that green influence creep into it. BP started out as a buch of people throwing around ideas on Usenet. Now, they’ve got a business model and some of those same folks have gone corporate.
In fairness to BP, they wouldn’t survive if they weren’t putting out something good, and while we may not know the exact machinations of how something like VORP works, we have a general idea through what they have said and the concept makes sense. Plus their stuff generally passes the “smell test” and things like PECOTA projections have been shown to correlate pretty well with actual performance. It’s not that I worry about their stuff being bad/wrong/misleading, it’s that I worry that the science part of Sabermetrics might stagnate. When I developed OPA! (my fielding system), I was writing the syntax that did all the calculating at the same time that I was posting the articles on it. I got some feedback through that process that I incorporated into the system. OPA! is a better measure because of it. Imagine for a moment that PECOTA were open-source. Someone out there would probably figure out a way to make it better. It’s not. But, that’s the trade off that you make in a closed source system. Nate Silver, who probably has poured hours on end into that system (as well as any of the BP folks who have helped him) certainly would see no return on his investment of time, at least financially. Would he have created the system to begin with had that incentive not been there? I don’t know Nate, and I don’t know what his answer would be. Maybe he would.
This is a grown-up moment for the field. This is a philosophical turning point and one that, like most philosophical issues, doesn’t have an easy answer, or even an real answer. Do we want to be scientists in the strict sense or are we OK with the idea that some people will keep a few secrets and charge a few dollars to see them?