“Those players who do well on some systems but not others will end up in the middle of the pack which I think is probably appropriate in most cases.”

I think that’s true for a great many players; the variance is fairly tight and centered around zero.
But I’m still concerned about the huuuge variance seen in a number of OFs. I’m just not comfortable averaging a -35 and a +33 or whatever Ichiro’s actual numbers are. Same w/Grady Sizemore. One of these numbers paints both players as hideously overrated when you factor in defense, and the other paints both as MVP candidates. When we’re trying to get reasonable estimates of team defense or OF defense specifically, that’s pretty important. Again, there’s zero doubt that what jinaz and tiger337 have done here is helpful and moves the ball down the field (and since I’m lavishing praise here, I can’t believe we haven’t even talked about Justin’s fielding translations with THT data here; I use it a lot). I know Derek’s said that if a system doesn’t work for one guy, it’s not reason enough to discard it. That seems fairly reasonable.

And yet, as M’s fans, it matters a hell of a lot if Ichiro is a lifesaver or an anchor in CF. It matters a lot if park factors or something else will make *every* RF (except ichiro) look like crap. The team can’t keep hemorrhaging runs through poor DER, and the team can’t keep trying out new OFs only to see each one worse than his predecessor. These data should clearly help the FO make decisions about who might perform well, and yet, we’ve all been surprised. I don’t think anyone thought a healthy Jose Guillen would’ve been one of the worst RFs in the league, and I know I didn’t think Wilkerson would’ve been at least as bad.

In summary – we need to quantify/explore the BIS/STATS differences, especially for the OF. We need to figure out how much natural variation to expect from year to year in these numbers. We’re getting closer, but we’re still at a point were we need 2 or 3 years of data from multiple sources and then we need to hope that those sources don’t contradict each other. That’s tough for teams to use to really inform their decision on extending, say, Wlad or Yuni a few years ago.

I know a couple years ago I was able to look at a couple systems and figure out that whatever one of them was doing at one position, it was just wrong — compared to all the other systems, they might as well have been randomly ordering players.

That’s extremely useful information when you’re doing evaluation.

Derek didn’t tell you? Huh. We all got emails the last few days with a Paypal link in it. Weird.

noble of you to refrain from rickrolling that Paypal link

Plus, I was pleased to see that the correlation between the fan fielding survey and numerical measures (Justin’s study) was reasonably strong. That gives me a little more confidence that the measures are working.

@18 – thanks. I agree that we should be skeptical about defensive metrics, but that’s part of the reason I (and Dave, and Sean Smith) advocate taking an average of the best available defensive statistics. If there’s agreement, you’ll get a solid number +/-. If there’s disagreement, you’ll estimate league-average or so, which is a good baseline to estimate anyone’s performance.

@19 – thanks for the link on the catcher work. It’s admittedly very incomplete, but it gives us a baseline to work from.
-j

Dude, that’s a scam. The checks should be going to Dave’s exiled Nigerian uncle.