Schwarz on PECOTA in NYT
Jeff · November 13, 2005 at 12:35 pm · Filed Under General baseball
Don’t miss officially endorsed Alan Schwarz writing about Nate Silver’s PECOTA player forecast system in the New York Times. Read it quickly, before it falls behind the terrible “Times Select” wall of paid content.
Comments
23 Responses to “Schwarz on PECOTA in NYT”
It’s nice to see that Dan Evans, in the last paragraph, at least implies PECOTA is something the Mariners use.
I found it interesting to hear about a couple of the scenarios for Burnett, too, even if the author may have selectively mentioned only the positive predictions.
He gives PECOTA enormous credence in this article, maybe more than it deserves. His objective is, of course, elsewhere.
His description of GMs and their assistants in febrile search of better projections in worth the whole.
I don’t remember ever seeing Schwartz in the NYT before.
Dan Evans quote makes me think of the radio ad for “The Nation” magazine. “You mean you could get fired for reading the Nation?…No Senator, just for being seen with it!”
Haha…I don’t suspect that BB is like that, but its funny…well, at least to me.
My only problem with PECOTA is the circular nature of some folks’ implicit trust in the thing.
“Ichiro hit .350? But PECOTA said he would only hit .325. Must have been a freak year.”
How about, “Oops, PECOTA was wrong”?
#4 – Interestingly enough, even though PECOTA was very close in its prediction for Ichiro in 2005, Silver mentioned recently that he believes the system is poorly designed to handle a player such as him, which is why he is adding in groundball/flyball tendencies.
Also, I had no idea Nate Silver is only 24. Isn’t he running BP now?
“My only problem with PECOTA is the circular nature of some folks’ implicit trust in the thing.”
So, is that really a problem with the tool or with the people using the tool?
PECOTA would come off a lot better if people paid more attention to the Similarity Index.
I don’t have an in depth knowledge of PECOTA, but given what I have read on this site and in that article, PECOTA wouldn’t say Ichiro would hit .325, correct? PECOTA would give you a mean expected value and probably the value for two standard deviations from the mean. I’m guessing (and now my statistical knowledge is running thin) that you would also get the information needed to know whether that is a bell-shaped curve or not, etc.
So a “point” forecast of .325 would not be worth quoting without the rest of the qualifying information to go with it.
[why do people think it’s funny to imitate other users? First Corco and now Bela? Come on]
PECOTA gives a range of performances based on the comparison with other players. So you get something like
Ibanez – 90% .400/.500/.600
…
50% .285/.340/.450
…
10% .100/.115/.220
And a “weighted mean” which is usually what’s quoted as “the projection”
Essentially, the precentages are the number of times out a 100 the performance is at or below that one. So the system thinks that there’s a 10% chance Ibanez is above that top one, and 90% he’ll be worse. And for the 10% one, same deal (only reversed).
There’s also, and as Jeff S points out, an important and oft-ignored piece of information, the “similarity score” which offers a caveat on how good the comperables PECOTA found were. For a pretty normal player, it’ll be confident that from that information it knows the likely range of performances that player might have next year, and so on. For someone like Ichiro where there aren’t any players that are comperable, this is where PECOTA says “look, this is my best guess but the data’s really poor so everyone be cool”
PECOTA, which I like a lot, isn’t even universally accepted at Baseball Prospectus. If you read BP’s yearbooks, you’ll see several dozen entries each year that read something like:
 “We think Player X will beat his PECOTA projection handily.”
 “PECOTA is strangely pessimistic about Player Y’s chances to bounce back.”
 “PECOTA is optimistic about Player Z’s chances; we’re not so sure.”
All of which goes to show that PECOTA is one more tool for evaluation among many, not the tool to end all tools. NOBODY thinks it’s gospel and when we lash out against it as though it’s assumed that it is gospel, we’re tilting at straw men. We have no right to piss all over PECOTA unless we have a better idea.
That being said, Nate Silver is a flippin’ genius, and it’s good to see him get his mainstream due.
PECOTA, which I like a lot, isn’t even universally accepted at Baseball Prospectus.
That’s just not true. PECOTA is limited both in the way it picks its comps and the way it calculates things. It’s a pure, limited stat program, and that’s all it is.
If someone’s badly injured with a wrist injury that saps their power for a year, and as a result PECOTA picks 10 guys who had severe power drops in a year and shared the other major characteristincs it weighs, then it’s likely to predict a further collapse, because those other guys didn’t bust their wrist but instead were at the end of their careers.
That’s both a strength and a weakness: it doesn’t look at anything else and doesn’t care to, but when you disagree with it, it’s like saying that a band saw isn’t the right tool to screw in a bookshelf: it’s an understanding that not every tool is right for every task. It doesn’t mean you think less of the band saw.
Re. #4: Remember, though, that Pecota never says, “Player A’s performance will be X.” It only says, “Here is how likely Player A is to put up performances like X, Y, and Z.” Therefore, one player’s performance can never prove Pecota “wrong.” The only thing that could make Pecota wrong would be if it consistently over- or under-predicted either the performance or the volatility (i.e. how likely they are to have a result much different from the middle-of-the-road prediction) of a significant group of players.
(In particular: If nobody in all of the major leagues exceeded his 95% Pecota performance level or fell short of his 5%, then Pecota would be “wrong,” because it would have over-predicted the volatility of future performances.)
For what it’s worth, I do have one concern about Pecota: It seems to produce rather odd results (in particular, overly optimistic ones) in its weighted-average forecasts for the major league performance of minor leaguers who are expected to get less than a half-season at the major-league level in the upcoming year. I suspect that this probably traces to a method that associates the best performances with the best playing time, and then does a weighted average on playing time.
However, for players who are likely to play a full season in the majors, Pecota does a good job as far as I can tell.
Well yeah, it’s not a perfect analytical tool. It’s got to be used along with other things such as scouting reports. I found it interesting the article mentioned the same shortcoming that I’ve seen one of the USSM authors state here. Since it compares similar players to reach a conclusion, it is kinda hamstrung when trying to analyze someone like Ichiro. I believe the comment was it has to pull Ichiro down so far to compare him to other players that it distorts the analysis.
Nate’s description of PECOTA’s accuracy with Ichiro’s 2005 was excellent. He thinks it was a fluke.
He wrote an entire article about how some players were too unusual (especially given that PECOTA only examines post-war players), and thus PECOTA was always going to get them wrong because it simply lacked relevant data. Here it is:
http://www.baseballprospectus.com/article.php?articleid=3497
(he mislabelled Bonds in the chart – that data point is actually Jim Edmonds)
But if it’s predicting a season for a player coming off an injury-free season, and he has a Similiarity Index over 50, then it’s a great tool.
Is there a thought or a sentence missing from your response to Jim Thomsen, DMZ? Becuase, even though you quote his assertion, it reads like an arguement against something else entirely.
I found the most interesting PECOTA individual player forecast of 2005 to be for Calvin Pickering. Even his 10th percentile forecast was was solid at .237/.355/.475 and he had a collapse rate of only 6.2%. Predictably, the Royals gave him only 27 AB’s where he encountered an unfortunate beginning of the season slump, and was then sent down to Omaha where he hit .275/.384/.528 for the year.
The Royals have given up on Pick. The M’s should find out what type of peanuts the Royals prefer and secure the best LH pinch hitter, part-time DH, that can be had for minimum salary.
The one time I saw him last year was a Felix start in Tacoma. He didn’t look good against Felix but then he wouldn’t have to worry about that if he was on the M’s.
As an aside on the PECOTA topic, doing a google search for the words 2005 and pecota turns up Joel Pineiro as the second result.
Just out of curiosity, does anyone know what Beltre’s PECOTA projections looked like last spring?
Beltre 2005 Pecota projections:
.255/.303/.413 603 AB’s – Actual 2005
.241/.295/.422 490 AB’s – 10th percentile Pecota
.259/.315/.453 509 AB’s – 25th percentile
.275/.333/.482 528 AB’s – 50th percentile
.279/.337/.486 529 AB’s – weighted mean
JP:
Schwartz is a regular contributor to the NYTimes Sunday Sports section. He has an article in there about once a month. But it’s typically buried 3/4 of the way through the section right before the car nonsense.
On Silver’s age, I think Schwartz meant Silver was 24 when he invented PECOTA several years ago. I’d guess he’s in ups upper 20’s now.