Look, I get it. Really, I get it.
You saw this headline and maybe you clicked on it because you just watched a game and your voice is sore because THAT UMPIRE MISSED SO MANY CALLS and now you’re rushing to the comment section to say umpires are awful and by god when are we finally going to have an automated strike zone?
Well first off, let me assuage your concerns a little bit by telling you it sounds like an automated strike zone is on its way, at least according to Rob Manfred, so don’t worry. But I’m here to tell you that, based on the data we’ve pulled together, it doesn’t seem like an automated strike zone would make all that much of a difference, because despite what you might think, umpires are really good at calling balls and strikes, and they’ve been steadily getting better.
Just How Accurate Are Umpires?
Before we dive into the data, I want to give a shoutout to our own Andrew Kurtz who first mentioned this idea and put together this data for me. He pulled the numbers, I just wrote the pretty words (and it’s debatable just how pretty those words actually are).
Moving on—accurately calling balls and strikes at the Major League level seems like an almost Herculean task to a layman like me, especially as pitchers are throwing fastballs harder than ever, throwing breaking balls with more movement than ever, and catchers are emphasizing pitch framing more than in past years.
Yet the data we’ve pulled has found that not only are umpires really good at accurately calling balls and strikes, but they’ve been steadily getting better.
Using Statcast data going back as far as pitch tracking goes (which is 2008), here’s how umpire accuracy looks year-to-year:
In 2008, 8.1% of pitches that should have been called strikes were actually called balls (which we’re calling “missed strikes” for the context of this article), and 7.7% of pitches that should have been called balls were called strikes (which we’re calling “missed balls”).
But in every season since then, the percentage of missed strikes and missed balls has steadily dropped. Last year, umpires only missed 3.3% of strikes and 3.9% of balls. That’s roughly a 50% improvement between 2008 and 2023.
Now, if you’re like me, the very next question you’re asking is “How accurate were umpires on the balls and strikes that were hardest to call?”
We figured that out too! And we used Statcast’s Attack Zones to do it, which look like this:
Unsurprisingly, umpires virtually never miss calls in the Heart zone. That said, they’ve still improved there—missed strikes in the Heart zone dropped from 1.1% in 2008 to 0.1% in 2023. Similarly, umpires literally never (as in 0.0% of the time) miss calls in the Waste zone.
Meanwhile, in the Chase zone, umpires do miss calls sometimes, but again, they’ve gotten significantly better. The percentage of missed balls in the Chase zone dropped from 1.7% in 2008 to 0.1% in 2023.
But it’s the Shadow zone you and I are interested in. Those are the borderline calls that are arguably the most difficult ones to call correctly. Unsurprisingly, it’s also the zone umpires have the highest percentage of missed balls and strikes in, but yet again, they’ve gotten significantly better at making accurate calls even in the Shadow zone:
The percentage of missed strikes in the Shadow zone dropped from 6.9% in 2008 to 3.2% in 2023 and the percentage of missed balls dropped from 5.9% to 3.8% in the same time period. And again, umpires saw improvement every single year in missed balls and strikes in every single zone between 2008 and 2023.
Now earlier I mentioned missing the balls and strikes that were the hardest to call, and that not only applies to where the pitches are located but what kind of pitches they are. It stands to reason that a big loopy curveball or a slider with tons of break on it is probably a bit more difficult to call accurately than a pitch that comes in straight as an arrow.
However, we found the opposite to be true. Here’s a breakdown of missed strikes and missed balls by pitch type for 2023:
Surprisingly enough, it was fastballs that umpires had a harder time with than breaking balls or offspeed pitches.
Now this is all well and good, but what about missed calls in crucial situations? That’s when an umpire’s missed call can really screw things up. If they miss a call on an 0-0 count, who cares, but if they miss a strike on a 3-2 count giving a walk instead of a strikeout, that can have a significant impact on the game.
Well if you’re noticing the trend in this article, then I’m sure you can guess that umpires have gotten better year-over-year at accurately calling balls and strikes in every count. But what’s really interesting is, umpires get better as they get further into an at-bat.
Here’s the percentage of missed balls and strikes from last year by count:
The highest percentage of missed calls are in 0-0 counts, but after that, umpires really crack down, and in crucial counts like two-strike or three-ball counts, they almost never miss.
We saw a similar trend in high-pressure situations as well, including in extra innings and when there are runners in scoring position. Again, missing a call in the second inning with no runners on is one thing, but missing a call when bases are loaded in the bottom of the 10th is a whole different thing.
But again, umpires really crack down in these situations. In extra innings, we found that umpires have improved year-to-year once again in missed calls, but what’s interesting is, they weren’t missing all that many calls in extra innings as it was. In 2008, umpires missed 0.2% of balls and 0.2% of strikes, but by 2023, that number was below 0.1%.
As for situations where runners are in scoring position, here’s the year-over-year trend for missed calls:
Yet again, umpires have improved and are currently better than ever at it.
I want to take a moment to talk about why we may have this perception that umpires are really bad at their jobs or have gotten worse when the data suggests otherwise.
There’s this theory that often gets discussed in politics that goes something like this: if you talk to your average voter and ask them about the state of the country, regardless of what party is in power, you’re always going to get a good chunk of respondents who say they’re concerned about the state of the nation or disapprove of the direction it’s going in.
And when you dive more into what they disapprove of, oftentimes they’ll respond by saying that crime is bad in the United States, often they’ll even say that crime has gotten worse over the years. I know I’ve heard a million conversations where someone says “oh well nowadays you have to be more careful, there are a lot of awful people out there,” suggesting there weren’t in the past.
The thing is though, generally speaking, violent crime has been on the decline for many years, yet many people think crime has actually gotten worse. Why is that? Because unlike 30-40 years ago, right now we have 24-hour news channels and the internet to amplify every single crime that we otherwise wouldn’t have heard about in the past—we’re more aware of crime, which tricks us into thinking there’s more of it.
So why on earth did I just ramble for three paragraphs about the public’s misconceptions about crime rates in the United States? Because I think there’s a similar phenomenon going on with umpires.
Now more than ever, we’re acutely aware of whenever umpires make bad calls. Thanks to being able to watch basically every single game in Major League Baseball and how prevalent clips of games and discussions of them are on social media, we’re hyper-aware of umpires’ mistakes more than we ever have been before. I mean, there’s an entire Twitter account dedicated to developing scorecards for every umpire (they’re an awesome account and you should follow them).
All of this means that even if umpires are making fewer mistakes than they ever have before (and as we’ve established, that is the case) when umpires do make mistakes, especially egregious ones, even if they’re pretty rare in the grand scheme of things, those mistakes get significantly amplified.
So ultimately, what’s the point of all of this? I’m not asking you to get a tattoo of Ángel Hernández over your heart, nor am I secretly an umpire in a fake mustache writing umpire propaganda.
I think it’s important to remember the data when we’re having the discussion about automatic strike zones because those automatic strike zones seem to be coming whether we want them to or not, and based on the data, it doesn’t seem like they’re going to suddenly be a panacea to all of our woes.
Credit where credit is due—umpires are consistently getting better at doing the exceptionally hard job of accurately calling balls and strikes, and right now, they’re better than ever at it.
Great article, but there’s a different conclusion worth considering. Let’s use the crime analogy. People worry about crime, but it’s hard to perceive a 50% relative drop in crime if, in absolute terms, it’s only down 4% overall. If I was worried about getting jumped at 8%, I’m probably still as worried about it at 4%, not half as worried. People aren’t perfectly rational.
The phenomenon of thinking umps are bad at their job may not be a misconception but instead revealed preferences. It turns out that that the maximum tolerance for missed calls is very low, lower than 4% and potentially lower than a human umpire can consistently achieve. Perhaps the best analogy is actually to air travel safety. People have zero tolerance for avoidable errors that can drastically effect their lives, or the game that they love. Actually, seems pretty rational to me.
VAR has truly been a panacea for soccer fans’ woes, at least concerning off-sides and scoring controversies. Like soccer fans, baseball fans can/will still find plenty to gripe about, but they shouldn’t have to bicker over fundamentals like balls and strikes.
First off, thanks for reading!
And second, I think that’s a fair push! Whether umpires are wrong 3% of the time or 1% of the time, I can understand the desire for them to be wrong 0% of the time. I’ll be honest, I’m not necessarily opposed to automated strike zones – certainly they wouldn’t be *worse* than human umpires – my main reason for writing this article was honestly surprise at just how accurate umpires actually are, and how accurate they are in crucial moments of the game. I think it’s just something worth keeping in mind that umpires are actually pretty good at calling balls and strikes and that automated strike zones might not make a *massive* difference in the aggregate.
automating the strike zone takes the human element out of the game. and we want to keep that. perfection is overrated. Roger Clemens came in the gas station the night he struck out 20 batters for the 2nd time. we had pay at the pump, which was new. all i saw was some guy with standing 7 ft over me and couldn’t see his chin, with tons of rings. it was late at night, he was going to Gedman’s house who lived up the street. a customer said ‘how’d you guys do tonite?’ and replied ‘we won’ and he left. the next day i realized he struckout 20 batters again. i forget if it was the SEA or DET game. it was an away night game though. and he flew all that way, and then decided to go in the gas station to pay in person, and the two idiots inside didn’t even see the game.
so whats the lesson here? well, it was probably wednesday, at night, the other team sucked, and everyone had a plane to catch, including the umpires.
when you take the human element out of the game you miss the small but important things in the game, like watching how Gordie Howe tapes his hockey stick before a game. it’s alarming how fast tradition is changing.
Thanks for reading!
I’ve always found the appeal of the human element of the game interesting, because ultimately when we’re talking about the “human element” of the game, we’re talking about the propensity for humans to make mistakes as opposed to machines. It’s interesting to me how incorrect calls could not only be a necessary evil of having humans officiate a game, but actually a preference to some people.
I say all of this not necessarily knowing myself how I feel about automating the strike zone. I can understand the appeal of the human element of the game – it certainly adds more drama, which in turn is typically more entertaining – but I can also understand the desire to have a game called accurately. Though given that umpires are generally pretty accurate at calling balls and strikes and the challenge system now exists for other calls, I think MLB has implemented the right safeguards to make sure games are called as accurately as possible.
i always wanted to start a website of baseball fans catching homeruns or foulballs. because it personalizes the experience. baseballs are funny, right? they are made by like one small company. and they are rubbed in secret dirt before they are shipped. and then you watch baseball, even players look at the ball trying to voir dire something.
or the great thing about life, everyone see’s something in their own way. this is hard science. spaceships passing by each other traveling at rest say, is he moving or am i? ppl with multiple personalities might be blind under one personality but not another – yet scientist can’t understand why. that’s the hard problem of consciousness.
stop taking away the human element of the game. but yeah sure, umpires piss me off sometimes but this is where were need to make a stand guys. and every time some player hit’s a homerun or pitcher finishes an inning and looks up to the sky. it’s the human element of the game. that we want to keep.
not sure what i am saying. for example where does the George Brett pine tar incident rank? or does it even matter? events in history that were important? ppl can say, i saw that. drama. and we will lose that stuff, while striving for perfection.
making mistakes sucks. .. but i live my life 9 innings at a time. during those 3 hours nothing matters. for those 10800 seconds im free.