Software cannot be estimated the way your organization wants, and it's time to stop trying
Managers and organizations want to know exactly when features, bug fixes, and other work will be done. Many have not been software engineers themselves, so they ask for exact dates. Sure, you can be off by a day or two—maybe!—but more than that, and it's a problem. After all, their boss needs to know what to promise customers. What's so hard about knowing when you'll be done?
Sadly, software cannot be estimated that way, and we need to stop pretending otherwise. It's a myth—a pervasive one—and perpetuating that myth only perpetuates its harm.
Sure, if a task is almost identical to some previous task, a precise and reliable estimate really can be communicated. Unfortunately, that almost never happens. If the work ahead were so similar, someone would have done it already using copy and paste. If it's similar and the engineering team wanted to set themselves up for success later, they'd refactor the system. That refactoring itself can be a hazy fog of unknowns. We've all been there many times.
The problem
I once worked with someone who liked to compare software development to parcel delivery: choose a day when the package is expected to arrive. It's okay if there needs to be a delay, as long as that delay is communicated clearly. While the flexibility with delays was genuinely appreciated, the broader analogy soured on me over time. In truth, estimating software is nothing like estimating package delivery. The former involves doing something that has never been done before, while the latter is completed billions of times each year. Does Amazon have an advantage when forming estimates? You'd better believe it.
Software development is one of the only professions where we are asked to accurately state how long it will take us to do something that has never been done before in the history of the universe.
—Unknown, from a screenshot on X
How long will it take for Amazon to deliver a board game to Zara in Cincinnati? Well, probably about as long as it took last time. Yes, every delivery is different, but there must be millions of deliveries of similar items to nearby locations with similar weather forecasts and other pertinent factors being similar or identical. How long will it take? Probably that long.
By contrast, how much time will be required for me to change code I've never seen? I truly have no idea. How about adding, say, karaoke support to a streaming music app, one with millions of lines of code I've never read, where myriad caveats exist for different software clients, different spoken languages, and different copyright laws. How long will that take? Except in extremely broad terms, your guess is as good as mine. It will take more than one week, and it can probably be completed in less than one year. Beyond that, there's not much more that can be promised, at least in the beginning. More on that in the next section.
No, estimating software development is nothing like estimating like parcel delivery. It's more like estimating the pace of scientific progress. We simply don't know what we don't know. We can't know until we do it.
As much as possible, organizations should communicate that software will be ready when it’s ready. If some software must be released by a certain date, the feature list should not be guaranteed. I know this is easier said than done, believe me, but reality is what it is. Software simply cannot be estimated the way organizations imagine. Some higher-ups, especially those who have not been software engineers, may find this outrageous. They may demand exact estimates regardless. To them, I would say, “Fine, but let’s be honest, how’s that working for ya? How often are you given estimates that actually turn out to be correct?”
There's a reason you're reading this blog post. Something isn't working, and it's not about discipline. It's not about communication. It's not about skill. You're on the wrong path. Skeletons and vultures surround you. It's time to turn around.
Thankfully, there is a better way. Your manager may not like it at first, but over time, they may come to see that estimates of this type are actually useful, unlike the estimates they are accustomed to.
The solution
To explain this kind of estimation, I need to define two terms. First, I'm using the term accurate to describe any estimate that is correct within a given range of dates. Defined this way, even a very broad estimate can be accurate. For example, an estimate that a feature will be completed some time between March 1st and December 31st would be accurate, as long as it turns out to be true. Precision is another matter. That feature estimate isn't very precise, but it would be more precise if it predicted a completion date between, say, April 1st and July 1st. This is how Rapid Development defines the terms, and I think it’s a good approach. In fact, most of this advice is lifted from Rapid Development. I recommend reading it. It's a tome, so focus on the sections that are most pertinent to your needs.
Crucially, estimates can and should become more precise as time goes on and details become more clear. If a team begins with an estimate of some time in Q1 or Q2, but the feature is more difficult than expected and they realize they need to revisit the database schema, perhaps the estimate could be narrowed to some time between May 1st and June 15th. If the team is nearing the end and the final code reviews go more smoothly than expected, maybe the estimate could narrow to some time between May 16th and May 31st. If a critical issue is found elsewhere in the codebase, taking the team off the feature for a while, maybe May 29th becomes the most likely date of completion. The point is, there's no way to predict those problems in advance, but they can inform better estimates over time, as long as the estimation method supports that. Did you instead randomly pick a day three months in advance? Too bad. You're screwed.
Reality-driven estimation
So how should we estimate? Strive for accuracy, start with low precision, and seek greater precision over time. As another example, you might consider communicating a project end date some time between Q2 and Q4, then later refine that to some time in the fall, then some time in October. Finally, at the very end of the project, you might be able to promise a specific week or even a specific day. That is an approach to estimation which acknowledges reality. That is an approach to estimation which acknowledges that surprises happen.
Do organizations want to hear this? Unfortunately, no, I don't think so. It's a tough sell. Organizations want certainty, and they want it now. Good luck telling your boss that you don’t know exactly how much time and money you're going to spend. Even the project management apps we use typically require that an estimate be recorded as a specific date. Talk about encouraging bad practices.
Pushing back is hard, and as someone who was raised to never question authority, even when that authority is obviously wrong, I happen to be particularly bad at it. That's too bad. Our bosses pay the bills, but we do the work. It's time we start saying, “The work can't be done that way, and if I were to tell you otherwise, I'd be lying to you.”
Reality is what it is, and the best we can do is provide estimates with gradually improving precision. Organizations can pretend otherwise, and countless surely will, but again, I would ask, “how's that going?” Organizations ignore these facts to their own detriment, at the cost of time, money, and frustration. I wish them luck.
For everyone else, consider embracing reality. It's not perfect, but it's the best we can do.