Estimates are difficult for developers and product owners

And I'd add that the need for them is a sign they aren't worth doing.

As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.

That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.

In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.

Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.

From another thought-experiment-y perspective:

Say you have problem A to solve. Then either one of those is true:

1) it has been solved before, ergo by virtue of software having a zero cost of copying (contrary to, say, a nail, a car, or a bridge), so there is no actual problem to be solved.

2) it hasn't been solved before, ergo it is a new problem, and thus at any moment you may turn a stone and discover something that was not foreseen (whether they are rabbits, yaks, bikesheds, dragons, or what have you eldritch horrors) and thus of unknown cost.

Any task that cannot be obviously fit into one or the other can nonetheless be split into an assembly of both.

Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.

It's all make-believe.

I am in a project where we have to give estimates in hours and days.

Needless to say we always underestimate. Or overestimate. Best case we use the underestimated task as buffer for the more complex ones.

And it has been years.

Giving estimations based on complexity would at least give a clear picture.

I honestly don’t know what the PO and TL gains with this absurd obscenity.

Another reason is that figuring out what the software to be written should actually do, and how it should work, is work that is part of the project and the time it will take needs to be estimated.

As well as the actual development work that will result, which isn't known yet at the time of estimation.

I call it the "Persistent Incompetence of Software Development", which is another perspective on estimation, focused more on expertise. A chef that cooks pizzas, cooks the same pizza over and over again and becomes amazing at it. If you are a developer that writes the same code over and over, you are terrible at software development. A good software developer should always be solving new problems, as by the nature of software, once they solve a problem, they never solve that (exact) problem again. So we are persistently incompenent.

Which is why software development can't be estimated, as well. Because it is all, as you say, novel. With infinite error bars.

At this point, I can't take anyone seriously that believes software dev can be estimated.

There is very little novel about most B2B CRUD and internal bespoke apps that most developers are doing. The novel part if any is implementing the business vertical logic

"... anybody with any brains has already left town..." -- Bob Dylan

Anybody with any business experience has already isolated themselves from the certainty of software project failure, where "failure" is a euphemism for "late." So it doesn't matter if software can't be estimated.

This can be nerve-wracking to a beginner, but one gets used to it over time.

Which if you try to do - those agile people will kill you for it.

They wrangle a number out of you which goes into an user story estimate, which feeds into a Gantt chart they use to make their pretty powerpoints they present to upper management to say that the feature will make it into the Q4 release.

If you move this number around the whole estimation will crumble, not that it wont in real life but you deprave them of two things - an illusion of control and somebody to blame when things go south.

I have noticed the same for a lot of long-running software projects. The estimate is created at the start an never revised.

Projects can and will fail or run late; but heck; a 6-months projects cannot found late after 5 months and 29 days; things must be discovered early, so that the most important issues can be addressed.

> But then your various managers have the ammunition to decide what to do next - allocate more resources to the project, descope the project, push back the release date, etc.

In your view, is this a good thing or a bad thing for developers? In other words, are developers incentivized to update their tickets because it gives their managers "ammunition" to make project changes?

I find that this is usually more evidence of organizational dysfunction.

Sometimes, its developers who don't have enough ownership - someone higher-ranking filled out all the tickets for them and left no room for interpretation, so changes require trashing a bunch of irrelevant tickets and writing a bunch of new ones, making for more communication overhead and less time spent in development. Developers are disincentivized to update tickets since doing so just becomes write-amplification for more bureaucratic overhead.

Sometimes, it's a communication breakdown. Tools like Jira are necessarily asynchronous, and decisions about what to build are necessarily political. Politics and asynchronicity don't mix, such discussions need to be held face-to-face. If they're held face-to-face, then having developers update Jira is pure overhead. Developers are disincentivized from updating Jira because either it's a one-sided conversation (lag in getting notified that your ticket was closed and the work on it should be thrown out, PMs don't always follow through with issue links from the closed issue to the newly opened issues, so developers lack context on what decisions were made on replacements), which is emotionally draining, or the conversation happens outside of Jira, in which case updating the ticket is a formality/not actual communication, in which case it feels like a waste of time.

For most development work, I'd argue that developers shouldn't be making changes in Jira, their managers should be, on the basis of frequent, synchronous communication with the developers they manage.

Exactly. And I’ve found this is incredibly hard to get people to do, developers especially. No one wants to say a task is going over / taking longer than expected but will gladly run over. It’s hard for pms to resource if estimates are not updated as scope and complexity changes.

Also doesn’t help when estimates become due dates.

This is why stuff like Jira is so polarizing for many developers. It is an additional translation tax on me, the developer, to have to go constantly keep this tracking tool thing up to date so that the product owners/managers stay off my nuts. The burden of effort is placed on me to make their tool functional that they can have what they need to provide roll up status. This tool virtually never benefits me, it's always a time consuming sidequest that breaks focus and scatters important information all over a kanban board where tickets disappear and makes it really hard to cohesively work as a unit to get anything done. Like many orgs we've had PM's and stuff try to mandate expectations on when those updates should be provided, somehow completely oblivious to the human nature of development: you should never expect that a developer can simply hop in and out of moments of deep focus and suddenly provide a bunch of updates regularly when it's convenient for you, as if that has zero impact on their work. It takes a toll, full stop. I won't say no to it, but I do expect PM's to know that style of PM'ing is effectively asking me to make your job easier by populating your tool with updates that are useful to you and not me because I'm well aware you aren't involved enough to assess where things are yourself.

We've gone through so many cycles of how to use Jira well at my org where these frustrations are shared and we try a different approach, and we're finally starting to converge on the idea that this has historically been a little too lopsided requiring too much tax on the developer doing the actual work. We agreed on a new approach that has actually been pretty awesome: the product owners or managers that are trying to direct a body of work must be a little more in the trenches with us to have an overall understanding of where the different pieces are in a moving body of work. We don't expect them to understand the nitty gritty work in the trenches, but at the same time no more 30,000 foot view product managers who just ask for status updates at EOD. _Everyone_, developers included, is responsible for keeping documentation up to date as we go. So we have central working-bodies of information to find details without having to cruise thru 100+ jira tickets to find details we're looking for. The expectation is that they're engaged enough with development whether in chat or on meetings that if they were blindsided by an executive asking for an update, they could speak to it with some authority over at the water cooler without having to go check Jira. This has really helped weed out the lazy product owners/managers, has forced them to thoughtfully consider their meeting schedules, and has placed the exceptional ones in the pod of work being done and really added a lot of velocity and together-ness about the things we're pushing along.

This approach we're using now was born out of some hurt feelings from projects that didn't go so well & we had to have some real restrospective convos where everyone aired out their beef. Those are good convos to have, I think a lot of teams would find that people aren't deceptively trying to screw you over. Being encouraged to level set human-to-human genuinely is one of the greatest parts of working where I work. Always walk away from those types of chats learning valuable things: for the most part our product owners really do care. Not just about their career aspirations but also about _us_ nerdy and sometimes socially maladjusted developers. They look forward to working with us, and they want to make this as easy as possible for themselves but also for the developers. In the past they spent a lot of time in planning phases trying to scaffold out a project in Jira and attaching timelines to it so that their needs are met to provide predictable timelines to their bosses... but also with the hope that by plainly outlining work it sort of 2 in 1 satisfies our needs and will make development timelines a breeze. We've had to ask them to cede rigidity on that latter part because even the software architects admit the work being done is often a moving target. And when that target moves, maybe you realized you need to add a pivotal software solution to the stack, you can sometimes throw like 45 planned tickets into the dumpster. New ship dates need to be assessed. This was our reality check that we were all collectively shit at adapting to the dynamic nature of any given project. Now, our product owners decided that the expectation they have of their own role is that they understand this dynamic and are prepared and willing to make the case about why the shipping timeline must change. So there's actually a pain point solved here: don't break your back doing so much up front work to try and guess/capture what the body of work might look like only for it all to possibly get thrown away, involve architecture a bit more in the planning phases, but most importantly let's engage throughout the project and we'll try our best to have shared ownership/interest in making sure where we are in the project is broadly understood by everyone involved.

We're currently in phases of implementing a major platform right now and it's just night and day better and dare I say fun. We're still keeping Jira up to date, but the product owners and PMs are more or less managing this as it is a tool they find useful. Removing the "can you update this ticket please" 24/7 has forced them to be a little more involved and have the right chats, but also makes us developers happier to jump in and get it updated on our own volition because we also want to help them have an easier time. If my PM pings me and says "hey I'm looking at this ticket that's stuck in blocked, I just wanted to make sure we got an update from so-and-so about provisioning this credential so I can follow up if needed" I will likely automagically jump in and be like "still stuck, but let me update that ticket for you there's a couple key details I want to make sure our there for you before you reach out". There's an inherent "we're both interested in seeing this ticket through" here that doesn't strike a nerve with either party. Pretty much everyone involved both developers & non developers has a really solid read on where anything's at and we're all just talking a lot more. And for developers I find that it's really good, even if you've got them committed in one narrow body of work, to understand the larger pieces in motion. When they're in tune with the broader orchestration of a projects timeline, they tend to weigh in during unsuspecting moments that might tie seemingly unrelated pieces together. They might be assigned to work on x, but in a group chat about y they notice y has a dependency on x and they'll speak up and call out the need to test that they both work together. We've had a lot of great callouts materialize like this, and on a human-psyche level I think it snowballs & avalanches encouraging developer participation in a way that is really meaningful for PMs. It's interesting that Jira & the expectation that predicting development time in an arena of uncertainty was previously in the way of forming the group dynamics we have now. Jira, despite just being a tool, can really amplify a lot of bad outcomes when it's used by those who aren't fit to be near development, it sort of devolves into a two dimensional behind-schedule tracker that detrimentally impacts how team members on the ground communicate with each other.

And since we're talking a lot more there's just like... way more memes. And memes are important in any development timeline. We prioritize laughing through the pain together.

Sorry, but how does Kanban come into this?

From your description, SCRUM, could work just as equally.

Don't get me wrong, I'm a fan of Kanban, it's awesome for visually presenting where bottlenecks are for tasks, but estimations aren't a feature of that.

But SCRUM maybe, where people are having a sprint planning meeting maybe more what you're thinking?

Another reason kanban doesn't work for large projects is because you have to coordinate your cycles with multiple dependencies teams roadmaps and releases.

> technically it is irrational

Only if your company operates in a vacuum, without investors or customers

> E = Equilibristic estimate. This is based on success as 50% likely such as for critical chains and simulations.

I've found giving probabalistic estimates to be hopeless to effectively communicate, even if you assume the possible outcomes are normally distributed, which they aren't.

Three-point or PERT estimates are in the same vein, but are just an old and established business process concept and not a trademarked "project" from a (defunct?) consultancy.

https://en.wikipedia.org/wiki/Three-point_estimation

https://projectmanagementacademy.net/resources/blog/a-three-...

And yeah, they can be useful when you can must put numbers to things and can decompose the work into familiar enough tasks that your estimation points are informed and plausible. Unfortunately, in a field often chasing originality/innovation, with tools that turn over far too often, that can be a near-impossible criteria to meet.

This sounds like that if you don't trust a fortune teller, you can mitigate it by going to 4 different fortune tellers, and then somehow combining their predictions into your much more certain future.

So now you have 4 wrong estimates to work with :) To have some predictability you should have small features. That's the only thing that can give you strong estimates. No industry has solved giving strong and correct estimates - it's in the name, it's estimation!

Movie people are doing essentially the same thing over and over for 100 years. Everyone in the industry does things the same way. Their projects last only 1-2 years and are totally done once "in the can". In my experience this is almost never the case with technology based projects.

If your database of software projects has a thousand web browsers in it, you have to ask why all those programmers couldn't work together and write just ten or fifteen web browsers; maybe they're hobbyists who are reinventing the wheel for fun? If so, is that a good basis for estimating a new web browser?

The first time I worked on a database-backed web site, my team had written their own ORM, and while I was there we put together a thing so we could use the same view on more than one web page. The second time, we used web.py and no ORM. The third time, we used Ruby on Rails, which solved most of the problems we had on the first project I mentioned out of the box. Most recently, I used Django, which is roughly as convenient as Rails but easier to debug. I haven't tried it, but I'm guessing Claude Code could probably put together a usable Django site in a few minutes.

So achieving equivalent results requires very different activities over time, because each time someone does it, we accumulate not just knowledge but code that makes it easier the next time—thousands of times easier, if the novelty is small enough. This impedes estimation.

Often, too, the things we're trying to estimate are things like "add automatic Dart dependency support to Blaze". The time-consuming part of this is not breaking the rest of Blaze in the process. What reference class do we use for estimating adding new languages to Blaze without breaking it? Is Dart more like C++ or more like Go? These are questions for lawyers, not actuaries.

Maybe the videogames industry is different because they don't share code and don't maintain it? But it seems like an awful lot of games these days are getting written in Godot and Unity where 15 years ago people would have written their own engines... and problems that were major technical challenges for Quake are trivial now.

If someone wants ragdolls in their game, they can just license your code, right? They don't have to write it themselves. A dataset of how long it took people to implement ragdolls from scratch wouldn't provide any insight into how long it should take to integrate your ragdoll library.

My employer does, but only analyses the overage, so when a project comes in under estimate they think it's because it went smoothly, the reality is actually that estimates only ever tell you how good you are at estimating.

One interesting angle for me is that I am seldom given complete specs or requirements when asked for an estimate. Of course you ask questions to try to determine key information that has not been specified but often the answers are not available or fully reliable.

So any estimate has to include uncertainty about _the scope of the work itself_ as well as the uncertainties involved in delivering the work.

The natural follow on question when you present a range as the answer to an estimate is: what would help you narrow this range? Sometimes it is "find out this thing about the state of the world" (how long will external team take to do their bit) but sometimes it is "provide better specs".

This is what I do but I don't try to make it complicated with too many numbers. "2 weeks but there's a 10% chance something bad happens and it takes longer".

I have no problem if they just hear the "2 weeks" part. If they come complaining in 3 weeks I just say "we hit that 10%".

The other important thing is to update estimates. Update people as soon as you realise you hit the 10%. Or in a better case, in a week I might be able to say it's now "1% chance of taking more than a week".

> The numbers actually have agreed-upon meaning

Theoretically, yes, but some managers go blank when given a hard concept like a probability distribution.

I do like the blind estimation aspect, but I don’t like:

* the arbitrary usage of the Fibonnaci sequence

* a non-clear conversion from complexity to time. Complexity and time aren’t always correlated. Some things can be easy but take a long time. Should that be a 1 or a 5?

Let’s just cut the layer of terminology, in this difficult task, and estimate in units of time directly.

What's wild about software is we don't do 1/10th as much planning as "real" engineering projects do, and also find ways to create problems akin to the ones they run into from interfacing with an unpredictable real-world environment, by inventing ways for all our platforms and libraries and such to be insanely unstable.

If you're not subject to the batshit insanity of the broader software market, and you crank the planning up a little closer to what "real" engineering does, software delivery gets extremely predictable. See: NASA's Space Shuttle software development processes.

(There are actual, not self-inflicted, problems with software development that "real" engineering doesn't see, though—for one thing, you can't on a whim completely restructure a building that's already been built, and for another, you generally don't have to design real-world engineering projects to defend against the case of intentional sabotage by someone who's just e.g. using the bathroom—it may happen, but it's rare and short of wacky Mission Impossible type plans, busting a water pipe in the bathroom isn't going to get you access to super-secret documents or anything like that)

Physical real-world projects include a buffer for this, called "contingencies" or "change orders" so that if a requirement changes or they discover something like previously unknown site geology that will require changes to the foundation they can absorb it. Based on a large history of similar projects their estimates are usually pretty good but occasionally they will run over.