Good Decisions vs Good Outcomes
Lately I’ve seen at least two product leaders I look up to make the same (very understandable) mistake — conflating decision quality with outcome quality. The mistake is very subtle, but it looks like this.
“X product didn’t work, which means we made a bad decision.”
They may have made a bad decision, I don’t know enough of the details to be sure of that, but the outcome being bad doesn’t mean the decision was bad. The outcome being bad only means the outcome was bad. It’s important to be able to make this distinction because as Product Managers we make and review decisions constantly. Which feature should we prioritise? Which stakeholder do we need to talk to next? Is Spotify’s model good, or did it just work for them?
If we can’t tell separate these qualities, we’re committing the sin of hindsight bias.
It is possible to commit no mistakes and still lose. That is not a weakness. That is life.
Jean-Luc Picard, Star Trek
Allow me to break this down into detail, and provide some examples.
A decision is a choice, made by deliberation. Typically, in Product Management, we’ll consider a range of option, ideally look at some data or stats which feed into our decision, and then we pick which thing we’re doing to do.
One of the key things to know about the decision making process is that the data we have is the data we have at the time. That sounds tautological, but it’s an important distinction. If you want to make better decisions over time it’s important to be able to reflect on decisions past, but we can only make the decision based on the information we had then. Evaluating that decision using information we gained later is called hindsight bias (this is going somewhere, I promise.)
There is a gap between evaluating which option is best, and that option then having a measurable outcome on the world.
In software, that idea is executed on by a team of engineers, designers, markets, salespeople, etc, etc. It takes time to go from an idea to a reality, and in that time, the world changes in some, generally small but perceptible, ways. Some fairly random effects play into that outcome too, which might include subtle effects like social media algorithms, news cycles, or hidden information coming to light.
The point here, is that a large number of things influence the outcome, including but not limited to, the decision we’ve made.
I’m going to continue to call these effects “random”, but specifically here I mean “things beyond our reasonable control or ability to forecast.”
When we put these things together, we come to an interesting conclusion — there is no perfect relationship between good decision making and good outcomes. It is possible to make the best possible decision and still reach a bad outcome. It’s possible to make a bad decision and succeed anyway.
To be explicit — good decision making, even perfect decision making, correlates with good outcomes but there are other factors which are important.
I don’t say this to excuse bad outcomes. We’d all be doing ourselves a disservice if we assumed all bad outcomes were due to these other random effects, and in fact, in principle, I believe we should assume that our decision was the cause of bad outcomes most of the time. (The reverse is also true — people and processes aren’t necessarily successful because of their decisions, but we shouldn’t assume it’s pure luck either.)
The reason I’m talking about this is for the purposes of reviewing our decisions — when we review our decisions, we need to be able to bring more to the table than “because the outcome was bad, we made a bad decision.” That is hindsight bias. What we should be doing instead is looking for which pieces of information were available which we ignored, misinterpreted, or weighted incorrectly.
An Example
This has all been very conceptual, so let’s use a simple example to demonstrate. (This is real, by the way.)
My friend and I decided to go out for dinner and we were evaluating restaurants. We came across a trendy Sydney restaurant (whose name I will with hold because this isn’t a restaurant review). This restaurant was newly opened and was rated very well — about 4.3 stars on Google Reviews, with over 100 reviews despite being only open for a few weeks. We needed a reservation, but only needed to wait a week or so to get one. It was self described as “modern Chinese food” and we both like Chinese food.
We decided to go to the restaurant, and had an appalling experience. The courses came out at weird times, every dish tasted the same, and we were seated next to a group of 15 people having what looked to be a rowdy bachelorette party.
Safe to say, we didn’t have a great time.
Despite the outcome here being pretty poor, I would argue that we didn’t make a bad decision. We used the information we had (quality and number of reviews, type of cuisine) to assess if this was something we’d probably enjoy. A number of random factors (i.e. variables that were out of our reasonable control or ability to forecast) turned what was probably a good decision into a poor outcome.
Now, given the same information again, would we make the same decision again? Almost certainly. Of course, in reality it’s fairly hard to make literally exactly the same decision again (if assessing the same restaurant again we now have the new data “I had a bad time last time I went there”.) If we were to assess a new restaurant with a 4.3 rating on Google, hundreds of reviews, and of a cuisine we like, would we go again? The answer is still yes.
Software is much more complicated than choosing where to go for dinner (mostly). The decisions we make as PMs tend to be much more fluid and lead to iteration on a topic or problem space until we reach a good outcome (or run out of money/patience.) The same thing applies for each loop, however. A decision which leads to a bad outcome becomes data for the next decision, but doesn’t make it a bad decision. Being a bad decision is a separate quality.
Concluding Thoughts
This post has been fairly self-indulgent, but being clear about the difference between a “bad decision” and a “decision which lead to a bad outcome” (and conversely a “good decision” and a “decision which lead to a good outcome”) is essential to be able to review decisions effectively.
A decision with a bad outcome might lead to all sorts of other second order things happening — looking for certain new information next time, weighting information differently, using a different process for looking at the available options —but what we can’t do is use that information to assess the original decision quality.
