Jamie Bernardi

Guest co-author: Ben Cottier

I’m often asked to give feedback on others’ ideas. A conversation between Ben and I helped me identify a common mistake I make when evaluating others’ ideas: I often overlook very valuable steps along the way to the end product, and only evaluate the end product itself. 

In this post I want to encourage myself and others to evaluate others’ ideas by their parts.

---

Startup founders tend to focus on the big idea

A common problem for startup founders and project ideators is to hyper-focus on the end vision. This can incur lots of work sunk into a project that fundamentally won’t work, or adding too much complexity early on when you could be adding value.

This is the topic of many books and articles advising project ideators: the modern literature will quickly encourage you to break down your own ideas into small, deliverable chunks.

I have seen much less written on evaluating others’ ideas, yet I feel a similar lesson is important. First let’s discuss the problem with evaluating ideas.

Problem 1: A grand vision is hard to evaluate

Any world-changing idea sounds both good in principle, but also unlikely in principle. This is because:

1. Most big ideas fail.
2. If you’re pitched on a vision that depends on lots of parts going well to succeed, you’re probably right to discount it based on the low likelihood of it all coming together in the way it’s envisioned right now.

But of course, some speculative ideas do succeed. If you naively went by the prior, you’d miss some great ideas, and might even shut down potentially great ideas with your poor advice.

The key issue is that, generally, the further out a step is in a plan, the less chance it has of succeeding as-planned, therefore the harder it is to be excited about it. This stands to reason: there will be a lot more to discover about the world between ideation and reaching the later stages, so it’s best not to be too specific about the far-out details in the beginning.

Problem 2: When asking you for feedback, founders will focus on their vision

Another issue is the way the project might be being pitched to you.

A key step every founder knows about is to get feedback on their ideas. Founders tend to lead with their vision of the final product, hence the problem the startup literature tries to address.

It’s tricky for evaluators to spot the steps along the way because it’s likely the ideator will only briefly mention them. If they are excited by their vision, it’s likely they’ll downplay and quickly pass over the steps it’ll take to get there. Instead they’ll draw your attention to the conclusion rather than the boring steps they’ll have to take to get there.

So flip the advice

I think we need to more explicitly flip the advice given to founders over to evaluators.

Since a founder – especially an an amateur founder – is unlikely to have chunked their idea, an evaluator should try to do it for them. Any grand vision with a chance of success will consist of lots of value-adding steps and products along the way. The evaluator should anticipate the end vision shifting dramatically depending on how those steps go.

As an evaluator, you should try to spot the steps along the way or – better yet – encourage the founder to figure out what the initial steps are for themselves and present them.

Since nearer-term ideas are easier for you to evaluate and predict the outcome of, you’ll likely do a much better job of being encouraging and maybe imparting some actual useful advice about how to succeed. In fact, imparting the advice to chunk up the idea into steps might be the most valuable thing you can impart in itself.

Example: missing a useful step because I thought the final step wouldn’t work

I was once pitched an idea that was along the lines of “use GPT-3.5 to read every nations’ AI policy documents and rate them in terms of how comprehensive they are for preventing extreme risk from advanced systems”. I thought that rating policy documents on addressing AI risks in this way would be very convoluted and hard, and probably wouldn’t work (as it was presented to me).

6 months later, someone published the AI Policy Portal which simply compiled all the documents published about AI governance from each nation. This was received extremely well. It was also the first step in the idea that was pitched to me.

Looking back, I missed this step in the original project because I got too hung up on whether the language model evaluation part was going to be accurate or useful. D’oh!

You might just think I’m a bad advisor and you wouldn’t have made the same mistake, but I hope that the advice in this post will help me to do a better job next time, and it might help others too.

Other adjacent advice on this topic

I looked around to see if anyone who I thought was likely to have written about this had made this point before.

I was in two minds about whether the above points are all that profound – an experienced ideator will likely transfer their experience chunking up visions into valuable steps whilst coaching others. The post largely calls for coaching amateurs.

There is some similar literature discussing ideas and how to evaluate them, but they’re all slightly different to the above:

• Zero to One: great change can only happen if someone dares to dream about something preposterous, and goes after it.
• Crazy New Ideas: if an idea sounds preposterous to you, but the person you’re listening to might know more about the topic than you, ask questions rather than shutting it down.