AI-generated Proof of Concepts (PoCs) make it easier, faster, and cheaper to turn ideas into demos… but from there you need a process to kill your Frankenstein creations.
I’m dating myself, but I used to love Pinterest Fails.
No clue what I’m talking about? Observe Exhibit A.
On the left, a beautiful cake for the cat lover in your life.
On the right, nightmare fuel.
Or what about Exhibit B.
On the left, a Hallmark photo.
On the right, a high school graduation shame photo.
Few things juxtapose expectations vs reality better than this series.
Which brings us to the recent debates around AI and its ability to generate hundreds of ‘proof of concepts’ rapidly.
Just like Pinterest Fails, we all have visions that are beautiful and perfect in our mind… but become epic fails when they meet reality. It’s best to have a strategy for dealing with these failures when they occur (beyond using them for self-shaming social media posts like the ones above).
If something is cheap, something else is expensive. #
This is known as the tradeoff principle.
Once AI made it simple to convert every wild-ass idea into a functioning demo, there were many second-order effects.
Yes, many of these 2nd order effects were positive!
Dramatic reduction in costs, time, and stress in delivering new features.
Prototypes give a more accurate feel/experience than mockups or written specs.
Empowerment for non-technical team members to contribute.
However, there is a laundry list of negative effects. Here’s a laundry list of complaints from people in my circles.
Lack of Followthrough. People will take a PoC to 80% in a few minutes, then stop. The dopamine rush of running 2-4 more experiments in parallel is more exhilarating than spending hours/days to polish it to 99%.
Decision Overwhelm. AI can generate volumes of code and docs that someone has to review. And these reviewers are also busy generating volumes of code and documents. At some point, we are over ourselves with things to review.
Sunk Cost Fallacy. Someone spent time building something, so there can be an inherent psychological pressure to keep going just because it’s already built.
Frankensteining. Features used to take time, so things felt more thought out. Now you can bolt on 10 new features a day.
Make work. Since devs have extra capacity, are they just throwing stuff at the wall and seeing what sticks?
Coordination. It’s difficult to keep the team and customers in sync with the amount of changes coming through.
The Downstream Effects #
So if AI made building cheap, what became expensive?
Human review, discussion, and decision time.
We’re buried in AI-generated everything, so we go into a sort of survival mode and abandon our process disciplines.
We let PRs stack up because who has the time?
We greenlight things we don’t have time to review.
We don’t follow through because 4 other agents are vying for our attention.
We don’t push back on a colleague because “at least they are being productive.”
We stop trying to keep up with Slack, emails, notifications, tickets, etc (white flag waved)
Again, your mileage may vary, but these are the types of challenges I’m seeing people discuss in the channels.
One Solution: Kill Criteria #
Annie Duke’s book Quit was written in a pre-AI world, but it’s more relevant than ever.
She tells the story of Google’s Moonshot division. This team is looking for the wild, 100-1000x ideas. Their goal is to attempt 1000s in the hopes of finding one massive win per decade.
So basically, they are willing to have 100 to 1000 bad bets for everyone that becomes a lotto ticket for the company. This means they have to be RUTHLESS about knowing when to abandon ideas that are either 1) straight up impossible or 2) possible but never going to be practical. To do this, they set up kill criteria up front. We must achieve X by Y or else we kill it.
This sounds intense, but it’s essential for focus. If they kept putting time, energy, and attention into all 1,000 bets, they would never discovery and succeed at the moonshots they are after.
My suggestion is to adopt a similar standard when you spin up a PoC. Have a high bar of what you hope to achieve by when, or a fast exit. Life’s too short, and the next idea will come to you soon enough. You don’t have to keep nurturing something that should have been killed at the idea stage. It’s fine, you invested some additional time, but now it’s gotta go. Kill it.
My Hit Rate #
I’m very liberal with how fast I’ll create PoCs. Sometimes I’ll one-shot an idea within 5 minutes, brain-dumping a stream-of-consciousness prompt and seeing what happens.
Most of the time, it’s ok. Every so often, I get goosebumps. And sometimes, what came out was outright embarrassing.
My recent fail whale? I tried to use ScreenPipe to create a word cloud of the apps I used in the last 24 hours. Sounds amazingly useful! Claude worked diligently for 2 hours, going through massive research, reviews, best practices, etc. It probably spent over $100-$200 in tokens. I opened the app with “Christmas Morning” anticipation and… Pinterest Fail
It was absolute fucking garbage. Like, I made better mockups in Photoshop in 2001. And it didn’t work at all.
Now if I kept all the “ok” results, I would probably give myself an 80% success rate. But a strong majority of those are PoCs that either 1) never made it to production and 2) I never opened again.
If I set the bar at a “wow” AND something I use at least weekly from then on, I’m probably at 5%. This is good to know. It means I can confidently reject what I’ve built 19 out of 20 times, knowing that each is getting me closer to the 5% that actually matter. It’s ok to kill stupid ideas faster. They may have sounded like a good idea, but now you know. Didn’t work out. Move on.
The key is building this muscle NOW so that you can prevent yourself from being a PoC hoarder… holding onto everything you’ve ever tried.
List of Anti-Patterns. #
I’ll end with a quick list of anti-patterns. If you’re experiencing these, you probably need to more aggressively 1) filter the PoCs you attempt or 2) kill the PoCs that didn’t work out.
Make-work.
Token Maxxing.
Ease > Value
Machine gun style
Feigning productivity
Building without goals
Forcing decisions
Are you struggling? #
This is an evolving topic and may change as tooling and standards emerge.
I believe PoCs are more beneficial than harmful (usually), but I’ve seen the downstream effects become harmful with some folks.
Would love to hear where you’re winning or losing on this front!