Juggling Chainsaws
Recognizing AI Reliability Tradeoffs
Frictionless Decisions brings you counterintuitive, original, jargon-free ideas for connecting data to decisions. Every week.
I’m excited to welcome Yakov Shkolnikov back to Frictionless Decisions. You can follow Yakov’s Substack here and read his bio at the end of this article. - ZH
I used a leading AI tool with plugins specifically designed for reviewing contracts. When I ran a contract through the tool, it flagged a liability issue it had never mentioned before. When I asked why it ignored the liability in earlier reviews, the tool said there wasn’t much liability language in the documents in the first place, so it had nothing to comment on.
AI had plenty to say about the contracts, but nothing about their biggest problem: the complete absence of the most important contract language.
The dread of realizing that work which looks plausible on the surface needs to be torn apart is specific and awful, and AI can make it more likely, not less. It can look correct yet be entirely wrong.
The excitement of AI is everywhere, with convincing demos, individual productivity gains, and the pressure to move fast. Yet using it right now is a lot like juggling chainsaws. Impressive. Interesting to watch. But one lapse in concentration and you lose a hand. The alternative is to juggle normal objects. It’s less dramatic, but you go home with both arms.
The Reliability Problem
There’s a reason manufacturing companies use Six Sigma, and IT teams measure system uptime in nines: in most businesses, avoiding failure outweighs improving average performance. They’re built on reliability, not improving the average individual job output.
AI can improve your individual output by 10 to 30 percent, depending on the task and the source. That’s real. But performance isn’t uniform across a distribution. The “tail” of those use cases - the part of business that determines reliability - is the low-frequency, high-stakes end: the difficult client and the deliverable where getting it wrong is expensive. That’s usually where businesses make money, keep clients, and protect their reputation. AI can make things worse in the edge cases.
Your business doesn’t simply need an improvement in its average productivity. It needs reliability on the edges.
Products and workflows that lean heavily on AI face a new problem: how do you cover the tail? Depending on the task, the cost of doing that can wipe out your productivity gain entirely. Token costs add to this; they eat into margins in ways many organizations didn’t anticipate, and the total bill often exceeds what the individual productivity improvement looked like on paper.
This reality forces your team to work harder to stay vigilant; you rarely remove human attention when you introduce AI into a process. More often, you redirect it. Someone still has to catch errors, verify outputs, and handle exceptions. Sustained vigilance is cognitively expensive, and humans are not built for it over long stretches.
AI adoption often requires something most organizations don’t have right now: time to prepare, practice, and focus. You cannot juggle chainsaws on a deadline with three deliverables due Friday.
Where It Actually Helps
AI can work well for prototyping, for generating rough versions of things faster than you otherwise could. For individual exploration, the productivity gains can be real.
Running your business on it is a different question. With enough effort and focus, it can produce excellent results, but that effort adds to the cost. The cost of watching the edges rarely disappears; more often, it moves. And for most organizations right now, that shift could end up costing more than the original productivity number suggested.
Postscript: I took my new contract and hired an actual attorney to review it.
About Yakov Shkolnikov:
Yakov is an AI systems architect with 20+ years building production AI, from early CNN systems deployed for radar applications in 2010 to enterprise GenAI platforms today. He has advised both internal and external organizations on AI and data strategy, built teams from the ground up, and led work in edge AI, forecasting, and optimization at Fortune 500 scale. His work has driven enterprise-wide impact across products, operations, and strategy.
Yakov holds a Ph.D. from Princeton and is an IEEE Senior Member.
Follow Yakov’s Substack here.
To remind you of this week’s data concept, enjoy Song From the Edge of the World by Siouxie and the Banshees, from the Frictionless Data Spotify playlist.
For the full story about making data flow faster and better, check out Frictionless Data on Amazon.
 | A guest post by
|