ep. 87. What AI Can (and Can't) Automate
5 min read
Over the past week, Matt Shumer’s essay, Something Big Is Happening, has been making the rounds in the tech world. He’s a startup founder who, like many of us, noticed a step-change improvement in the latest frontier models. He describes them as exhibiting “judgment” and “taste.”
TLDR: We have 1-5 years before AI takes over much of the work that people currently do on screens. Therefore, get your financial house in order and start really learning how to use AI tools. If you ever wanted to build something, now’s the time. Follow your dreams and vibe code it. Don’t delay, because the window between Opus 4.6 and technological unemployment is brief.
Shumer’s essay was compelling, both in how it was written, but also because he’s naming a familiar pattern. I’ve personally experienced this capability increase since Opus 4.5. I’ve heard similar accounts from close friends and in my broader professional network. However, I don’t thing something that big is happening.
Yes, frontier models continue to improve at unprecedented pace. “Assume this is the worst AI you will ever use” (thanks, Ethan Mollick).
Yes, automation fundamentally changes the nature of work. If it replaces a “doing” task, then the person who used to do the thing will now be monitoring the thing. Because of that, AI will fundamentally reshape roles as it automates more “doing” tasks (see: IBM redefining junior roles around AI supervision and human-centric client engagement. They recently announced they’re tripling their number of entry-level jobs).
No, just because AI is getting better at certain types of cognitive work, it doesn’t mean it will get better at all cognitive work on the same curve.
Boundary Definitions
The conventional wisdom about automation has historically gone something like this: machines handle manual, repetitive tasks; humans handle cognitive ones.
Models are currently performing cognitive tasks, such as writing code, drafting contracts, building financial models, and “designing” UI.
At the same time, these examples share something else in common: The person delegating work to AI can specify what “done” should look like before the work begins. Any task that falls under this description is fair game for automation.
It makes sense: Tasks like financial modeling and diagnostic screening usually take years of pattern extraction practice to hone. LLMs are really good pattern extractors. Because of that, expect to only see models getting better here.
At the same time, there are also many tasks where “we know more than we can tell”, a phenomenon called Polanyi’s Paradox. Because they depend on tacit knowledge, they resist explicit formulation, and therefore, automation. These are the “non-routine” tasks that defy full automation.
Generative AI moves the “non-routine” boundary by inferring pattens about what that tacit knowledge might be, learning from the outputs of human expertise without needing access to the process that produced them. My read is that a lot of folks in Cerebral Valley think that boundary will disappear within the next 5 years.
I don’t buy it.
Limits on Codifying Tacit Knowledge
When codifiable cognitive work gets automated, you’re left with work that relies on knowledge that can’t be separated from the context in which it operates. For instance: Recognizing that the problem your team was asked to solve isn’t the actual problem, or reading a room to understand that the team’s resistance to a proposal isn’t about the proposal.
These are forms of knowledge, tied to particular people, relationships, histories, and organizational contexts. When we act in the world, we apply these different forms of knowledge in what anthropologist Lucy Suchman called situated actions: flexible, real-time adaptation to what’s happening.
AI systems, including today’s agents, operate on representations of context, like the tokens in a prompt or the data in a connected system. Humans operate within a situation, drawing on tacit knowledge that hasn’t been explicitly represented. This doesn’t matter if you assume codifiable knowledge roughly = tacit knowledge. My point is there’s actually less overlap between the two than you’d expect, given the current dialogue around AI automation.
Isn’t AI Already Acting on Tacit Knowledge?
The counterargument is that with enough data and long enough context windows, you could approximate all of this context. Shumer’s language suggests that models already have tacit knowledge, making “the right call” with something resembling judgment and taste. Whether it’s real understanding or extremely good mimicry, he argues, doesn’t matter. It’s just doing the thing.
This feels plausible within a specific scope. LLMs are trained on the artifacts produced by people exercising tacit knowledge (designs shipped, code written, and so on). In domains with clear feedback signals and well-documented conventions, the output can be functionally indistinguishable from judgment.
A model that selects the right spacing, visual hierarchy, and interaction patterns for a checkout flow is drawing on patterns underlying thousands of shipped e-commerce interfaces in its training data. The result resembles taste because the conventions are so well-established that following them well is what good taste means in that context.
At the same time, a model producing ‘the right call’ is converging on the most probable “good” answer given its training data. That means it reproduces consensus well. It gives you what competent practitioners would do in a well-understood situation. That’s useful when conventions are established. It’s much less useful in ambiguity, where the right answer might be the unlikely one that only makes sense given context the model doesn’t have. It’s like getting AI to generate feature ideas: They’re sound but generic, and especially unhelpful if you’re working in an undefined product space.
Why Precision Matters
If you treat all cognitive work as one undifferentiated category that AI is steadily eating, you’ll automate things you shouldn’t. You’ll hollow out the organizational knowledge that makes good judgment possible. You’ll optimize for speed on tasks where speed wasn’t the bottleneck, and lose capability on the things that actually differentiated your team.
Instead, for every task you’re considering automating, ask: Can the knowledge required to do this task well be specified? Do you have a clear sense of what “done” looks like before you begin? If yes, automate. If it’s more ambiguous and requires ample context, then you need a human in the loop.
Humans operating in the world are more complex than we’re giving them credit for. As a result, we’re likely to need humans in the loop more tasks (and for longer) than you might expect, no matter how good frontier models get. The nature of the work will shift, but the human is still there in some capacity.
None of this changes that building AI fluency still matters. Many roles are changing, and people will need to adapt quickly. Learn these tools and leveraging them where they work best. However, my assumption is that that rate of collapse will be less extreme than predicted because of the need for humans in (or on) the loop.
For anyone looking to jump to full automation: Be precise about where AI still falls short. The work that’s hardest to codify is also the work that’s hardest to rebuild once you’ve automated it away. That’s the tricky part: Tacit knowledge is by definition hidden, so you don’t notice it until it’s gone.
If you liked this episode, you’ll love my book. Designing Automated Futures is coming this year with Rosenfeld Media. 📣 Sign up to be the first to know about new book releases, sales, and events.
📣 Call for Proposals: EPIC People Conference
Are you working on questions about modeling context or deploying context? Consider submitting a proposal to the EPIC People Conference, held October 25-27, 2026, in Chicago.
⏰ Deadline: Extended to March 2. Learn more here.
I’ve participated in EPIC in various capacities over the years and consistently find it to be a rewarding experience. It’s a rare space where practitioners talk with each other about messy, real-world problems and how to constructively move forward. The conference is worth checking out even if you’re not submitting.
This year, I’m excited to be co-chairing EPIC’s Immersion Day with Todd Palmer. We’re planning an interactive day of learning and making before the main conference. More soon.
⏪ Recent Episodes
ep. 86: ServiceNow’s Real Moat: Context Orchestration for AI
ep. 85: Back to School Vibes
ep. 84: The Importance of Anticipation
📖 Good Reads
Why the next internet will be agent-first x Amy Webb and Andrew Hornstra.
An AI Taxonomy x Narain Jashanmal
AI Doesn’t Reduce Work—It Intensifies It x Aruna Ranganathan and Xingqi Maggie Ye (HBR)
That’s a wrap 🌯 . More on UX, HCI, and strategy from Sendfull in two weeks.
Thanks for your important work in helping us understand this technological moment! Loved the reference to Polanyi’s work, The Tacit Dimension was critical in my thesis trying to distinguish different forms of design labor (i.e. manual drawing vs. digital drawing)… I’m still haunted by it.