Execution is Cheap, Context is Expensive
A few years ago, I was sitting with an Engineering Manager at Allegro, and we got into a discussion about the future. I said I'd love to get to a point where I could just tell a computer, "Now build me a billing system," and the billing system would be ready.
We're kind of there, but… not quite.
The tech world bubble is flooded with hype about AI "doing" everything, solving every problem, and making us 10x more effective. This won't be another article presenting AI as a silver bullet for every problem. I'll focus on real situations, the changes happening in product management, and both the positive and negative sides of the AI revolution.
Building "some" solution, a feature, etc. is becoming easy. And in fact, it has always been easy. But building things isn't the core of a PM's job.
Artifacts vs. Knowledge and Decisions
Thanks to AI tools, everything PMs "produce" — PRDs, user stories, summaries, presentations, roadmaps, notes — has become easier and faster to create. I personally cut my preparation time for quarterly reviews from 2 days to 2 hours.
But that doesn't mean the critical part of our job — making decisions — has become cheap too.
I'd argue that making the right decisions, based on actual knowledge, is now even more important. So we can focus on ideas that actually deliver value, rather than those that just look good. AI can help with this, but it can also lead us into traps.
What really matters is deciding WHAT to build. And this is where AI/LLM tools can help, but can also get in the way in some rather unexpected ways. Here are a few examples of where AI falls short and a PM's input is essential.
Transcript vs. Reality — The Lying User
Who has used AI to process research results, surveys, interviews, the so-called user research? For example, who has uploaded a table of survey results into an LLM and asked it to draw conclusions? And how many of you later read those conclusions or summaries and felt they were either too good to be true or too shallow? How many actually went back and verified the details? I had exactly this kind of situation.
A usability study in a lab. A researcher with a user, while the rest of the team sits in a separate room, watching the screen and listening to the interview. The researcher is interviewing the user about a new tool. At some point, she asks:
"Do you have any doubts about what you're seeing here?" — referring to the information, data, and how it's presented on the screen.
The answer: "No, everything is clear. I understand everything."
And then the user performs an action that completely contradicts what he just said. He does something entirely inconsistent with the information displayed on the screen. Everyone notices. But in the interview transcript, all we have is "I understand." The conclusion drawn by the LLM processing the research results: the UI is fine, users understand it. Without the context of observing the user's actual behavior, without human input, we would have missed that information.
AI processes text, not truth. A transcript tells us what the user said. Not necessarily what they actually did or understood.
The "Faster Horse" Trap — The Missing Signal
Another situation: analyzing user feedback collected by the sales team and through customer support tickets.
The feedback repeatedly mentioned problems with bulk data migrations, migration scripts, and the need to support customers through the process. AI suggested improvements to the existing process: better documentation, script templates, and ways to make those scripts easier to use.
But the real problem was somewhere else.
Those tickets revealed a repetitive workflow that shouldn't require support in the first place. Customers needed a way to perform migrations themselves, not more instructions on how to use scripts. The right solution was to build a new tool with a UI that would eliminate the need to perform those operations manually.
AI was suggesting ways to improve the existing process. What we actually needed was to question the process itself.
Data Silos
You've probably heard these promises: an LLM will analyze all our data sources, draw conclusions, and we won't have to worry about the mess anymore.
Unfortunately, it doesn't work like that.
When every system describes the customer differently, AI does not create truth. It creates a polished average of organizational confusion.
Continuing the story from the previous section: after realizing there was actually an opportunity to build a new service, I started looking for similar signals in other sources. The idea: I've got an LLM, so let's throw all the data sources into it and have it find confirmation from customers.
I did that and got a result. Nicely formatted, of course. But… a bit too short.
It turned out that two things defeated the LLM: the context window and messy data. Different formats, different identifiers for the same customers, different conceptual models. Which, by the way, reflected the existing reality: each of these tools was managed by a different team, and those teams didn't talk to each other much. Does that mean an LLM is completely useless for this kind of task?
Quite the opposite.
An LLM can actually be part of the solution to this mess, but not by following the "throw everything at the wall and see what sticks" approach. Preparing each record containing product insights in a common format and centralizing them is a great task for an LLM.
Tribal Knowledge
One last story, this time about actual data.
I have a hypothesis I want to validate: new customers increase their use of the data migration API after migrating. I don't have an analyst. But I have AI 🙂 I feed AI the problem description, context, etc. It understands. I provide the data structure — descriptions of Snowflake tables extracted from a well-maintained data catalog. I get a result: a nice SQL query (which would probably have taken me 2 days to write myself) and actual numbers.
I start working with the results, and something doesn't add up.
I look at the number of user accounts. For a single year, it's several times higher than the total number of customers. Something is wrong. I check the query, run some tests. Find a few minor issues, but that's not it. I start digging into individual data records. I find a column called "account type", filled with some numeric values. I investigate and discover there's no dictionary for this field in the documentation.
And this field contains values representing things like "trial" accounts and… test accounts created by developers for testing features, deployments, etc.
An analyst familiar with this data would probably have spotted this immediately and pointed out that we needed to filter by this column. AI didn't have that information in its context, so it didn't know those accounts needed to be excluded.
The query was great, but it didn't account for "tribal knowledge". And at first, everything looked perfectly fine.
Where AI Wins — The Actual ROI
Four examples, four problems.
Of course, that doesn't mean LLMs are useless. They're great tools that, like any other tool, need to be used thoughtfully and safely.
In the examples above, AI actually helped.
- It helped clean up and organize records from UserVoice. Data that had previously been chaotic and difficult to analyze could be standardized and categorized.
- Despite the tribal knowledge issues, it reduced the work from days to hours. Even though the final results required verification, preparing and iterating on SQL queries was significantly faster.
- It also helped develop a PRD. Preparing the first draft, defining scope, organizing information — AI can help with all of that. Judging what actually makes sense is still our responsibility.
- And finally, AI gives us another reason to fix the areas that aren't working well.
Data quality problems, missing shared definitions, undocumented knowledge — all of this existed before. It's just that now, when we try to use AI to analyze this data, those problems become even more visible.
The New (old) Product Manager
The PM role isn't disappearing. It won't be replaced by a bunch of agents reading feature requests, setting priorities, and creating Jira tickets. The most human part of a PM's job is still very much relevant:
Critical thinking, working with users and customers, understanding, and making decisions.
Understanding business context, understanding people, uncovering hidden information and needs, drawing conclusions. Actually working with users and customers. AI won't replace human-to-human conversations. It can summarize transcripts, but it won't catch every nuance, ask every question, or get curious about what's not immediately obvious.
The classic: "Distinguish what the customer needs from what they want."
AI won't replace critical thinking. What's more, the widespread adoption of AI tools will require even more rigorous analysis of results and conclusions. Going deeper into data, analytics, and connecting different sources. LLMs are here to stay, in one form or another. PMs will need to ensure better quality of the "input" we feed into LLMs, but above all, remain responsible for critical thinking and decision-making.
In fact, thanks to AI, we can focus more on what this role is actually about.
(On the other hand, we also have to fight off those vibe-coded solutions that the CEO threw together in Claude Code on a Saturday night and now wants to deploy.)