A year after the MIT study, three lessons from putting AI agents into production, illustrated by one project that runs through all three.
It has been just over a year since MIT's Project NANDA, The GenAI Divide, published the study that made the most noise about AI in companies. The headline went around the world: 95% of generative AI pilots in large companies delivered no return. Anyone working with this day to day did not need the study to know that few projects were generating real value. What the study did was put a number on the pain, and show that there was a small group, the 5%, doing things differently.
For those of us still in the room, who understand that generative AI in companies is here to stay, the question that mattered was a different one: what does that group have in common? This year, we put AI agents into production across entire sales, billing and customer service journeys, and three of MIT's findings were confirmed in practice with a precision that surprised me:
- Generative AI MVPs need to be even smaller than we expected.
- AI agents are like new hires, and they need promotion criteria.
- Generative AI excels in some places and struggles in others, and the architecture needs to respect that.
Below, I explore each point through a project we built with one of the leaders in mobility in Brazil.
1. Generative AI MVPs need to be even smaller than we expected
We have grown used to building MVPs in digital transformation. Start small, validate, expand. What surprised us is that generative AI demands even smaller starting points, for two reasons that seem contradictory. Because it is more flexible, it delivers value much faster than a traditional application. Because it is less predictable, the potential for failure multiplies just as fast.
Teams that generate real value with AI agents do not try to automate an entire process from day one. Projects like that drag on, trapped in endless edge-case management. The ones that deliver results start with something so focused it is almost embarrassing to justify. And that is where the real work begins: aligning the expectations of people who, dazzled by the technology's potential, expect it to solve problems of a fundamentally different nature all at once.
Car subscription is a good example. It is a product that has existed in Brazil for years but has not yet built a broad audience: the people who arrive do not know the offer. At the company we worked with, more than 70% of leads only moved forward after long conversations with a salesperson, and the entire journey had similar pains. We mapped all of it in a one-week immersion with the sales, marketing, customer success and service teams. And we started with a single stage: the sale. The prototype was ready in 3 days and was tested with real customers on WhatsApp, before any build contract. Less sexy than an agent that does everything, of course. But it is what put value in the customer's hands within weeks, and each following stage of the journey came in afterwards, with the same method.
The MIT study saw the same pattern: solutions focused on small but critical workflows reach US$ 1.2 million in annualized value within 6 to 12 months, and take about 90 days from pilot to implementation. Broad, complex processes take 9 months or more, often with nothing to show for it.
2. Treat AI agents like new hires, and set promotion criteria
Your best employees did not arrive on day one with full autonomy. They earned it. The same applies to AI agents. Most companies grant too much autonomy, too fast, and watch the failure. The opposite is just as risky: agents that are too restricted, waiting for perfection, may never deliver value.
A framework proposed by researchers at the University of Washington, Levels of Autonomy for AI Agents (Feng, McDonald and Zhang, 2025), has been decisive in our projects. It describes five roles a person takes when working with an agent: operator, collaborator, consultant, approver and observer. The central point is that the agent does not skip stages. It is promoted as it delivers results.
That is how it went with the car subscription agent. Human supervision existed from the prototype, and it was through it that the client's team corrected the agent while it was being built. Autonomy grew in stages, as the agent proved it was getting things right. In credit analysis, for example, the credit desk kept the decision, and started receiving the recommendation ready-made.
What tells you an agent is ready to take on more are clear business metrics from day one, not the model's subjective accuracy: the time the team saves, how much of what the agent produces needs correcting, how much of what the agent qualified converts.
3. Understand where generative AI excels (and where it struggles)
The first wave of applications created an illusion: that everything could be solved with prompt engineering and context retrieval (RAG). There is no silver bullet. Purely generative systems attacking complex workflows failed repeatedly, and every workflow that looks simple reveals surprising complexity when you try to automate it, especially at high volume, in front of the customer.
What the 5% seem to understand is that successful projects look more like traditional software than like magic. The winning architecture is hybrid: deterministic flows where reliability matters, generative AI where it truly excels, which is understanding and generating language, handling ambiguity and adapting to context. Business logic, predictable rules and exception handling are still better served by traditional engineering.
In the car subscription agent, price, term and payment conditions come from rules defined with the business, never from the model. The model comes in to qualify, answer questions and converse. And what the customer sees as one agent is, on the inside, a graph of smaller agents with defined roles: one understands the need, another checks price and stock, another explains the contract, another keeps the memory of the conversation, another gives the final answer in the same tone. An orchestrator distributes the conversation among them and acts as a safety lock. When needed, the conversation is handed to a person.
What this asks of whoever builds
Extreme focus, earned autonomy and hybrid architecture are not tool choices. They are approach choices, and it is the approach, not the technology, that separates the 5% from the other 95%. In practice, this has three consequences for whoever is going to build.
The first is that the solution is custom-made. An off-the-shelf platform, however impressive in the demo, does not know your credit rules, your billing policy or your business's edge cases. That is where it breaks.
The second is that design comes before technology. Understand the workflow first, and only then adapt the AI to it, not the other way around.
The third is that none of this gets done without the business teams in the room. They are the ones who know the workflow, the rules and the exceptions the agent will run into, and it is in the exchange with them that the design is born. In the car subscription project, that exchange started with a one-week immersion with four teams, and the sales prototype was ready in 3 days. With Kariisma, a live commerce startup from the United States, it was a 3-day immersion with the founders and the investors in the room, and the platform went live in 4 weeks. How each project went from there, with the numbers, is in the cases: the AI agent for the car subscription sales journey and the AI product that trains, directs and analyzes live streams on TikTok Shop.
Generative AI is still a learning journey, and the knowledge of those who are building is the most valuable asset. If you are one of those still in the room, we would love to continue this conversation: write to us.