Few companies face higher stakes when deploying AI than Waymo, the self-driving car company under Alphabet that spun out of Google. Its models do not merely generate text or automate back-office tasks: They help vehicles navigate unpredictable streets, respond to human drivers and make split-second decisions in the physical world.
But the methods Waymo uses to manage those risks — continuous evaluation, carefully curated data, human oversight and clearly defined business outcomes — offer a broader playbook for enterprises deploying AI agents in nearly any industry.
Manasi Joshi, Waymo’s director of engineering for systems intelligence and machine learning, explained at VB Transform 2026 how the autonomous vehicle company trains, tests and deploys AI at scale. To date, Waymo has driven more than 220 million fully autonomous, or “rider-only,” miles, with 17 times fewer serious crash injuries than human drivers over the same distance, according to the company.
To achieve these impressive results, Joshi said Waymo has adopted what she called “eval-forced development” or “eval-centric development,” making evaluation a core part of engineering rather than a final check performed before deployment.
“The stage at which our projects are maturing can be easily kind of transpired based on the eval maturity that they showcase,” Joshi said.
In practice, Waymo assesses a project’s readiness partly by examining the maturity of the tests surrounding it. That approach has clear implications for enterprises building customer service agents, coding assistants, financial systems or other AI applications: If a company cannot reliably measure a system’s performance, it may not be ready to place that system into production.
Evals must continue after launch
Joshi said much of Waymo’s quality work has shifted toward evaluations, including tests conducted during model training, after training and inside open-loop and closed-loop simulations.
“Eval is not a one-time task to launch a model,” she said.
Waymo instead treats evaluation as a continuous process spanning driving, simulation and validation. Its methodology combines datasets, performance metrics and infrastructure capable of operating efficiently at scale.
For enterprises, that means testing an agent before launch is insufficient. Teams must continue evaluating it as underlying models, business processes, user behavior and incoming data change. Those evaluations should also connect to actual business outcomes rather than relying solely on broad industry benchmarks.
Joshi cautioned that model-quality measurements are only as trustworthy as the evaluation data behind them. Waymo therefore pairs its performance claims with information about the properties of the datasets used to test its systems.
Like what you are reading? Sign up for our newsletter.
0dbfe65a-fbfb-ab5f-0180-748d34f627af
If you only read one newsletter each week, this is it. AI Weekly collects the weeks most pressing news, research and analysis into a concise summary of insights from both VB and around the web.
Currated by VB Managing Editor, Karyne Levy. Delivered every Thursday.
Your email
Submit
I agree to the terms and conditions for digital communication in accordance with the privacy policy. You can opt-out any time.
New content is available
Testing the rare and dangerous cases
Waymo’s evaluation hierarchy remains grounded in one overriding objective: safety.
The company draws on first-party driving logs, some third-party data and realistic simulations that expose its systems to scenarios spanning billions of synthetic miles. Task owners choose specialized data and metrics for situations involving vulnerable road users, railroad crossings, construction zones and other complex environments.
The same principle applies outside autonomous driving. Enterprises need to test not only the routine requests their agents handle successfully, but also uncommon situations where errors could create financial, legal, security or reputational damage.
Joshi emphasized that Waymo does not leave release decisions entirely to automated systems. Its production-readiness reviews include extensive human oversight, while internal safety leaders approve software releases and service-area expansions.
“This is not AI-driven and completely automated and zero human oversight,” she said. “Human lives are at stake.”
Efficiency cannot come at the expense of reliability
Waymo faces another problem familiar to enterprise AI teams: Demand for compute, storage, memory and network capacity is growing faster than the resources available.
The company pursues efficiency across data extraction and storage, distributed model training, model distillation, simulation and evaluation. It also emphasizes “data efficiency,” selecting the most useful training examples instead of treating greater volume as inherently better.
Waymo began using transformers in 2017 and subsequently expanded into large language models, vision-language models and vision-language-action models. Joshi said the company now uses generative multimodal models as part of its foundation-model strategy.
Waymo divides its technology between onboard systems inside each vehicle and off-board infrastructure used for model development, data processing and simulation. That combination forces the company to optimize both real-time inference and the larger systems supporting it.
Agents need their own evals
Waymo also uses AI agents internally as productivity tools for engineers. Joshi said agents help analyze data distributions, assess data efficiency and triage problems found in vehicle telemetry, training runs and failed evaluation jobs.
The goal is to accelerate investigative work so engineers can devote more time to judgment and difficult technical problems. But Waymo also evaluates those agents to ensure they produce trustworthy, accurate results rather than sending employees down unproductive paths.
For enterprise leaders, Waymo’s larger lesson is that agentic AI requires more than choosing a powerful model. Organizations need a clearly defined objective, representative evaluation data, continuous testing, infrastructure that can operate efficiently and named human decision-makers who remain accountable for deployment.
“Earning trust is supremely important,” Joshi said.
Partner Content
The enterprise AI challenge nobody solves with code generation alone
VB Staff
8:00 am, PT, July 9, 2026
AdobeStock 1878104375 Preview
Presented by SAP
Generating code with AI is fast, but getting that code to run reliably inside a large enterprise, integrated with live systems, governed for compliance, and maintainable over years requires foundational work that most organizations underestimate.
While 81% of all organizations have a detailed strategy, only 12–16% reach AI‑driven execution, says SAP’s Michael Ameling, CPO of SAP Business Technology Platform, and the reasons rarely come down to the quality of the generated code.
“Across industries, enterprises that have invested heavily in AI tooling are hitting a wall when generated code meets the reality of their existing environments, because generating code and operationalizing it are not the same problem,” Ameling says.
There are specific requirements for deploying AI-generated logic at enterprise scale: what data and integration readiness actually look like, how governance works when AI agents move from producing recommendations to executing workflows, and how development teams are changing their role as AI takes over more of the coding work.
Why AI code generation fails in enterprise production environments
The productivity gains from AI code generation are real and well-documented, but the ease of prototyping has given many organizations a misleading sense of how far along they actually are.
“Generating code is one thing,” Ameling says. “Enterprise customers, including multinationals and large organizations, need to ensure there are no compromises in compliance or security. Code that runs reliably for ten or twenty years, as it does at many of SAP’s largest customers, also has to be maintained, patched, and understood by whoever inherits it. Life cycle management, in other words, does not generate itself.”
The issue is rarely the generation quality. Teams build something compelling, then discover they lack access to the data it depends on, or the integrations it assumes, or the permissions required to run it in a real environment. The problem is essentially that AI amplifies an organization’s existing data and process maturity, but it can’t substitute for it.
This dynamic intensifies as AI moves from producing code to executing actions. Latency, cost, and system load all increase when logic runs continuously against live data rather than rendering a one-time output. The performance requirements of an autonomous agent operating across a multinational’s transaction systems are categorically different from those of a developer copilot.
How to connect AI-generated logic to fragmented enterprise systems
The architecture challenge that most enterprise AI projects underestimate is integration. Real enterprise environments are not clean slates: they combine cloud systems, legacy on-premise infrastructure, fragmented data stores, and dozens of business applications that were never designed to talk to each other. Getting AI-generated logic to operate reliably across all of them requires a layer that unifies data access, process context, and governance, and it has to be in place before any agent starts executing. And organizations that see AI as a reason to defer infrastructure modernization are making a mistake.
“The question is not whether to modernize or not. Of course you need to modernize,” Ameling says. “But the value you get on top of this is much higher with AI. Federated data access and harmonized process layers are not alternatives to upgrading a fragmented landscape, they’re what make the upgrade worthwhile.”
At the platform level, this translates into a set of practical requirements: structured data integration, end-to-end process visibility, and the ability to discover and connect to APIs across both modern and legacy systems. SAP’s approach with the Business AI Platform draws on tools including its Joule Studio, Integration Suite, Business Data Cloud, and SAP AI Agent Hub enterprise architecture layer to provide that context. The goal is to give AI-generated logic accurate, current knowledge of what a business is doing and how, rather than just access to raw data.
AI agents handle large challenges by dividing them into smaller, autonomous tasks, with each agent responsible for a specific domain, and all coordinated toward a shared outcome. A financial close, for example, involves dozens of discrete sub-processes. Agents handling each task in parallel, within defined constraints, can compress cycle times dramatically, but only if the underlying systems they interact with are coherent and accessible.
The governance and oversight that AI agents require in production
When AI moves from assistant to operational actor, the governance questions loom large, because agents that trigger workflows, update records, and interact with live business systems need the same accountability framework that applies to human employees, i.e., identities, defined privileges, and auditable behavior.
There are two distinct models:
Principal propagation, where an agent acts on a user’s behalf, inheriting that user’s permissions and scope.
System-triggered agents, where the agent operates under its own identity and role-defined privileges, functioning more like an automated HR role than a personal assistant.
Both models require the same underlying infrastructure: an agent hub where operators can see which agents exist, what APIs they can access, and what they are authorized to do. Observability also needs to be operationalized correctly for AI, combined with both technical and business evals.
“In production, openness is very important,” Ameling says. “We use OpenTelemetry as a framework, so we can integrate with other solutions, for end-to-end observability of the tool, third-party agents and the like.”
On top of that, standard technical evals, which test whether an agent produces consistent outputs, are necessary but not enough. Business evals assess whether an agent is actually moving the performance indicators it was deployed to improve, but it has to work end-to-end.
Where the testing happens is equally important. The traditional software development cycle across dev, test, and production environments breaks down when a model produces different outputs depending on whether it is running against test data or live data. Getting to trustworthy AI in production means accepting that validation looks fundamentally different from what engineering teams have practiced for decades, with live environment testing, even A/B/C testing to ensure outcomes are reliable.
How AI-driven code generation is changing software engineering roles
The role of the developer is not disappearing in this environment, but its center of gravity is shifting. The productivity multiplier is significant when developers can run multiple coding agents in parallel across open terminals, each working on a separate problem and each taking several minutes to complete. But it introduces a new kind of cognitive demand, because humans have to stay in the loop. That means tracking context across concurrent workstreams, evaluating outputs that range across large codebases, and making architectural judgments that no agent can be trusted to make alone.
“The more specific and complete the prompt, the less intervention is required, and developers are learning that bringing more context upfront pays dividends in reduced back-and-forth,” Ameling says. “But the output still needs to be understood, not just accepted.”
The competitive edge will remain intellectual property, not tooling. The companies that pull ahead will be those that most effectively encode their domain knowledge into the systems they build.
“A manufacturer’s process expertise, a financial institution’s risk logic, a logistics firm’s routing intelligence, these are the assets that AI can accelerate, but only if the organizations that hold them do the work to make them accessible and usable,” Ameling says. “Protect that, and apply AI to accelerate your differentiation.”