Rapid Scaling Turns into AV Achilles’ Heel

AV operations aren't scaling because their vehicles have gotten safer. In contrast, scaling is exposing AVs' weaknesses, as evidenced by each recall involving software updates that proved ineffective.

Where Waymo is serving today (white dots), and up next (blue dots) (Image: Waymo)

For autonomous vehicle companies, scaling, not safety, is the top priority because it makes—or breaks—their long-term business survival.

Virtually every AV developer, led by Waymo, has been pushing scale at breakneck speed, while justifying this strategy in the name of saving lives.

This accelerated AV rollout strategy could stall, however, if the National Highway Traffic Safety Administration (NHTSA) follows through on a letter sent to the still nascent industry’s developers and operators.

In a scathing message, NHTSA chief Jonathan Morrison warned AV companies that their vehicles’ interference with emergency response operations is “unacceptable.”

He wrote:

Let me be clear: the inability to detect and appropriately respond to such situations represents a functional insufficiency. Emergency scenes are not rare or extreme “edge cases.” As such, NHTSA is today issuing a call to action for AV developers and operators to immediately focus their resources on fixing this issue.

NHTSA summoned driverless automated driving system developers for meetings, scheduled by the end of this month, to hear their solutions.

While Morrison’s intent is clear, what remains unknown is the following:

1. Although aware of their vehicles’ chronic interference with emergency responders, AV companies have temporized and alibi’d. Are they now ready to commit substantial time and resources to fix this persistent problem?

2. What solutions do AV companies plan to present in the NHTSA meeting?

3. What’s NHTSA’s plan to oversee and manage AV companies’ software updates for each recall? And what happens if recalls fail?

On this issue, when recalls don’t work, should NHTSA step in and exercise its authority, developing minimum safety standards to test and certify AVs?

That route would mark a clear divergence from the current “self-certification” norm to which the auto industry is accustomed. Automakers have long self-certified their own compliance with Federal Motor Vehicle Safety Standards (FMVSS). While NHTSA audits and orders recalls for vehicles’ FMVSS violations, it does not pre-approve vehicles.

Politically, the chances of NHTSA developing minimum AV safety standards are very small. Particularly under the current administration, NHTSA, long known for a more laissez-faire policy, has focused on “deregulation” in support of “innovation,” observed Michael Brooks, executive director at the Center for Auto Safety.

Technically, this may be untenable. NHTSA has already struggled to keep pace with software complexities in modern vehicles. Now the added complexity of recall fixes involving artificial intelligence, Brooks added, is exacerbated by recent cuts to NHTSA staffing.

Brooks can’t foresee a break in the status quo, “until there are minimum safety standards in place that force the AV industry to address this issue.”

The likeliest triggering for such a change would combine AV companies’ rapid scaling with repeated recall failures.

Brooks speculated, “Ultimately and particularly, if AVs scale quickly in the immediate future, I think NHTSA will be forced to take action and require the AV industry to certify to minimum operational safety standards.”

Botched software updates

With each of its recalls, Waymo, for example, promised software updates. These patches sought to fix many problems such as vehicles unable to stop for school buses, robotaxis plunging into flood waters, or AVs crashing construction zones.

Each recall involved software updates that proved ineffective

Making this dilemma worse is the absence of independent parties to test and certify software updates.

“Today, we sometimes know that an update was released, but not whether it really fixed the problem,” observed Bryan Reimer, a research scientist at MIT. AV companies may possess substantial internal evidence that their systems are improving, “but outsiders rarely receive enough information to evaluate those claims,” he added.

“The developer should not be the only party grading whether its own corrective action worked,” summed up Reimer.

If so, who should be monitoring software updates for recalls?

Phil Koopman, safety expert and professor emeritus at Carnegie Mellon University, said, “Regulators are the only ones in the position to take action when software updates and recalls fail to deliver on safety promises.”

Koopman anticipates that “market pressure will push AV companies to scale up faster than they can fix messy behavior problems with their vehicles.”

When that happens, in his view, NHTSA will have to step in.

A conscientious regulator must “keep a close eye on scaled-up operations to provide safety guardrails, while making sure that safety does not degrade.”

This might be wishful prognostication. Not everybody anticipates NHTSA to suddenly go heavy-handed with AV regulation.

Bryant Walker Smith, associate professor at the Schools of Law and (by courtesy) Engineering at the University of South Carolina, observed that local governments are burdened with auto safety nowadays, often with very little authority over automated driving companies. “NHTSA is watching,” he said, “ but it’s a tiny agency with responsibilities far beyond just automated driving.”

Walker Smith would like to see AV companies uncover and explore ways to become more transparent with the public.

For example, he suggested that each company maintain separate webpages addressing each issue of concern (pedestrians, lateral blind-spot sensing, construction sites, floods, school buses, power outages, congestion, etc.). “Each webpage should be a timeline: Each regular update should explain what the company knows about the issue, what it doesn’t know, what it’s doing, and what has changed,” he noted. “ This will help people track the company’s progress, and it will therefore help the company earn trust.”

Is consumer trust getting lost?

Consumer trust in AV companies is already waning.

Lisa Boor, director of Customer Success Auto Benchmarking and Mobility Development at J.D. Power, told us that consumer trust will depend “on the existence of standardized, independently validated safety standards.”

J.D. Power’s 2024 Mobility Confidence Index (MCI) findings revealed that “consumers want trusted third-party verification, transparent reporting, and consistent federal frameworks that allow AV safety claims to be evaluated objectively. This independent oversight is fundamental to broad AV adoption.”

Boor summed up: “Consumers are not simply looking for assurance that the technology is advancing; they want confidence that safety claims are being evaluated objectively and consistently.”

Cooper Lohr, senior transportation and safety policy analyst for Consumer Reports, worries about the rapid scaling of robotaxi operations.

“Rapid scaling does not equate to safer vehicles. If AV companies want to bridge the existing trust gap, they must scale responsibly and prove their systems can handle myriad complex real-world challenges before expanding.”

But there’s a caveat. Lohr said, “We can’t just rely on corporate goodwill to ensure the safety of driverless cars.”

In his opinion, “We need strong federal performance standards to ensure a baseline level of safety.”

A federal focus on AVs’ interaction with emergency responses would align well with consumer expectations.

Consumers asked to identify reasons why they are less comfortable with the idea of riding in a fully automated self-driving vehicle, personal safety is the leading concern, cited by 60% of consumers, J.D. Powers’ Boor explained. “That is closely followed by how emergencies [or] unexpected situations are handled (58%) and performance in difficult conditions such as bad weather, construction [and] heavy traffic (51%). Adoption hinges on proving that AVs can protect passengers in unpredictable, high-risk situations.”

Technically fixable?

Encountering events—whether deemed “edge cases” or not— that are unanticipated by AV operators, self-driving cars have been consistently vulnerable. They tend to misbehave, because their ability to adapt to rapidly changing real-world situations, at real time, is limited. Unlike human drivers, AVs can’t think on their feet in a low-data or no-data situation for which they have never been adequately trained.

Among safety experts, the conventional solution to edge cases was believed to be geofencing. The idea was “if you’re seeing something you can’t do, geo-fence it out or exclude it from ODD (operational design domain).” In the latest episode of his podcast, “There Auto Be a Law,” Koopman acknowledged that he was wrong. Geofencing, as it turns out, is hardly the 100-percent solution.

For example, if companies aren’t fixing an underlying problem. e.g., AVs unable to gauge the depth of water on the road, the AVs will continue to dive in.

If the National Weather Service’s alert went out only after the flooding had already started, robotaxi companies’ geofencing plan wouldn’t have prevented AVs from driving roof-deep into the water.

AVs don’t learn on the fly

The difficulty of developing effective software updates after each recall may also have to do with how today’s AI-driven AVs are fundamentally designed.

Missy Cummings, professor at George Mason University, made clear that “AVs do not learn on the fly. Their models are updated in batches.”

One of her concerns is the rise of End-to-End (E2E) machine learning, known to demand massive datasets. It requires immense computational power and creates a “black box” that makes it difficult to debug the software.

“Given the expense of end-to-end learning, there is not as much learning as you think is happening,” said Cummings. “Novelty is the real problem,” and “neural network ‘learning’ is regression to the mean, so it is never going to be good at novel situations.”

Referring to the many emerging problems of AV misbehavior, she said, “They indicate end-to-end learning carries much more risk. To my knowledge, it has actually never truly been validated in testing.”

Given the barrage of problems it is experiencing as it scales up, Cummings sees scaling as a major weakness for Waymo.

Why scaling matters

It’s important to understand the impact of robotaxi’s rapid scaling on the current AV problems.

Koopman offered a simple explanation. “If you have a constant failure rate and you double the number of cars, you’re going to see twice as many bad things happen.”

More cars screwing up more often doesn’t necessarily mean that the technology is getting worse. “But it’s clearly a failure of the technology to improve fast enough to keep the events under control as they scale,” Koopman said.

Cummings agreed, “Scaling highlights self-driving car weaknesses.” In her opinion, “Companies are trying to plug gaps with remote operations, but it clearly is not working.”

Here’s how to think about scaling. “Problems that were infrequent enough to not worry about, with a few dozen vehicles, start being much more serious concerns with thousands of vehicles,” noted Koopman. “A once-a-year incident with 100 vehicles becomes twice a week for 10,000 vehicles.”

This is the sort of math that tends to alarm regulators and consumer advocates, eventually leaking to the general public.

But perhaps, not Waymo, whose leaders are “scaling so that they can get to an IPO,” Cummings speculated. “Once the IPO happens, senior leaders will cash out and there will be a big mess to clean up for investors.”

Bottom Line:

It has long been taboo to consider minimum safety standards for testing and certifying AVs, especially for an agency in league with a US auto industry accustomed to lax federal regulation.

The recent repeated failures of post-recall software updates, however, might finally force NHTSA to exercise its enforcement authority.

Related story: