California Disengagement Reports tell us a little more
Submitted by brad on Wed, 2022-02-09 17:36
Topic:
Tags:
There isn't usually too much learned from these reports, but we can compare year to year, so here are some things learned this time:
Share on:
Comments
Richard Young
Fri, 2022-02-11 13:05
Permalink
Disengagements meaningless safety metric
The only meaningful safety metric is crashes. I analyzed all CA AV crashes in my new book. "Critical Analysis of Prototype Autonomous Vehicle Crash Rates: Six Scientific Studies from 2015–2018."
Order from: https://www.sae.org/publications/books/content/r-515/
brad
Fri, 2022-02-11 19:32
Permalink
It is a shame
That SAE puts stuff like that behind a paywall. Usually authors are allowed to send them out free. You can mail btm@templetons.com if you hope for commentary on a paper.
Crashes are the strongest safety metric. However, I think there are some others you can look at, and also track their relations to crashes. Those include violations of the law (like unsafe lane changes even if they don't result in crashes) and other similar things (aggressive merges, forcing others to brake, needing to brake hard for both good and bad actions by other cars.) You can also track getting too close to VRUs even if you don't hit them. Just drifting out of your lane without intending to is worth noting, in fact doing anything you didn't intend to is of concern to most teams. Finally you want to track traffic disruption, where you block traffic.
Ben Thurston
Sun, 2022-04-10 12:36
Permalink
Tesla
I agree that Tesla would be among the worst of all makes shown on your chart, if they were included in this report. But there's a good reason for this—unlike the other major players, including Cruise and Waymo, they are developing FSD to work on all streets and highways, and not just a geofenced city area that's precision mapped, and using Lidar. This is a far more difficult task, and their results show this; in most videos, there's a disengagement or intervention every few miles at best.
In the end, it may turn out that Tesla is unable to achieve an acceptable level of safety without geofencing, precision mapping and Lidar. But it's a worthy goal; and certainly a system that's slightly less safe but operates everywhere is much safer overall than a system that a bit more or even significantly more safe, but operates only in a limited area. And they're dedicating huge resources to this, and getting tens of millions of miles worth of data every month to use for training the software; so I wouldn't bet against them.
brad
Mon, 2022-04-11 19:20
Permalink
Darn right it's more difficult
That's why most people think it's crazy. Why limit your ability and quality so much, just to avoid the cost of mapping. You can see my articles on why that cost is not nearly so much as most people imagine, and why keeping the map fresh is not that big a deal either.
You call it "geofencing" (and many others do) but that's a bad name for it.
Why is it a worthy goal to make your vehicle less capable and less safe, so it can drive poorly everywhere, rather than driving acceptably well in useful markets? Sure, if you could make it drive at that level everywhere that would be nice, but that's a goal for a long way down the road, not for getting the first one out in production and being a real product.
Elon Musk calls LIDAR a crutch because he doesn't think you can make a car work at all without sufficiently good computer vision. That's a possible argument. But if maps are also a crutch, they are an odd one to refuse to use when you have a broken leg. Unlike LIDAR which could possibly be obsoleted by CV, as Elon argues, maps are always a benefit, no matter how good you are without them. They let you see over that hill and around that corner. They add human understanding. Humans use maps all the time and can't drive other than slowly without them. (They just don't think of them as maps, they think of them as markings on the road which tell you what you can't immediately see about the road.)
Add new comment