Home > On-Demand Archives > Keynote Presentations >
Truths and Myths about Automated Vehicle Safety
Philip Koopman - Watch Now - EOC 2024 - Duration: 01:21:55
I'm glad the presentation was thought-provoking and appreciate your thoughts.
I have done quite a lot of work on liability with a lawyer. We have a detailed proposal suitable for use by state legislators to amend liability law a bit by defining a "computer driver" (with manufacturer responsible) that has the same duty of care to other road users as a human driver would have in any particular situation. That will put more pressure on developers to deploy at a responsible pace without requiring detailed technical regulations. More here: https://safeautonomy.blogspot.com/2023/05/a-liability-approach-for-automated.html
Remote monitoring is tough because if you have a person remotely watching for issues you end up with automation complacency, and the situation doesn't scale (one person for 4 cars will provide worse supervision for example). Having a remote driver drop in to help can make sense to help a car get unstuck, but the get up to speed time will be too long for any dangerous situation while the care is moving. And yes, communication latency and drop-outs will be an issue. If we want robotaxis the vehicles will need to ensure safety while moving entirely on their own, even if they get remote help once in a while.
An entirely different way to go is make vehicles better at supporting human drivers who actually drive. That is already paying safety dividends, but it's not a robotaxi...
Thanks for the links.
In regard to remote person supervision. I didn't mean exactly that. I thought about one more layer of traffic control above the self-driving vehicle. The autonomous traffic supervisor, if you please. Maybe it even be considered as a part of the "smart city" concept.
Being located above the streets (on a tall building or a mast), it "sees" multiple vehicles at the same time, knows their speed and direction, definitely knows where the roadway and sidewalk are. So, in theory, it could assist the "robotaxis" in fulfilling their duties. For instance, by broadcasting messages like "Be careful, there are multiple small objects near the edge of the sidewalk behind that parked truck, probably children".
It will be quite expensive, on the one hand. But on the other hand, there are already a plethora of cameras deployed in each city, and such service may be shared by police or ambulance. Like "Ambulance #3 avoid the route X, there is a traffic jam there".
Of course, it is just my humble imagination. :-)
13:32:58 From Phil KOOPMAN to Everyone: Hi Everyone! 13:33:14 From Phil KOOPMAN to Everyone: I'm here live in chat if you have quick/clarification questions while the talk is in progress. 13:33:36 From Phil KOOPMAN to Everyone: I'll be live audio/video when the talk concludes. I'm looking forward to a lively Q&A 13:40:08 From Phil KOOPMAN to Everyone: Tesla recently changed from FSD(Beta) to FSD(Supervised). No idea whether it was because of my nomenclature though. 13:43:11 From Phil KOOPMAN to Everyone: It is typical to see thousands of incidents and very few or no fatalities for a software defect that results in a recall. 13:44:49 From Phil KOOPMAN to Everyone: People who want to declare a safety crisis usually just show this chart starting with the 2010 data and it goes up from there. Context is everything 13:46:43 From Elecia White to Everyone: Why did the pandemic make for worse drivers? Didn't the amount of driving go down? 13:47:51 From Phil KOOPMAN to Everyone: Miles went down, but the rate per 100M VMT went up. No definitive conclusion, but part of it was higher impact speeds due to less congestion. 13:48:21 From Phil KOOPMAN to Everyone: The UK increase during Covid was much smaller proportionally than for the US. 13:50:15 From Phil KOOPMAN to Everyone: This strategy dates back to the 1950s: "The Nut Behind The Wheel" narrative 13:52:48 From Phil KOOPMAN to Everyone: The Uber ATG settlement amount is unknown, but it is probably comparable to one day cash flow to run their operation. 13:53:07 From Phil KOOPMAN to Everyone: (or maybe two days; you get the idea) 13:54:15 From Phil KOOPMAN to Everyone: Since I tape this a Tesla on AP killed a motorcyclist and the driver is facing criminal charges. 13:59:14 From Phil KOOPMAN to Everyone: Software has bugs … who knew??? 13:59:25 From Eric to Everyone: Reacted to "Software has bugs … ..." with 😊 13:59:27 From Junnan Pan to Everyone: Reacted to "Software has bugs … ..." with 😊 14:00:25 From Jui Yen to Everyone: Reacted to "Software has bugs … ..." with 🤯 14:03:13 From Eric to Everyone: As Long as they stay in the car, there are safe … (faraday cage) 14:03:22 From Junnan Pan to Everyone: I think the biggest problem of AV is the false positive or false negative 14:03:38 From Phil KOOPMAN to Everyone: Eric -- yes (probably) but what if the passenger panics and exits? 14:03:55 From Eric to Everyone: Agree ... 14:04:11 From Phil KOOPMAN to Everyone: Junnan -- FP/FN is an issue, but a big root cause of that issue is a heavy tail distribution of unusual events that were mostly not seen in training/testing 14:04:59 From Phil KOOPMAN to Everyone: Phrase of the day: "inhuman error" 14:05:13 From david.fraustro to Everyone: AI hallucinates too, no alcohol required 14:05:25 From Brian Reiche to Everyone: Reacted to "AI hallucinates too,..." with 😂 14:05:38 From Eric to Everyone: Reacted to "AI hallucinates too,..." with 😂 14:06:01 From Phil KOOPMAN to Everyone: David -- I have a different mental model for hallucination we can discuss in Q&A if you like (not disagreeing, just a different framing) 14:06:12 From Elecia White to Everyone: Earlier in the talk you gave fatalities per miles (for humans), are there similar statistics for AP? 14:06:32 From Junnan Pan to Everyone: agree, that is the big challenge of software development, software cannot predict all Situation and reacts properly toward unpredicted Situation. 14:07:40 From Phil KOOPMAN to Everyone: Elecia -- I skipped that slide to make this talk fit in an hour. Short version: if you adjust tesla data for operational conditions AP is about the same on or off. Any safety gains are from automated emergency braking. IIHS recently stated based on insurance data automated steering has no safety benefit 14:08:14 From Phil KOOPMAN to Everyone: (IIHS: Insurance Institute for Highway Safety) 14:10:39 From Phil KOOPMAN to Everyone: Alcohol is a 28% problem, not a 94% problem. Distracted is less than a 10% problem, counter to what you might hear. 14:10:57 From Phil KOOPMAN to Everyone: Yes, they are problems, but no single source fixes the issue; it's complicated 14:12:01 From Phil KOOPMAN to Everyone: And the alcohol + speeding + distracted numbers overlap, so the effect is less than the sum against 100%. At a guess a non-compromised driver is 2x better than a compromised driver (half the fatality rate) 14:14:26 From Elecia White to Everyone: Do you think we can eventually get to self-driving vehicles? Or is just not possible? 14:15:23 From Phil KOOPMAN to Everyone: Elecia -- we are already there on a few roads in a few cities. The issue is more whether it can scale and operate economically. So far the answer is "not yet" except on low speed shuttles on semi-protected roadways. 14:15:53 From Elecia White to Everyone: Reacted to "Elecia -- we are alr..." with 👍 14:15:57 From Phil KOOPMAN to Everyone: The heavy truck folks say they will make this happen in the next 5 years for middle mile freight on interstate highways. We'll see. 14:18:25 From Phil KOOPMAN to Everyone: In 1995 a CMU self-driving car when across the US almost 3000 miles at 98% hands off the wheel. That last 2% is a bear. 14:18:39 From Phil KOOPMAN to Everyone: (went) 14:20:14 From Phil KOOPMAN to Everyone: Ride hail drivers, for example, routinely pass by sight impaired riders with guide dogs. 14:20:27 From Junnan Pan to Everyone: do we (or government) have already any progress to solve the ethics problem? 14:20:38 From Phil KOOPMAN to Everyone: Junnan -- basically no. 14:21:03 From Phil KOOPMAN to Everyone: California PUC has placed no equity requirements on robotaxis at all. So none of them have wheelchair capability for example. 14:21:27 From Phil KOOPMAN to Everyone: The industry has lobbied hard to make it difficult to sue companies for harm. 14:22:05 From Eric to Everyone: The trolley Problem assumes that there is an social scoring for every person available online in milliseconds … we are not in China. 14:23:33 From Junnan Pan to Everyone: as long as this problem not solved, do the companies still have the motivation to proceed the development of fully AV? As they all do not want to shoulder the responsibility for such dilemma 14:23:36 From Phil KOOPMAN to Everyone: You heard that correctly -- Waymo decided to stop the data window one day before a passenger injury, then claimed "no passenger injuries" and are STILL making that claim today by referring to that old study. 14:24:10 From Phil KOOPMAN to Everyone: Junnan -- their motivation is in part if they do not show progress they do not get the next round of funding. 14:24:38 From Elecia White to Everyone: What would the release package look like for MyNewAutoPilot in order to release new software? A canonical simulated million miles that all auto-drive systems need to run? (Setting aside vehicle difference) And then would it go to a public site for commentary? Or to a certification site (like FCC testing for devices)? 14:25:25 From Phil KOOPMAN to Everyone: The companies have started advertising 70 million miles, but that is all companies including test driving on all software versions 14:26:42 From Elecia White to Everyone: Replying to "What would the relea..." Is this covered in your book? 14:27:27 From Phil KOOPMAN to Everyone: Elecia -- good live question; a lot to unpack there; please feel free to ask in a couple minutes since recorded me is wrapping up 14:30:18 From david.fraustro to Everyone: I think not all miles are made the same, a 10mile test in a busy downtown is more complex than a 100 mile in a empty road 14:30:21 From Brian Schmalz to Everyone: That was an absolutely FANTASTIC talk! 14:30:39 From david.fraustro to Everyone: very nice talk! 14:31:06 From Junnan Pan to Everyone: very inspiring 14:31:07 From Brian Reiche to Everyone: Replying to "That was an absolute..." Hehe, the irony in the initials... 14:31:22 From Brian Reiche to Everyone: Replying to "That was an absolute..." Seems appropriate for this talk 14:33:53 From BobF to Everyone: No mention of any 'Black Box' fitted i.e. the 'red' type as found in aircraft ... any particular reason why these arn't discussed, for cars? 14:34:48 From Nick to Everyone: What do you mean by not being taught "safety"? 14:38:40 From Junnan Pan to Everyone: good point 14:40:38 From Elecia White to Everyone: When I was learning to drive, my stepdad (professional driver) said that I was too smart to drive well (his sense of humor was very dry). His advice was that I needed use my brain to be as predictable as possible to the other drivers; it wasn't about driving, it was about driving for everyone else. Can that be applied to robot driving? 14:40:44 From SuziO to Everyone: It wasn't an EE or software specific class. Honestly I can't even remember what the course was. 14:41:50 From david.fraustro to Everyone: In my experience iso 26262 is not enough because it all depends on the safety goals, do you know if there is an effort to have some standard safety goals for self-driving cars? 14:49:32 From Eric to Everyone: I think autonomous driving will only come about when the human brain can be emulated almost 100%. Too many unpredictable situations. 14:49:42 From Elecia White to Everyone: Really great talk, thank you @Phil KOOPMAN! 14:51:39 From Stephane to Everyone: Reacted to "Really great talk, t..." with 👍 14:51:43 From Junnan Pan to Everyone: Reacted to "Really great talk, t..." with 👍 14:51:47 From René Andrés Ayoroa to Everyone: Reacted to "Really great talk, t..." with 👍 14:51:54 From Stephane to Everyone: Thank you Philip! 14:51:59 From René Andrés Ayoroa to Everyone: Thank you, Phil! 14:52:02 From Lyden Smith to Everyone: Thank you Phil! 14:52:03 From Jui Yen to Everyone: Thank you! 14:52:06 From Eric to Everyone: Thank you Phil! 14:52:32 From SuziO to Everyone: Thank you for such a great talk! 14:53:30 From Carlos Hidalgo to Everyone: Excellent! Thanks! 14:53:34 From Vishwa to Everyone: Thank you for your great talk
Hello! Thank you for sharing your thoughts in this session. :-)
Never thought there are so many ethic problems/concerns pertaining to the self-driving vehicles. But if one thinks a little, that's true. Probably the biggest issue will be the absence of the person to blame. There is an accident, there is a fatality, but there is no person in charge of that. Just a soulless piece of metal and silicon. For many it will be hard to accept, because someone must be punished. And what about the law? It should be accommodated for this new reality too.
Quite sure a certain groups of people will emerge claiming ban of those "devil machines". And rise of "Choose a human" taxi services for those willing to have a small talk while riding to the destination. :-D
Anyway, how do you think, aren't the developers too "biased" squeezing in all possible features into the mere vehicle? Won't it be better to "lend a hand" to the autopilot from the outside? In the cities there are plenty of high points where the sets of equipment can be mounted. Those may be quite handy in preventing catastrophic events. Just because they see things normally hidden from the "eyes" of the car. Like a small kid suddenly running out of the parked car. And because they have a less volatile environment (tracking the same piece of street/crossroads for years). That, of course, requires a real-time communication channel with the vehicle.
What do you think? Or it is just one more failure point?