How will robocars and pedestrians interact?

This is a complex question, one that many of us have been thinking about since the very dawn of the field. It's one thing to get the car to deal well with other vehicles -- even motorcycles and bicycles -- but pedestrians are both the most complex and most vulnerable user of the urban street. And they are not just on the street, they are on the sidewalks and in parked cars and can enter the street at any time.

Summary:

• Robocars will be cautious around, and stop, for any pedestrian on the road (in a crosswalk or not,) but...
• Pedestrians will probably abuse that, changing the car-pedestrian dynamic in a way we don't desire, so we may seek other fixes.
• Pedestrians can and should make changes to their behaviour, but this is an expected factor in any new technology.
• One surprising issue will be understanding that robocars are always looking in all directions and always see you, so the old dynamics of "eye contact" may not make sense.
• It's easy to slip too far into both a pedestrian dominated world and a car dominated world, so we must work to find a balance in the middle.
• This is for high-traffic roads. Low traffic, more "walking" streets will have a different dynamic.

Robocars can sense pedestrians well enough, and thanks to neural networks they can even get good at understanding body language and facial expressions to improve what they learn from LIDAR. But there's more to it than that.

Robocars will work hard to identify and avoid hitting all pedestrians if the laws of physics allow it. At the same time, the easiest strategy for that -- be ultra cautious around pedestrians -- is not a very good answer.

A classic problem involves a pedestrian standing on a corner. The first thought will be that the pedestrian might be about to cross. The car should slow down and prepare for the potential of a pedestrian in the crosswalk. But not all people standing on corners are about to cross. Sometimes you spot a phone to their ear; they are on a call. Sometimes they are waiting for somebody. Sometimes a group of people are talking before they split up. A car that brakes for every pedestrian who might enter the road can't be a productive road citizen.

The corner is the most obvious place of ambiguity, but pedestrians enter the road away from the corner all the time. They cross (legally or illegally) at non-crosswalks. They play street hockey. They cross against the light and yell, "I'm walkin' here!" And much more.

My least favourite dance involves crossing mid-street. I see a car coming, followed by a long gap. I want to cross quickly behind the car as soon as it passes. I advance so I will get there just as the car passes. The driver, of course, freaks out: "Pedestrian in the road!" He brakes. He waves at me to cross in front. I don't want to cross in front, of course. He wants to be in control and doesn't want to move when I might cross in front. Eventually the pedestrian usually gives up and crosses in front. In the worst case, the delay of this little dance means that other cars come which would have not been an issue if the car had just kept going. Both pedestrian and driver exit frustrated. There are many stories like this.

Pedestrians vs. cars?

A recent interview with Andrew Ng (former head of Baidu AI and now an AI investor) caused quite a bit of stir. He suggested that pedestrians should do some adapting to self-driving cars in order to make the streets safer.

Media reaction was swift. Was Ng suggesting that cars reign supreme on the streets, and pedestrians should bow to their robot overlords? Or was he saying that there will naturally be some adapting, as there has always been for all forms of transport. Everybody remembers the first lesson they are given about traffic: "Look both ways before crossing the street!"

Robocars do have an absolute duty not to put pedestrians at more risk than existing human drivers do, and in fact to do better than they do. At the same time, new patterns and adaptations will arise that make the most sense for both pedestrians and robocars. The city and streets belong to both the pedestrians, other road users, and the people using the robocars, and we need the balance that makes the streets most pleasant, most safe but also most useful.

Eye contact

Today, drivers and pedestrians (and other human road users) make use of eye contact and body language. Pedestrians are particularly keen to know, "does that driver see me?" In fact, we often presume he doesn't until we get confirmation, in the form of mutual eye contact. A car, however, does not have eyes or a head.

The underlying reality does not match our old instincts. The car always sees you. Any car that is not aware at some level of every pedestrian in reasonably close range shouldn't be on the road unsupervised. Cars may not pay much attention to pedestrians on sidewalks away from crosswalks, but they still detect them. A robocar is looking in all directions at once, all the time. That's very much unlike a person. So there should be no question of "does it see me?" It does. And even though nothing is perfect, the car's ability to detect a pedestrian within a decently wide circle should be so close to perfect that it should be treated that way. What is more difficult to do and more difficult to communicate is "does it understand what I plan to do?"

Because we don't have this as an instinct, there have been various experiments about how cars might signal things to pedestrians and other drivers. Of course, that's not a new idea -- early on cars developed things like turn signals and brake lights to show intent. Drive.ai has a big LED panel that tells pedestrians what's coming. Several concept cars have included lights or signage, or even (at night) a laser projecting a walkway path onto the road for the pedestrian.

I've made my own proposals in that area, at Google and Starship, but so far no conclusions have been drawn. Cars could detect when a pedestrian is looking at them, and flash back an acknowledgement. Some cars have LED strips to show their image of the world. Some robots (not usually cars) have screens that show faces and eyes to give the car an artificial "direction of gaze" that can tell people "I see you." (Again, this is just comfort for the pedestrian, the car is actually always looking in all directions at all times.)

Such measures can provide comfort to pedestrians, of course. We're definitely not used to thinking of a car as all-seeing. We're naturally afraid of getting in front of a big heavy car that might run us over. But we actually will get used to it pretty quickly, I predict. We will think the idea of an "I see you sign" is as silly as having one on the elevator doors which we let close on us all the time.

Naturally, you have to know a car is a robocar in order to know it's watching you. A robocar might have a human in what would be the "driver's seat" but that person might not be making eye contact with anybody. This may justify some sort of way to make robocars distinct. For now, they are adorned with obvious sensor arrays, though nothing says if the vehicle is being manually driven (until you can see that the person in the correct seat has hands on the wheel.) Vendors have resisted any attempts to standardize this sort of thing during the prototyping phase, as it would serve little purpose, but it might eventually be a good idea.

Recognizing eye contact from the pedestrian

There are a few ways to detect if a pedestrian has seen the car. For example, thanks to the same effect which causes people to have bright red eyes in flash pictures when shot with a compact camera, you can tell quickly with an infrared flash and camera if people are looking right at your camera, though it takes somewhat high resolution to be able to see the pupil at a distance. Neural networks are now capable of detecting when a head, if not the eyes is turned to you.

And interesting question is, do you need to know this? Just as we can assume a car has seen us, a car might drive presuming pedestrians have not seen it. It's less important for the car to understand what you know as what you are going to do, which is communicated with body language, but most of all with motion. Gaze from a pedestrian mostly means "I am not about to step out in front of you."

So while a car can find value in tracking the gaze of pedestrians, it can always stay safe if it doesn't know their gaze and assumes they have not seen the car.

Do the roads belong to pedestrians or cars?

Cities were born before cars, and cars generally make them worse -- once you get to where you are going. There is who group who advocate that cars should be forbidden, or barely tolerated, in cities, with public transit being the way to get places you can't walk to. This is not a mainstream opinion, and the vast majority of people use cars for their travel in all but a handful of western cities. Roads have come to be designed for cars first, for this reason. In the 1920s the concept of "jaywalking" -- crossing outside official crosswalks -- was created, with some encouragement from the automobile industry.

Jaywalking is not usually illegal, not even in the USA. Some cities have local rules against it, and many states state that cars have the right-of-way over pedestrians when there is not a crosswalk (implied or painted) -- but not having right-of-way and breaking the law are two different things. It just means if the car hits you, you are the one who failed to yield. Any legal consequence is the least of your problems, then.

Robocars can and probably will be programmed to make most arbitrary pedestrian crossing a safe thing. With human drivers, even if there are no rules giving cars right-of-way, pedestrians are naturally distrusting and cautious. They try not to randomly walk out in front of cars, for good reason. In the robocar world, they could do so safely, as long as they don't make their appearance so sudden that physics bars the car from stopping in time. The more slowly cars traveled, the shorter those distances would be. Even without a change to jaywalking laws, robocars will be programmed to stop for pedestrians in the road for whatever reason.

This is probably not what the modern world wants. Though it might be safe for the pedestrians, it would involve very jarring trips for car passengers, and very slow and chaotic traffic in some areas. The majority of pedestrians will be in a car (or road based transit vehicle) later and don't want roads that work like this.

Pedestrians abusing safety of robocars

It is worth noting that while in many places there are no legal prohibitions on jaywalking, pedestrians usually do it with caution, their lives being at stake. It is likely that in a world of mostly robocars, the fact that it can be done safety (just at major inconvenience and some risk to passengers) might make it happen much more often.

In other words, before robocars, jaywalking has been a self-regulated thing, while this might vanish as robocars come to dominate the road, or if robocars are clearly marked.

While vacant vehicles (such as delivery vehicles and vehicles on their way to pick up somebody) won't care much about sudden stops for pedestrians, the cars behind them will, and occupied vehicles will care a lot. Especially if they become social spaces without seat belts.

Robocars will be capturing video of the world around them, and probably retaining it for a certain amount of time -- quite possibly as required by law in the case of an incident. Occupants of a car could request the car save video of somebody who forces their car to a stop. It's a political issue what they might be able to do with the video, but one possible choice is sending it to an enforcement division. That division might run facial recognition software on these submitted videos and identify people who frequently force cars to a halt, and allow people to bring complaints against them. If so, people would probably stop doing it. This approach, by relying on citizen complaint, is not as Orwellian as other methods of enforcement might be.

How much caution

For the robocar, the question will be "how cautious does the vehicle have to be in the presence of pedestrians?" For example, since pedestrians have the right of way in crosswalks, even the legal right to run into them from out of nowhere,

Travel in crosswalks

We've already seen one case where a Cruise test car, attempting to make a turn through a crosswalk, was (incorrectly) ticketed for not allowing a sufficient gap between the car and pedestrians using the crosswalk. As drivers know, you often make turns through crosswalks when pedestrians are also crossing. They either cross right behind pedestrians who have passed where the car will turn, or they do it quickly, well before pedestrians get there. Robocars can actually be very good at estimating pedestrian paths, and could even in theory time their turns to "just miss" pedestrians, which would work but which would be quite disconcerting to them.

While the police officer wanted to ticket Cruise for getting too close to a pedestrian, the truth is human drivers routinely get closer. This will become a challenge.

Sudden pedestrians

Much of the concern in a robocar is over the "surprise" pedestrian. This is one who enters the road, in particular the car's lane, or is on a path to doing so -- and who does so with insufficient warning. You might hit such a pedestrian, or be forced to swerve.

Nobody wants to swerve. That vastly increases the complexity of the situation, because then you become a surprise for everybody else. You definitely don't want to have to swerve into oncoming traffic and absolutely never onto sidewalks. No matter how much effort you put into it, your system will simply never have a lot of experience in driving on sidewalks or into oncoming traffic, except in simulator. It's the last choice you ever would want to make. And, of course, in many cases there will be other people or vehicles in those places, and swerving is not an option. Nobody is going to perform ridiculous "trolley problem" analysis to decide to hit one person vs. another.

This means you want to be able to stop in time for a surprise pedestrian. You set your speed and sensor capabilities so that you can stop for any surprise pedestrian in places they are expected to be -- namely crosswalks. The problem is that you can't do that outside of crosswalks. That's why the rules of the road give cars the right of way outside of crosswalks, or even make jaywalking illegal. If somebody darts out from behind a parked car into your lane, the laws of physics make it impossible to brake in time, and that's why the law puts the fault on the pedestrian in this case. Even so, you want to minimize it.

To minimize it, the most direct approach is to slow when you think it might happen. Most obviously for pedestrians standing on the sidewalk at the entrances to crosswalks. But what do you do if you can't see the entrance to a crosswalk, because of parked vans? This is the sort of "accommodation" Andrew talks about -- pedestrians being smart enough not to dart into the street from places where they know that cars can't see them. And likewise not jumping into the street outside the crosswalks.

Most pedestrians, when they cross outside a crosswalk, make an "accommodation" to human driven cars, and do it with lots of caution. Mostly they try to wait for a big gap in the traffic. If gaps are small, they step onto the street, but not into the driving lanes and show clear intent to cross. And as I noted above, often want to play the failed dance where they want to time the car and walk behind it after it passes, rather than make the car stop for the person.

It would actually be better for everybody if that dance were worked out. That the pedestrians trust the cars to continue on in their right-of-way, so the pedestrians can clearly and safely walk behind them. That the cars even map a safe path for the pedestrian, using their own knowledge of the other cars on the road.

We don't want -- not even the pedestrians want -- the alternative, which is every robocar slams on the brakes the moment a pedestrian puts a foot on the road. Or every robocar slows down to 15MPH on a 30MPH street when they see a pedestrian on the sidewalk who might be about to cross.

Today, robots won't be as good as humans at reading the body language, gaze and gestures of those pedestrians. In time, they might get much better, even better than humans, but the reality is even humans aren't that great at this, not from a distance, and not at high speed. That's why humans have developed a different, less risk-averse dynamic, where we expect pedestrians to fear the cars -- fear them highly outside of crosswalks and even have a decent fear inside the crosswalks. That's not the ideal for a pedestrian centered city, but it's the reality.

Slow roads

There are different considerations for very slow roads, particularly small roads where pedestrians and cars interact in very different ways. Such roads are very common outside the USA and in dense areas with very narrow streets. There, the lightning reaction times of robots actually might present new options for the dynamic of people and robots. We've all notice how people, walking through large, medium density crowds, can travel in every which way, very rarely bumping, and almost never very hard. While people can change direction instantly and cars can't, there are interesting potentials here too. But first, the medium speed urban streets where cars and pedestrians mix must be dealt with. This is a suitable topic for a future article.