Are today's challenges of making robocars dealbreakers?


There's been a lot of press recently about an article in Slate by Lee Gomes which paints a pessimistic picture of the future of robocars, and particularly Google's project. The Slate article is a follow-on to a similar article in MIT Tech Review

Gomes and others seem to feel that they and the public were led to believe that current projects were almost finished and ready to be delivered any day, and they are disappointed to learn that these vehicles are still research projects and prototypes. In a classic expression of the Gartner Hype Cycle there are now predictions that the technology is very far away.

Both predictions are probably wrong. Fully functional robocars that can drive almost everywhere are not coming this decade, but nor are they many decades away. But more to the point, less-functional robocars are probably coming this decade -- much sooner than these articles expect, and these vehicles are much more useful and commercially viable than people may expect.

There are many challenges facing developers, and those challenges will keep them busy refining products for a long time to come. Most of those challenges either already have a path to solution, or constrain a future vehicle only in modest ways that still allow it to be viable. Some of the problems are in the "unsolved" class. It is harder to predict when those solutions will come, of course, but at the same time one should remember that many of the systems in today's research vehicles were in this class just a few years ago. Tackling hard problems is just what these teams are good at doing. This doesn't guarantee success, but neither does it require you bet against it.

And very few of the problems seem to be in the "unsolvable without human-smart AI" class, at least none that bar highly useful operation.

Gomes' articles have been the major trigger of press, so I will go over those issues in detail here first. Later, I will produce an article that has even more challenges than listed, and what people hope to do about them. Still, the critiques are written almost as though they expected Google and others, rather than make announcements like "Look at the new milestone we are pleased to have accomplished" to instead say, "Let's tell you all the things we haven't done yet."

Gomes begins by comparing the car to the Apple Newton, but forgets that 9 years after the Newton fizzled we had the success of the Palm Pilot, and 10 years after that Apple came back with the world-changing iPhone. Today, the pace of change is much faster than in the 80s.

Here are the primary concerns raised:

Maps are too important, and too costly

Google's car, and others, rely on a clever technique that revolutionized the DARPA challenges. Each road is driven manually a few times, and the scans are then processed to build a super-detailed "ultramap" of all the static features of the road. This is a big win because big server computers get to process the scans in as much time as they need, and see everything from different angles. Then humans can review and correct the maps and they can be tested. That's hard to beat, and you will always drive better if you have such a map than if you don't.

Any car that could drive without a map would effectively be a car that's able to make an adequate map automatically. As things get closer to that, making maps will become cheaper and cheaper.

Naturally, if the road differs from the map, due to construction or other changes, the vehicle has to notice this. That turns out to be fairly easy. Harder is assuring it can drive safely in this situation. That's still a much easier problem than being able to drive safely everywhere without a map, and in the worst case, the problem of the changed road can be "solved" by just the ability to come to a safe stop. You don't want to do that super often, but it remains the fail-safe out. If there is a human in the car, they can guide the vehicle in this. Even if the vehicle can't figure out where to go to be safe, the human can. Even a remote human able to look at transmitted pictures can help the car with that -- not live steering, but strategic guidance.

This problem only happens to the first car to encounter the surprise construction. If that car is still able to navigate (perhaps with human help,) the map can be quickly rebuilt, and if the car had to stop, all unmanned cars can learn to avoid the zone. They are unmanned, and thus probably not in a hurry.

The cost of maps

In the interests of safety, a lot of work is put into today's maps. It's a cost that somebody like Google or Mercedes can afford if they need to, (after all, Google's already scanned every road in many countries multiple times) but it would be high for smaller players.

Even today's costs are a tiny fraction of the cost of building and even maintaining the roads. It doesn't take many users sharing the cost to make it quite affordable. This cost is going to go down fast, on a Moore's law curve, as the software gets better and better and less human oversight is needed. As I'll discuss below, only mapping and driving the most popular of roads remains a viable option. Even so, it would be folly to think the cost of the current first-generation mapping approaches won't drop a great deal as the need to map roads expands.

Unmapped traffic signals

One particular problem raised is the concept of an unmapped traffic signal, such as the temporary traffic signals that construction crews use. In the long term, I do believe that all road crews and city planners will come to never modify roads without recording it in a database first -- the trend of cheap networking and sensors that marketers call the "internet of things" will make this automatic in time. One can even imagine car makers paying to put a transponder (ie. a used cell phone) on every temporary traffic signal out there. (Google happens to know a bit about phones.)

Until that happens, we are not without a solution. While computer vision is not as good as humans at recognizing the millions of things in our environment, it's actually pretty good at recognizing specific things, especially static things, if you take the effort to train it. And as such, efforts are underway to make systems that will reliably identify troublesome items, like new road signs, new traffic signals, and even police officers and construction flaggers. By reliably, I mean "as well as humans are able to recognize them." The systems will be doing this not just with cameras, but the fused result of 3-D LIDAR and cameras, which gives them a much better ability in this area.

Gomes is also worried about the sun blinding a camera looking for a traffic signal. While that happens to humans too, the systems take a very simple approach: "If you don't confirm green, you don't go." If blinded, the systems will slowly stop. Even at a green light, which is annoying if it happens with any frequency, but is generally safe. (If there's a person inside or available to help remotely, they might tell it to go.) I joke that if you stop for more than a few seconds at a green, an audible alert will sound from any cars behind you, but that's probably not enough to go without confirming green!


Gomes suggests that the lack of parking effort presents an issue. I would guess the reason Google hasn't done a lot on parking is because it's actually pretty easy. There are already commercial products out there that have addressed many of the problems of parking.

Weather can be challenging

Nobody has worked really hard on driving in snow yet. There are lots of markets where snow is not an issue, of course, or where telling people they must take the wheel during snow is not a deal-breaker. Still, in the academic community there is promising research on techniques to handle snow, such as ground penetrating radar. In the event those techniques don't pan out, one sure approach is to do some relatively inexpensive additions to the road in places with snow, such as low cost magnets punched into the road, or special poles at the side of the road -- which are already used in many areas to let the snowplows figure out where the road is when it's snow covered.

There are issues with very heavy rain and blizzards, but new generations of LIDAR are showing superior results there, and frankly rain that is so heavy it blinds the sensors of a car is probably not rain you should be driving in, though you could choose to take the wheel of a regular car in such situations.


More work is needed on identifying debris and other road hazards. Again, at first, the vehicles will just be conservative. They might foolishly slow or stop for a crumpled newspaper on the road. Annoying, but tolerable if it's not too frequent, and in time the systems will rise to this challenge too. While the list of "rare" events is large, what matters is that they are rare. As long as there is a conservative solution and the events are truly rare, a product can be useful. Each problem will be solved, one by one, based on how important it is.

Promising the moon

Gomes' largest mistake is he feels that Google and other vendors have promised the world a complete solution any day now. While that will come, it is certainly not something people should expect on day one. No technology works that way.

The first vehicles that operate without supervision will only drive limited areas. That means if they do go between your origin and destination, they work for you on that trip. If they don't, you use all the existing methods -- manually driven cars, taxis, Uber-like services or transit to name a few. The more roads they can cover, the more trips they can serve.

Some vehicles will still have steering wheels, so you can have them drive where they can and you drive where and when they can't. This is what all the car companies are making. Google has said their experiments suggest that switching from human operation to automatic operation while underway is risky, so their latest prototype has no steering wheel. Vehicles without wheels won't be driven by you but they can take you to the border of their service areas where you can switch to a seamless rental car -- think Zipcar or Hertz -- or your own car or any other transport mode.

At the same time, while Google is concerned about the transition from automatic to manual driving, it's quite reasonable to build a vehicle that has a pop-out handlebar or wheel which only allows the transition from manual to automatic when the vehicle has stopped or is otherwise in an assuredly safe state. So it might take you to the edge of its service areas, stop and pop out the handlebars and let you take it from there until you enter the service area again. Not ideal, but still very, very good.

It is perhaps harder to understand as well that the goal with these vehicles is not perfection. Not even perfect safety. They will probably never be as safe as an elevator. They will have accidents. The real goal is to make vehicles that drive more safely than humans do, and thus reduce the number of accidents. It is thus an error to suggest you can't have people use the vehicles until every safety situation can be resolved completely. Rather, you just must assure that those situations are rare enough, and what the system does is good enough, that it outperforms people. Like many other computer systems, it is far from necessary for the systems to have full human situational awareness to outperform us.

The courts might take a different tack, and heavily punish anything less than perfection. If they do, and the legislatures don't fix that and make them look at the overall safety record, then the technology will just flourish outside the jurisdictions that punish it.

The challenge of predictions

The great challenge of a forecaster is to look at a nascent technology and judge where it's going:

  • It may be impossible to refine it to usability, and thus will not be a success until many major breakthroughs come.
  • It may seem clunky and primitive, but this is deceptive and it will soon take the world by storm.
  • It may truly be clunky, and need decades to really reach a level of major use.

All 3 of these things happen when we predict the future of new technologies based on the prototypes or first products. The Apple Newton might be viewed as having done all of these depending on how you look at the time scales. The Newton was flawed and failed, but inside was the germ of something world-changing, though that didn't happen for 2 decades.

One common cause of error is the difference between linear extrapolation, which most people are used to, and exponential extrapolation, to which only people in the computer industry are truly accustomed. On an exponential path you double every interval. On the linear path of 10 intervals, halfway along the path you are halfway there. On an exponential path of 10 doublings, halfway along the path you are only 3% of the way there and can easily conclude failure is certain -- but you would be very wrong.

That doesn't mean every grand exponential prediction is right, of course, just that it's very hard to make good predictions. Everybody knows that overly optimistic prediction is common, but the existence of massive underprediction is less well understood.

That doesn't mean there isn't a lot left to do, and plenty of uncertainty over when it will get done. But there is a reason for the optimism. More to the point, because the changes to be wrought by robocars are so grand, even a 10% chance means that everybody should start getting ready.

As always, I should note that while I have consulted for Google's self-driving car team, I am not speaking on their behalf.


I bought a new Ford Fusion with lane keeping, adaptive cruise etc. last year. I bought it from a dealer in a larger city, so I had to drive it home after I picked it up. As it happened, I drove (after dark) through a large snow storm most of the way home, complete with blowing snow, reduced visibility and drifts covering large parts of the highway (luckily it was a divided highway, so even if one lane was blocked, the other lane was typically clear.)

Anyway - it was a brand new car to me, and I hadn't done any of the configuration of the advanced safety features, so I was pretty surprised when the steering wheel started vibrating as I drifted toward and over the centre line to avoid some of the drifts. In the heaviest parts of the storm, the car could see the lines on the road far better than I could. Obviously that's not the same as driving on completely snow covered roads, but as someone who lives in one of the colder climates where we get our share of snow, I have no doubt that self-driving vehicles handling snow is a solvable problem.

The best part of it is that the vehicles won't forget how to drive in snowy conditions over the summer/fall like significant numbers of humans seem to (I haven't done the research, but anecdotaly there is always a large spike in the number of accidents after the first significant snowfall of the year around here)

About two weeks ago I was driving a local major road through Coquitlam that had much higher congestion than normal. Google Maps not only showed the congestion (red indication on the roads where traffic was backed up) but also had a small icon at the source of the congestion. The icon was a little one indicating construction. Turned out to be a crew blocking off one lane so they could do some tree trimming.

No idea how Google Maps got that info. But it was correct (well almost tree trimming not road construction, but the effect was the same) and in the correct location.

Actually, most construction projects are planned and go into the databases of the road authorities and they are willing to feed that out. Sadly, you can't expect perfection from that until a procedure is made that it's considered part of the safety checklist to not alter the road without logging that you are going to do it. Though you will probably get 99% logging just as part of normal business. Sometimes construction is done on an urgent basis -- but even this could be accurately logged. All road construction lives and dies on keeping safety procedures followed.

Google also owns Waze where people report construction and accidents and other things socially.

At least locally (Lower Mainland BC) municipalities do Highway Use Permits for anything that requires other than normal use of a road or highway.

I put on a couple of dozen bicycle road races a year and it would be nice if Google Maps was informed about our events automagically through the municipalities. Generally they are permitted a minimum of four weeks ahead of time. And have specific locations with start and finish times.

Humans don't need to see the road to drive in snow, they just drive in other cars tracks. The robocar can do the same, but even better, it can accurately triangulate its position from well-known static points in its surroundings. In snow-prone areas with a low density of clearly visible registration markers, poles could be erected.

One application of robocar technology might be a fleet of automated snowplows. (Same goes for fleets of pothole-fixing robots).

Lastly, I am curious how robocars will handle skidding on ice. Presumably, much better than humans. They will better judge stopping distance, keep to a proper speed, report areas of black-ice to other cars, (to be precision-bombed with salt.), and if a skid was to happen, it should be much better able to resolve it, by being able to independently brake and apply power to each wheel. It can also notify other cars to get out of the way if possible.

Yes, once there are tracks you can probably make a car follow them (along with other cars.) But they do get pretty confusing actually, and often cross one another etc.

The problem is the freshly covered road with no tracks.

They actually did built automated snowplows that follow magnets in the road.

Most cars today are very hard to skid due to ABS and ESC. The robocar will do a bit better but humans do pretty well now.

Thanks for the great post Brad. Clear and sensible as always. I was interested to see you mention infrastructure changes such as magnets in the road. For some reason this often seems to be associated with autonomous car pessimism (if we have to change the roads in *any* way, the technology is definitely not happening). The example of the snow poles I've seen at high elevations is a great example of where cheap infrastructure changes can solve a variety of problems of interest to autonomous cars. The entire interstate freeway system is a massive weird expensive infrastructure change. I just don't like seeing options and technology that involve letting the roadways help out be withdrawn reflexively from consideration. I agree with Steve that computer triangulation off of known landmarks will be vastly superior for snow driving than human "good" sense. He makes many good points supporting snow driving optimism. Another one is that with autonomous cars the chances will be much higher that they will be fleet managed. How hard would it be to fit studs, chains, or even snowplows on those cars? Or some of them to coordinate routes that are definitely clear. That will certainly be more plausible than ideal snow equipment on privately owned passenger cars we own today. Snowplows may be the first killer ap for self-driving vehicles. Also, the mapping issue seems a bit silly. As autonomous cars go about their business, they'll all be confirming and augmenting existing maps all the time. Talk about data that will grow exponentially. Eventually the slightest change, for example a makeshift construction barrier, should be noticed and shared with all other cars pretty much immediately.

To me, changing the infrastructure is a last resort. You do it if there is no better way.

Changing the infrastructure to drive at all is a bad idea, because you can't drive that road at all until they change it. Changing it to support unusual conditions is more tolerable because usually those conditions -- like a snow covered road -- are rare, so you are giving up a lot less.

If you need new infrastructure, then:

  • You can't drive there until somebody else, not you, installs the changes.
  • They install them on their schedule, with their bureaucracy.
  • If they are a government, they do it for policy reasons, not to help customers like a business does.
  • You have to work with thousands of different jurisdictions and follow their rules and schedules.
  • You will get political opposition, rather than technological challenges. Ie. people who don't like your cars will block them by blocking your infrastructure.

Some of these go away if the city just has to approve it, and you pay for it and install it once you meet the city guidelines. But many of these factors still get in the way.

Virtual infrastructure, like maps, is under your control. You decide to create it on your schedule and budget. Companies can compete or work together, and they do it only to serve customers.

Off-road infrastructure can make sense if there's nobody to block it. Things like high-accuracy GPS augmenters, or recognizable shapes installed on private land to the side of the road.

The shapes are still an issue because snow changes the shape of anything with any flat surfaces. Poles won't accumulate snow but other things will.

Magnets are really cheap, and if you get approval from the city, a truck can just drive the road punching a magnet into the pavement every so often. The city does not have to pay for it, but they do have to let it happen.

The question is, how much more difficult is mapping for robocars, from Google Maps mapping?
In other words, could Google Street View cars someday map at the scale required, possibly not in real-time but by recording and uploading their records to big computers that can crunch the data?

Is it possible that one day robocars could be driven manually once over an unmapped road, then the data processed, so that driving on that road can henceforth be automated? That would allow a car to hand over driving to a human, who would be that road's first driver/automapper. Since the data would not be available right away, several such firsts would be required, assuring that the road is driven from sufficient angles and enough conditions.

It is a much more involved map. However, while all you see in Streetview is the photos, streetview also does a LIDAR scan, though not at the resolution of the robocar. You might be able to get the detailed maps from that, especially as those sensors get cheaper.

The expensive part is the human evaluation of what the software did. The better the software and sensor readings get, the less you need of that. And the trend is to constantly improve that.

Though it's also good to have humans -- this time trained ones -- drive the road again to confirm that the map is correct. You're going to bet your safety on it.

Add new comment