I suspect the Google team will not be ok with an airbag deployment every 500k miles, if that's what constitutes an interesting disengagement event. Maybe they will surprise me.

Data from "unusual" situations has special value to human analysis, and to certain types of algorithms which are not yet used in production control systems. For basic NN training unusual stuff needs to be represented in the dataset in approximate proportion to it's representation in actual use - so you need thousands or millions of "usual" examples for each "unusual" one. With more sophisticated training techniques, which are under development but still immature, you can dilute that, but generally working systems will require representations of all the situations they encounter, not just the ones that a human analyst might find interesting. So you need a huge number of examples of things going right for every example of things going wrong because that's representative of the distribution of real situations that must be managed.

To date the straightforward way to accomplish getting statistically useful cross section of all the situations, both usual and unusual, has been to accumulate a large database of training data. There are ways to make smaller datasets go further but all other things being equal there's still no substitute for having a lot of data, with no upper bound to 'a lot'. 10 million is better than 1 million. A billion is better than 100 million, a trillion is better than 100 billion and so forth. Any player who has a multi-order of magnitude advantage is going to be tough to overcome.

Of course, if your software runs on heuristics or more conventional statistical techniques and you do lots of simulation you can make better use of smaller datasets, but your system is probably not going to exhibit the flexibility and robustness that NN techniques bring to the table. Google started their effort long before NN techniques were dominating perception and even longer before they were being used for planning and other higher level analytical functions. I don't doubt that you can make a system work that way, but it's not going to be as good as one using the best NN techniques. This means that to be competitive with later entrants that a lot of their pre-NN functionality would be displaced by NN techniques and that's going to present a dilemma to teams making that choice. You have a team that has developed a heuristic, who knows that system well, and their program has accumulated significant experience with it. If a radically different but potentially superior technique arrives it's going to be hard to get people up to speed and then make a fair evaluation of the alternate technique. You might have to change the team quite a bit. Teams are generally reluctant to start over with a new skillset that nobody yet understands well. And even when you commit to the new skills deciding where to draw the lines and how to proceed generally does not go smoothly. Generally those changes won't be made until well after the new technique has been demonstrated to be superior in some parallel effort or competing program, and then retrofitting it in to the main branch of the project is going to present integration issues. Preserving the stuff that still has value while abandoning the stuff that has been superseded is a hard balance and you are almost always going to be making the wrong decision in the face of a competing technology that continues to grow and outperform the systems that you have already made big investments in. This is a central reason why companies rarely - essentially never - continue to dominate their industry after a paradigm change.

Of course it can be done, and the Google guys have excellent management in this area. If anyone can do it it's probably them, but it's by no means a sure thing.

There's one other area where lots of data is almost indispensable and that's in making the statistical case that a particular system is safe. It's possible to extrapolate a small data set into conclusions about a larger dataset if you have a good model of a system but I doubt that anyone is going to suggest that they have a good model of everything that can happen to a car in the real world. Musk has suggested that a billion miles is the kind of data you need to have in order to make a solid statistical argument that a self driving system is better than a human in the real world. That gives you maybe 1000 avoided airbag deployments and the opportunity to avoid around 10 fatalities so in that sense it seems like a good number. Anything much less than that and you can't be confident that it is better than a person over the full range of common driving circumstances. Tesla should have 2 billion miles on their HW2 platform by the end of 2017 and another 3.7 billion on their HW1 platform so it's conceivable that they could have the miles to make a good case if they have a system that works. But I'm not expecting anyone else to have even 1% of that. So it raises the interesting question of what kind of hand waving might be involved in getting regulators to agree to deploy a system that is, in the strictest sense, unproven. And is any conservative entity, in particular Google, going to be willing to do that? Currently I'm expecting them to wait until 2020 but I'm excited to think they will come up with a clever way to get into the field before events overtake them.

I'd like to address the assertion that "there is nothing that says this would not be true of lidars".

Comparing optical data and sensing equipment to silicon in terms of it's ability to radically decline in cost over time is a mistaken association. The cost learning curves of technologies vary enormously and very few processes have the learning curve that silicon data processing demonstrates. There's overwhelming evidence that optics technologies do not demonstrate silicon-like learning curves. Note that I am not referring to lithographically produced linear response devices like camera sensors, but to optical lenses, waveguides, mounting systems, and especially nonlinear optical sources like lasers and other NLO beam manipulation devices.

Camera lenses, for example, have been around as consumer offerings since long before computers entered that realm, but they've never seen a million fold improvement in cost performance over a span of a few decades despite enormous investments in optical manufacturing, dramatic growth in manufacturing volumes, and extraordinarily accurate scientific models WRT their underlying physical properties. For a direct comparison you might want to take note of how how the price performance improvement curve in networking interconnects degraded after the transition from copper based interconnects over to fiber optic interconnects. In the earlier period performance was driven by improvements in silicon and enjoyed several orders of magnitude of improvement in a couple of decades, but as the channel bandwidth properties of copper became problematic a transition to optical media necessitated the adoption of electro-optical transducers and optical interconnection systems, at which point improvement in link cost performance slowed to a relative crawl. This despite continued high rates of investment. Even today, decades after we hit the wall on copper bandwidth for wired ethernet interconnections and despite enormous increases in manufacturing volume the optical successors have not managed to make it out of the enterprise datacenter due to their inability to scale costs. Nonlinear optical devices, as much as they have improved over time, continue to be hard to manufacture with maddeningly high variation and associated low yields except for the most undemanding applications. Efficient coupling of microscopic electro-optical devices to the real world via lenses and waveguides continues to present manufacturing tolerance issues that have yet to be addressed by any highly scalable manufacturing process. That could change any day, but a lot of resources have been poured into this problem for at least the last 20 years and there is as yet no good candidate. At this point there is no expectation that high bandwidth optical links will ever become a consumer technology.

And so, while there is good reason to believe the price performance of data processing systems associated with self-driving vehicle technologies can be expected to continue their dramatic improvement in the coming years, I find any similar expectation of lidar technologies to be misplaced. Of course they will improve, but improvements in processing are very likely to be dramatically greater. In my 16 years in the business of designing optical systems I watched numerous efforts, both at startups and at industrial giants, march in with the conviction that they were going to bring radical improvements to the field. I mean, how hard can it be right? We have this great demo, and this lovely plan, we just have to get it into the factory. Give us a year, maybe two, just to work out the details. Except it never worked out that way and almost all of those efforts are gone, mostly with nothing to show for their labors aside from gaping holes in checkbooks.

I believe your analysis suggesting that the cost sensitivity of high duty cycle fleet vehicles is likely correct. Fleet operators will not care about an extra few thousand dollars for a sensor if it provides significant benefits in terms of safety or utility. But is lidar ever going to provide the utility to offset the cost? It's possible, but it's far from certain. Lidar will keep getting better, but then so will radar, and multispectral imaging, and thermal imaging, and commercial versions of these systems are much less expensive than lidar is today. As the computational ability to process sensor feeds grows by leaps and bounds the utility window for difficult and expensive sensors will shrink. In the end there will be a multitude of very cheap sensors backed by enormous processing power. In those fully mature systems lidar will only be present if you can make it for a price comparable to a camera, which is unlikely to happen. I say that because the camera based systems will be so good that lidar's incremental safety advantage won't matter.

So will it be useful for a window of time prior to when vehicle vision systems become adequate? That it would is a conjecture that a system employing lidar would be able to achieve a given desirable level of safety at a lower operating cost than a comparably safe non-lidar system. There was a time prior to 2012 when the world had no expectation that camera based systems would be able to do the job in the foreseeable future, so using expensive and difficult sensors was the only option. But now the expectation has shifted, at least in the opinions of people who have not already made a large investment in those expensive and difficult options, to expecting cameras to be able to do the job in the foreseeable future. And unless that expectation turns out to be wrong, there's not going to be a big role for lidar. Tesla is moving forward on this updated expectation. Other developers are largely still in the belt-and-suspenders world of trying all the sensors in parallel to see what works, and that's appropriate for them. Because aside from Tesla, nobody is fabricating hardware for a commercial system in volume. All the others are just development platforms.

It sounds like you believe Tesla is wrong, and so you must expect that Tesla will fail. Is that correct?