Comma.ai, the brash startup attempting to make a self-driving system entirely from a neural network has announced it will cancel the "comma one" add-on box it has planned to sell to owners of certain Honda vehicles. The box stuck on the rear-view mirror and used the car's own bus commands to provide an autopilot similar to those offered by car makers, with lane-keeping and adaptive cruise control.

Of particular importance is the letter from NHTSA to comma.ai which I suggest you read. This letter creates several big issues:

1. There are many elements of this letter which would also apply to Tesla and other automakers which have built supervised autopilot functions.
2. Of particular interest is the paragraph which says: "it is insufficient to assert, as you do, that the product does not remove any of the driver's responsibilities" and "there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose." That must be very scary for Tesla.
3. I noted before that the new NHTSA regulations appear to forbid the use of "black box" neural network approaches to the car's path planning and decision making. I wondered if this made illegal the approach being done by Comma, NVIDIA and many other labs and players. This may suggest that.
4. We now have a taste of the new regulatory regime, and it seems that had it existed before, systems like Tesla's autopilot, Mercedes Traffic Jam Assist, and Cruise's original aftermarket autopilot would never have been able to get off the ground.
5. George Hotz of comma declares "Would much rather spend my life building amazing tech than dealing with regulators and lawyers. It isn't worth it. The comma one is cancelled. comma.ai will be exploring other products and markets. Hello from Shenzhen, China."

To be clear, comma is a tiny company taking a radical approach, so it is not a given that what NHTSA has applied to them would have been or will be unanswerable by the big guys. Because Tesla's autopilot is not a pure machine learning system, they can answer many of the questions in the NHTSA letter that comma can't. They can do much more extensive testing that a tiny startup can't. But even so a letter like this sends a huge chill through the industry.

It should also be noted that in Comma's photos the box replaced the rear-view mirror, and NHTSA had reason to ask about that.

George's declaration that he's in Shenzen gives us the first sign of the new regulatory regime pushing innovation away from the United States and California. I will presume the regulators will say, "We only want to scare away dangerous innovation" but the hard truth is that is a very difficult thing to judge. All innovation in this space is going to be a bit dangerous. It's all there trying to take the car -- the 2nd most dangerous legal consumer product -- and make it safer, but it starts from a place of danger. We are not going to get to safety without taking risks along the way.

I sometimes ask, "Why do we let 16 year olds drive?" They are clearly a major danger to themselves and others. Driver testing is grossly inadequate. They are not adults so they don't have the legal rights of adults. We let them drive because they are going to start out dangerous and then get better. It is the only practical way for them to get better, and we all went through it. Today's early companies are teenagers. They are going to take risks. But this is the fastest and only practical way to let them get better and save millions.

"...some drivers will use your product in a manner that exceeds its intended purpose"

This sentence, though in the cover letter and not the actual legal demand, looks at the question asked so much after the Tesla fatal crash. The question which caused Consumer Reports to ask Tesla to turn off the feature. The question which caused MobilEye, they say, to sever their relationship with Tesla.

The paradox of the autopilot is this: The better it gets, the more likely it is to make drivers over-depend on it. The more likely they will get complacent and look away from the road. And thus, the more likely you will see a horrible crash like the Tesla fatality. How do you deal with a system which adds more danger the better you make it? Customers don't want annoying countermeasures. This may be another reason that "Level 2," as I wrote yeterday is not really a meaningful thing.

NHTSA has put a line in the sand. It is no longer going to be enough to say that drivers are told to still pay attention.

Black box

Comma is not the only company trying to build a system with pure neural networks doing the actual steering decisions (known as "path planning".) NVIDIA's teams have been actively working on this, as have several others. They plan to make commentary to NHTSA about these element of the regulations, which should not be forbidding this approach until we know it to be dangerous.

It is challenging. It's hard to do QA on neural networks. You can examine any single state of them, but not really understand them. You can fix the errors they make, but not know how you fixed it or whether your fix is going to work in other cases. On their own that sounds too scary, but the problem is they are outperforming other algorithms at many of the problems they are being applied to.

If you have two systems:

1. A conventionally coded system which you fully understand, which has one safety incident per 100,000 miles
2. A black box machine learning system that has one safety incident per 150,000 miles, but you have minimal understanding of

Which is the one that is better to put on the road? It's not a no-brainer.

Scaring people way

George Hotz is about as brash as they come, so some might discount his bluster about moving his projects to China. But this is a real thing. When regulations come, they tend to favour the big players. The big players have large QA teams and compliance departments. They can afford them, and they move conservatively because of them. Big players help write the regulations to favour their way of doing things -- sometimes selfishly but sometimes just because it is the only thing they know.

The USA and California led the way in robocars in part because it was unregulated. In the USA, "everything is permitted unless it was explicitly forbidden" and nobody thought to write "no robots" in the laws. Progress in other countries where "everything is forbidden unless it is permitted" was much slower. The USA is moving in the wrong direction.

Countries like China are another animal altogether. There, "everything is forbidden unless a smart autocrat allows it" but once the smart (or stupid) ruling official makes it happen, the other red tape can vanish. It is not the best system overall, but it can win at things they put their minds to.

Tesla

Has Tesla received such a letter? What would they have done if they received it in their early days? While Tesla and Elon Musk love the autopilot project, the hard truth is Tesla's biggest problem is going to be making enough Model 3s to meet demand and doing it profitably. Taking out autopilot isn't going to cut their sales at all -- they have too many sales and not enough cars. Only their forward vision and guts will keep them going if NHTSA does the same thing to them. I do believe Tesla can answer questions like those in the demand letter more easily than comma.ai. But not trivially, and they almost surely will need to bump the countermeasures.