E-mail is facing a decline. This is something I lament, and I plan to write more about that general problem, but today I want to point out something that is true, but usually not recognized. Namely that E-mail today is often secure in transit, and we can make better use of that and improve it.

The right way to secure any messaging service is end-to-end. That means that only the endpoints -- ie. your mail client -- have the keys and encrypt or decrypt the message. It's impossible, if the crypto works, for anybody along the path, including the operators of the mail servers as well as the pipes, to decode anything but the target address of your message.

We could have built an end-to-end secure E-mail system. I even proposed just how to do it over a decade ago and I still think we should do what I proposed and more. But we didn't.

Along the way, though, we have mostly secured the individual links an E-mail follows. Most mail servers use encrypted SMTP over TLS when exchanging mail. The major web-mail programs like Gmail use encrypted HTTPS web sessions for reading it. The IMAP and POP servers generally support encrypted connections with clients. My own server supports only IMAPS and never IMAP or POP, and there are others like that.

What this means is that if I send a message to you on Gmail, while my SMTP proxy and Google can read that message, nobody tapping the wire can. Governments and possibly attackers can get into those servers and read that E-mail, but it's not an easy thing to do. This is not perfect, but it's actually pretty useful, and could be more useful.

We're all shocked at the idea of a growing neo-Nazi movement, at the horrible attack in Virginia and the lack of condemnation by the President. It's making us forget that the neo-Nazi radical right are trolls with many parallels to online trolls. And the only thing to do is not to feed the trolls, and definitely don't attack the civil rights that they make use of.

A protest march has 3 main functions:

I will be heading to western Idaho this weekend to watch my sixth total Eclipse. That makes me a mid-grade eclipse chaser, so let me tell you some important things you need to know, which are not in some of the other eclipse guides out there. For good general sites look at places like NASA's Eclipse Guide which has nice maps or this map.

Totality is everything

The difference between a total solar eclipse and a partial one -- even a 99% partial one -- is literally and metaphorically night and day. It's like the difference between sex and holding hands. They are really two different things with a similar sounding name. And a lunar eclipse is again something vastly different. This does not mean a high-partial eclipse is not an interesting thing, but the total eclipse is by far the most spectacular natural phenomenon visible on this planet. Beyond the Grand Canyon, Yosemite, Norway, etc. So if you can get to totality, get there. Do not think you are seeing the eclipse if you don't get into the zone of totality.

People debate about how deep into totality you want to be

Many people seek to get close to the centerline of the eclipse. This provides the longest eclipse for your area. You will only lose a modest number of seconds if you are within 15 miles of the centerline, so you don't have to get exactly there, and in fact it may be too crowded there.

On the other hand there are those who deliberately get close to the edge, giving up 30-40% of their eclipse time in order to see more "edge effects." Near the edge, the edge effects are longer and a bit more spectacular. In particular the diamond ring will be a fair bit longer, and you may see more prominences and chromosphere for longer. If this is your first eclipse, I am not sure you want to get too close to the edge. But try any of the map web sites that will tell you your duration, and get somewhere that has within 30-40 seconds of the centerline time.

You look at the total eclipse with zero eye protection

You've been hearing endless talk about eclipse glasses and how well made they are. Eclipse glasses are only for the boring partial phase. They give you a way to track the progress of the moon while waiting for the main event. Once totality is over, everybody packs up and does not even bother to watch the 2nd half of the partial eclipse, that's how boring the partial part is.

But don't be one of those people who, told about the danger of eclipses, does not watch totality with your bare eyes. In fact, use binoculars in addition to your naked eyes, and perhaps a short look through a telescope -- but not during the diamond rings or any partial phase.

Update: There is a nice large sunspot group that should still be there on Eclipse day, making the partial phase more interesting to those with good eyesight.

In totality you are looking not at the sun, but its amazing atmosphere -- the "corona" -- full of streamers, and many times the size of the sun or moon. You may also see jets of fire coming off the sun, and at the start and end of totality you will see the hot red inner atmosphere of the sun, known as the chromosphere.

If you are crazy enough to be outside the total zone but close to it, you sadly still can't look with your bare eyes at any part of the eclipse.

There are some cool things in a 99% partial eclipse (which you see just before and after totality.)

An eclipse is most glorious in the sky but a lot of other things happen around it. As it gets very close to total you will see the nature of the sunlight change and become quite eerie. Shadows of trees will turn into collections of crescents. About 20-60 seconds before and after totality, if you have a white sheet on the ground, you will see ripples of light waving, like on the bottom of a giant swimming pool. And the shadow. You will see it approach. If you are up on a mountain or in a plane this will be more obvious. It is going at 1,000 to 2,000 miles per hour.

Almost all robocars use maps to drive. Not the basic maps you find in your phone navigation app, but more detailed maps that help them understand where they are on the road, and where they should go. These maps will include full details of all lane geometries, positions and meaning of all road signs and traffic signals, and also details like the texture of the road or the 3-D shape of objects around it. They may also include potholes, parking spaces and more.

The maps perform two functions. By holding a representation of the road texture or surrounding 3D objects, they let the car figure out exactly where it is on the map without much use of GPS. A car scans the world around it, and looks in the maps to find a location that matches that scan. GPS and other tools help it not have to search the whole world, making this quick and easy.

Google, for example, uses a 2D map of the texture of the road as seen by LIDAR. (The use of LIDAR means the image is the same night and day.) In this map you see the location of things like curbs and lane markers but also all the defects in those lane markers and the road surface itself. Every crack and repair is visible. Just as you, a human being, will know where you are by recognizing things around you, a robocar does the same thing.

Some providers measure things about the 3D world around them. By noting where poles, signs, trees, curbs, buildings and more are, you can also figure out where you are. Road texture is very accurate but fails if the road is covered with fresh snow. (3D objects also change shape in heavy snow.)

Once you find out where you are (the problem called "localization") you want a map to tell you where the lanes are so you can drive them. That's a more traditional computer map, though much more detailed than the typical navigation app map.

Some teams hope to get a car to drive without a map. That is possible for simpler tasks like following a road edge or a lane. There you just look for a generic idea of what lane markings or road edges should look like, find them and figure out what the lanes look like and how to stay in the one you want to drive in. This is a way to get a car up and running fast. It is what humans do, most of the time.

Driving without a map means making a map

Most teams try to do more than driving without a map because software good enough to do that is also software good enough to make a map. To drive without a map you must understand the geometry of the road and where you are on it. You must understand even more, like what to do at intersections or off-ramps.

Creating maps is effectively the act of saying, "I will remember what previous cars to drive on this road learned about it, and make use of that the next time a car drives it."

Put this way it seems crazy not to build and use maps, even with the challenges listed below. Perhaps some day the technology will be so good that it can't be helped by remembering, but that is not this day.

The big advantages of the map

There are many strong advantages of having the map:

• Human beings can review the maps built by software, and correct errors. You don't need software that understands everything. You can drive a tricky road that software can't figure out. (You want to keep this to a minimum to control costs and delays, but you don't want to give it up entirely.)
• Even if software does all the map building, you can do it using arbitrary amounts of data and computer power in cloud servers. To drive without a map you can must process the data in real time with low computing resources.
• You can take advantage of multiple scans of the road from different lanes and vantage points. You can spot things that moved.
• You can make use of data from other sources such as the cities and road authorities themselves.
• You can cooperate with other players -- even competitors -- to make everybody's understanding of the road better.

One intermediate goal might be to have cars that can drive with only a navigation map, but use more detailed maps in "problem" areas. This is pretty similar, except in database size, with automatic map generation with human input only on the problem areas. If your non-map driving is trustworthy, such that it knows not to try problem areas, you could follow the lower cost approach of "don't map it until somebody's car pulled over because it could not handle an area."

Levels of maps

There are two or three components of the maps people are building, in order to perform the functions above. At the most basic level is something not too far above the navigation maps found in phones. That's a vector map, except with lane level detail. Such maps know how many lanes there are, and usually what lanes connect to what lanes. For example, they will indicate that to turn right, you can use either of the right two lanes at some intersections.

Ten years ago, I asked Cell companies to let me have more than one phone on the same number. Recently I noticed the ability to almost do that with Google Project FI cell service.

Something I do from time to time is a road trip in a rental car. And while car rental companies much prefer the business customer who rents a big car at a high price, then just drives it to their meeting and back to the airport, they are not averse to the less profitable road trip business.

So here are some things they could do to make it better for that sort of customer.

I've written before about overbooking and how it's good for passengers as well as for the airlines. If we have a service (airline seats, rental cars, hotel rooms) where the seller knows it's extremely likely that with 100 available slots, 20 will not show up, we can have two results:

Earlier I noted that Nidi Kalra of Rand spoke at the AVS about Rand's research suggesting that purely road testing robocars is an almost impossible task, because it would take hundreds of millions to a billion miles of driving to prove that a robocar is 10% better than human drivers.

(If the car is 10x better than humans, it doesn't take that long, but that's not where the first cars will be.)

This study has often been cited as saying that it's next to impossible to test robocars. The authors don't say that -- their claim is that road testing will not be enough, and will take too long to really work -- but commenters and press have taken it further to the belief that we'll never be able to test.

The mistake is that while it could take a billion miles to prove a vehicle is 10% safer than human drivers, that is not the goal. Rather, the goal is to decide that it's unlikely it is much worse than that number. It may seem like "better than X" and "not worse than X" are the same thing, but they are not. The difference is where you give the benefit of the doubt.

Consider how we deal with new drivers. We give them a very basic test and hand them a licence. We presume, because they are human teens, that they will have a safety record similar to other human teens. Such a record is worse than the level for experienced drivers, and in fact one could argue it's not at all safe enough, but we know of no way to turn people into experienced drivers without going through the risky phase.

If a human driver starts showing evidence of poor skills or judgments -- lots of tickets, and in particular multiple accidents, we pull their licence. It actually takes a really bad record for that to happen. By my calculations the average human takes around 20 years to have an accident that gets reported to insurance, and 40-50 years to have one that gets reported to police. (Most people never have an injury accident, and a large fraction never have any reported or claimed accident.)

Today's news is preliminary, but a U.S. house committee panel passed some new federal regulations which suggest sweeping change in the US regulatory approach to robocars.

I have recently managed to dig up some old documents from the earliest days of car regulation. Here is a report from NHTSA on the state of affairs near the turn of the 20th century.

National Horse Trail Safety Administration (NHTSA)

Regulation of new Horse-Auto-mobile Vehicles (HAV), sometimes known as "Horseless carriages."

In recent years, we've seen much excitement about the idea of carriages and coaches with the addition of "motors" which can propel the carriage without relying entirely on the normal use of horses or other beasts of burden. These "Horseless carriages," sometimes also known as "auto mobile" are generating major excitement, and prototypes have been generated by men such as Karl Benz and Armand Peugeot, along with the Duryea brothers, Ransom Olds and others in the the USA. The potential for these carriages has resulted in many safety questions and many have asked if and how NHTSA will regulate safety of these carriages when they are common.

Previously, NHTSA released a set of 4, and later 5 levels to classify and lay out the future progression of this technology.

Levels of Motorized Carriages

Level 0

Level zero is just the existing rider on horseback.

Level 1

Level one is the traditional horse drawn carriage or coach, as has been used for many years.

Level 2

A level 2 carriage has a motor to assist the horses. The motor may do the work where the horses trot along side, but at any time the horses may need to take over on short notice.

Level 3

In a level 3 carriage, sometimes the horses will provide the power, but it is allowed to switch over entirely to the "motor," with the horses stepping onto a platform or otherwise being raised to avoid working them. If the carriage approaches an area it can't handle, or the motor has problems, the horses should be ready, with about 10-20 seconds notice, to step back on the ground and start pulling. In some systems the horse(s) can be in a hoist which can raise or lower them from the trail.

Level 4

A Level 4 carriage is one which can be pulled entirely by a motor in certain types of terrain or types of weather -- an operating domain -- but may need a horse at other times. There is no need for a sudden switch to the horses, which should be pulled in a trailer so they can be hitched up for travel outside the operating domain.

Level 5

The recently added fifth level is much further in the future, and involves a "horseless" carriage that can be auto mobile in all situations, with no need for any horse at all. (It should carry a horse for off-road use or to handle breakdowns, but this is voluntary.)

In San Francisco, I'm just back from the annual Automated Vehicle Symposium, co-hosted by the AUVSI (a commercial unmanned vehicle organization) and the Transportation Research Board, a government/academic research organization. It's an odd mix of business and research, but also the oldest self-driving car conference. I've been at every one, from the tiny one with perhaps 100-200 people to this one with 1,400 that fills a large ballroom.

Toyota Research VC Fund

Tuesday morning did not offer too many surprises. The first was an announcement by Toyota Research Institute of a $100M venture fund. Toyota committed $1B to this group a couple of years ago, but surprisingly Gil Pratt (who ran the DARPA Robotics Challenge for humanoid-like robots) has been somewhat a man of mixed views, with less optimistic forecasts.

Different about this VC fund will be the use of DARPA like "calls." The fund will declare, "Toyota would really like to see startups solving problem X" and then startups will apply, and a couple will be funded. It will be interesting to see how that pans out.

Nissan's control room is close to live

At CES, Nissan showed off their plan to have a remote control room to help robocars get out of sticky situations they can't understand like unusual construction zones or police directing traffic. Here, they showed it as further along and suggested it will go into operation soon.

This idea has been around for a while (Nissan based it on some NASA research) and at Starship, it has always been our plan for our delivery robots. Others are building such centers as well. The key question is how often robocars need to use the human assistance, and how you make sure that unmanned vehicles stay in regions where they can get a data connection through which to get help. As long as interventions are rare, the cost is quite reasonable for a larger fleet.

This answers the question that Rod Brooks (of Rethink Robotics and iRobot) recently asked, pondering how robocars will handle his street in Cambridge, where strange things like trucks blocking the road to do deliveries, are frequently found.

It's a pretty good bet that almost all our urban spaces will have data connectivity in the 2020s. If any street doesn't have solid data, and has frequent bizarre problems of any type, yet is really important for traversal by unmanned vehicles -- an unlikely trifecta -- it's quite reasonable for vehicle operators to install local connectivity (with wifi, for example) on that street if they can't wait for the mobile data companies to do it. Otherwise, don't go down such streets in empty cars unless you are doing a pickup/drop-off on the street.

Switching Cities

Karl Iagenemma of nuTonomy told the story of moving their cars from Singapore, where driving is very regulated and done on the left, to Boston where it is chaotic and done on the right.

I've written a few times that perhaps the biggest unsolved problem in robocars is how to know we have made them safe enough. While most people think of that in terms of government certification, the truth is that the teams building the cars are very focused on this, and know more about it than any regulator, but they still don't know enough. The challenge is going to be convincing your board of directors that the car is safe enough to release, for if it is not, it could ruin the company that releases it, at least if it's a big company with a reputation.

While very few details have come out, Reuters reports that new proposed congressional bills on self-driving cars will reverse many of the provisions I critiqued in the NHTSA regulations last year.

Plans are underway to ask for a legal mandate to install radio communications devices in all new cars, starting around 2020. These radios would do "vehicle to vehicle" (v2v) and vehicle to infrastructure communication using a wifi-derived protocol called DSRC.

Waymo (Google) has announced a pilot project in Phoenix offering a full ride service, with daily use, in their new minivans. Members of the public can sign up -- the link is sure to be overwhelmed with applicants, but it has videos and more details -- and some families are already participating. There's also a Waymo Blog post. I was in Phoenix this morning as it turns out, but to tell real estate developers about robocars, not for this.

There are several things notable about Waymo's pilot:

1. They are attempting to cover a large area -- they claim twice the size of San Francisco, or 90 square miles. That's a lot. It's enough to cover the vast majority of trips for some pilot users. In other words, this is the first pilot which can test what it's like to offer a "car replacement."
2. They are pushing at families, which means even moving children, including those not of driving age. The mother in the video expects to use it to send some children to activities. While I am sure there will be safety drivers watching over things, trusting children to the vehicles is a big milestone. Google's safety record (with safety drivers) suggests this is actually a very safe choice for the parents, but there is emotion over trusting children to robots (other than the ones that go up and down shafts in buildings.)
3. In the videos, they are acting like there are no safety drivers, but there surely are, for legal reasons as well as safety.
4. They are using the Pacifia minivans. The Firefly bubble cars are too slow for anything but neighbourhood operation. The minivans feature motorized doors, a feature which, though minor and commonplace, meets the image of what you want from a self-driving car.

Apple is in the game

There has been much speculation recently because of some departures from Apple's car team that they had given up. In fact, last week they applied for self-driving car test plates for California. I never thought they had left the game.

Not everybody loves video calls, but there are times when they are great. I like them with family, and I try to insist on them when negotiating, because body language is important. So I've watched as we've increased the quality and ease of use.

The ultimate goals would be "retinal" resolution -- where the resolution surpasses your eye -- along with high dynamic range, stereo, light field, telepresence mobility and VR/AR with headset image removal. Eventually we'll be able to make a video call or telepresence experience so good it's a little hard to tell from actually being there. This will affect how much we fly for business meetings, travel inside towns, life for bedridden and low mobility people and more.

Here's a proposal for how to provide that very high or retinal resolution without needing hundreds of megabits of high quality bandwidth.

Many people have observed that the human eye is high resolution on in the center of attention, known as the fovea centralis. If you make a display that's sharp where a person is looking, and blurry out at the edges, the eye won't notice -- until of course it quickly moves to another section of the image and the brain will show you the tunnel vision.

Decades ago, people designing flight simulators combined "gaze tracking," where you spot in real time where a person is looking with the foveal concept so that the simulator only rendered the scene in high resolution where the pilot's eyes were. In those days in particular, rendering a whole immersive scene at high resolution wasn't possible. Even today it's a bit expensive. The trick is you have to be fast -- when the eye darts to a new location, you have to render it at high-res within milliseconds, or we notice. Of course, to an outside viewer, such a system looks crazy, and with today's technology, it's still challenging to make it work.

With a video call, it's even more challenging. If a person moves their eyes (or in AR/VR their head) and you need to get a high resolution stream of the new point of attention, it can take a long time -- perhaps hundreds of milliseconds -- to send that signal to the remote camera, have it adjust the feed, and then get that new feed back to you. There is no way the user will not see their new target as blurry for way too long. While it would still be workable, it will not be comfortable or seem real. For VR video conferencing it's even an issue for people turning their head. For now, to get a high resolution remote VR experience would require sending probably a half-sphere of full resolution video. The delay is probably tolerable if the person wants to turn their head enough to look behind them.

One opposite approach being taken for low bandwidth video is the use of "avatars" -- animated cartoons of the other speaker which are driven by motion capture on the other end. You've seen characters in movies like Sméagol, the blue Na'vi of the movie Avatar and perhaps the young Jeff Bridges (acted by old Jeff Bridges) in Tron: Legacy. Cartoon avatars are preferred because of what we call the Uncanny Valley -- people notice flaws in attempts at total realism and just ignore them in cartoonish renderings. But we are now able to do moderately decent realistic renderings, and this is slowly improving.

My thought is to combine foveal video with animated avatars for brief moments after saccades and then gently blend them towards the true image when it arrives. Here's how.

1. The remote camera will send video with increasing resolution towards the foveal attention point. It will also be scanning the entire scene and making a capture of all motion of the face and body, probably with the use of 3D scanning techniques like time-of-flight or structured light. It will also be, in background bandwidth, updating the static model of the people in the scene and the room.
2. Upon a saccade, the viewer's display will immediately (within milliseconds) combine the blurry image of the new target with the motion capture data, along with the face model data received, and render a generated view of the new target. It will transmit the new target to the remote.
3. The remote, when receiving the new target, will now switch the primary video stream to a foveal density video of it.
4. When the new video stream starts arriving, the viewer's display will attempt to blend them, creating a plausible transition between the rendered scene and the real scene, gradually correcting any differences between them until the video is 100% real
5. In addition, both systems will be making predictions about what the likely target of next attention is. We tend to focus our eyes on certain places, notably the mouth and eyes, so there are some places that are more likely to be looked at next. Some portion of the spare bandwidth would be allocated to also sending those at higher resolution -- either full resolution if possible, or with better resolution to improve the quality of the animated rendering.

The animated rendering will, today, both be slightly wrong, and also suffer from the uncanny valley problem. My hope is that if this is short lived enough, it will be less noticeable, or not be that bothersome. It will be possible to trade off how long it takes to blend the generated video over to the real video. The longer you take, the less jarring any error correction will be, but the longer the image is "uncanny."

While there are 100 million photoreceptors in the whole eye, but only about a million nerve fibers going out. It would still be expensive to deliver this full resolution in the attention spot and most likely next spots, but it's much less bandwidth than sending the whole scene. Even if full resolution is not delivered, much better resolution can be offered.

Stereo and simulated 3D

You can also do this in stereo to provide 3D. Another interesting approach was done at CMU called pseudo 3D. I recommend you check out the video. This system captures the background and moves the flat head against it as the viewer moves their head. The result looks surprisingly good.

Luminar, a bay area startup, has revealed details on their new LIDAR. Unlike all other commercial offerings, this is a LIDAR using 1.5 micron infrared light. They hope to sell it for $1,000.

There's a lot of bad information circulating on the famous United/Republic "passenger drag" so I wanted to consolidate a 2nd post with some of them.

Myth: This was an oversold flight

It turns out the flight was probably not oversold. A UA spokesman said it wasn't. It was a fully sold flight, but a sudden need arose to move 4 flight attendants to SDF (Louisville) and they arrived at the gate after the flight had boarded. In United's contract of carriage, it defines an oversold flight as a flight where there are more passengers with confirmed reservations checked in by the check-in deadline than they have seats on the plane. That does not appear to be the case on this flight, but Republic and UA got confused about it.

That, in turn, means Republic did not have the right to invoke the clauses of the contract for oversold flights. If so, they are just plain in the wrong, and this becomes a case with far less interesting nuance. United has changed their tune (of course due to public pressure) and are going full mea culpa.

Airline reservation computers oversell all the time, and carefully calculate exactly how much to oversell. It looks like the algorithms decided to not oversell this flight. And they were right -- when they called for volunteers, nobody accepted, even at a very high price ($800 to $1,000) for a flight where most tickets are under $200. The algorithms performed perfectly.

Myth: This was United Airlines

Technically it was Republic Airline, a small regional airline dba "United Express." However, United sells and and manages the tickets and they use the brand, and it's under United's contract, so United certainly gets a lot of the responsibility. And I am impressed that UA has not tried to throw Republic under the bus here.

Republic Airways actually operates lots of regional flights for United, AA and Delta, so this could have probably happened to any of them. I don't know if they have a lot of airline specific training on bumping procedure for their teams. United may have just gotten some very bad luck of the draw here -- and then made it worse by defending it at first. And it may be that the bumping policies UA gives to Republic might have made this more likely than the ones Delta and AA give it, but I don't think they are tremendously different. Some hinges on whether the flight crew was a Republic crew, or a United crew.

But still, though it was not United, the buck stops with United, and at least now, they are not resisting that at all.

Myth: On an oversold flight, they can pull passengers off the plane.

If this had been an oversold flight, their contract still does not let them remove passengers from the plane involuntarily. It says they can "deny boarding." Deny boarding does not mean remove -- there is another section of the contract on removal. More bad news for United/Republic, but again, it makes the case less interesting as it's an example of something you sort of expect -- junior employees of a regional affiliate not being properly trained on what to do in an unusual situation and thus screwing up. That happens in 100 different ways all the time, but each particular incident is rare and probably does not indicate a systemic problem. That's good -- but it is only systemic problems that are of interest to the public, and which would make you boycott a company. If the junior employees make mistakes like this too often, then you have a systemic problem to worry about. (United does not have a good reputation on this count, of course.)

Update: These flight attendants were "must ride" passengers

New information reveals the flight crew declared themselves "must ride." I don't have a lot of details, but this is a special designation in the law (not the UA contract) which declares the crew are needed somewhere to avoid cancellation of a flight. Once a passenger is declared "must ride" the plane is required, reports say, to do everything possible to get that passenger to their destination, including delaying the plane and apparent, yes, even involuntary bumping. I am waiting for more information on this status, which would invalidate partly what I say above. They can't pull you for an oversell, but they may be able to pull you for a must-ride. The law is there to keep the aviation system humming. Once flight crews don't get to flights, it can mean disruption to more than just that flight.

Myth: If the doctor had just handed one of the police officers a Pepsi, it all would have been defused.

No, but that's the best joke on this that I've seen.

Myth: It's overselling that's the problem

With the mistaken impression that overselling was the cause here, a lot of people are stating overselling is evil, and Chris Christie has even called to prohibit it. That's a big mistake. Overselling is very good for airlines and the flying public. I explained that in yesterday's post but I will go into more details below. You want an airline that does at least some overselling, though one can debate how much you want.

Myth: The airline prioritized employees over paying customers

In this situation, it needed to move those employees to crew a flight first thing out of SDF. If they had not gotten a crew there, that flight gets cancelled. Roughly 70 paying passengers get stranded against their will. While clearly nobody wants to be stranded against their will, the hard truth is you want to fly an airline that will strand (with good compensation) 4 people to avoid doing it to 70. (Here I am talking about the normal approach, which is to deny boarding to 4 people, not to try do drag people off the plane.) Still, I have to view it as prioritizing 70 passengers over 4, not employees over passengers.

Maybe: They could have driven the flight attendants there or chartered a jet

This is possibly true but possibly not. First of all, these airlines are all about procedure. They don't authorize junior employees to be innovative or authorize them to spend money. So chances are if somebody thought of that, they had no system with which to do it. That is a fault of the airline but the sad norm of corporate bureaucracy.

Secondly, while I don't know this to be true, all flight crew operate under a set of complex rules about required rest. You don't want a sleepy pilot landing your plane, or a sleepy flight attendant helping people get onto the evacuation slides. These rules are very hard and fast. I suspect trying to sleep in a car doesn't count, and an overnight ground ride is out of the question. Had they acted very quickly, and had a system in place, they could have probably gotten the crew there a bit after 11 -- not long after the flight actually landed due to the chaos -- so that might have worked, in hindsight.

They could have offered the passengers a limo ride, but again probably had no way to do something that out of the ordinary.

The same applies to an air charter. Getting an air charter on short notice is difficult, but they could have gotten one for the flight crew (or another flight crew) in the early morning if they had a system in place. This is very expensive of course, and so not likely to be in their playbook.

Maybe: They should never have gotten to the point where it was so urgent to get that flight crew moved

Airlines move flight crews a lot. There are airline pilots who live on one coast and mostly work on the other, commuting by "deadheading" on one of their airlines planes.

When you design a system that needs various parts -- planes and flight crew -- you have to "overprovision," which is to say leave some wiggle room. That means you have some number of planes, crews and other resources sitting idle or on call, and you use them when something else fails. Everybody does it because you don't want to run so close to the wire all the time. If you do, the slightest problem causes a cascade of cancellations. Airlines have to worry not just about small problems but even big ones like storms that cancel or delay many flights.

It's not practical, however, to overprovision to the point that you never fail. You can do it, but it's really expensive. You have to waste a lot of money, and you don't have a competitive company. So every systems designer tries to figure out how to overprovision just the right amount. An amount that will have a few failures, but not too many. On top of that, you try to plan so you handle those failures with the least amount of pain, but you accept they will still happen.

What that means -- and I don't have any specific facts about this flight -- is that sometimes you will be skating the edge, and sometimes you will fail. Sometimes you will find that crew are not going to make a flight unless you do something a little extra.

The bumping law, I think, is where the airlines find their "extra." They don't want to bump paying customers -- it's expensive and hurts customer relations. But they don't want to cancel flights even more. So every so often, every airline has to find a solution. The bumping law offers them that solution. They can legally deny boarding to paying passengers against their will to make room for crew. This is much more workable, and under their control, than other options like using charter jets, or if distances are short, ground service.

True, but: Just about anything would be cheaper than the hit they've taken

That's true -- but only in hindsight. No playbook for these situations is going to say, "If you have to, spend $10,000 rather than bumping passengers just in case it turns into the PR nightmare of the year." By definition, nobody knew that would happen.

In reality, airlines involuntarily bump 50,000 pax per year and while they grumble, this is the first time it's ever gone done like this, with eviction from the plane, blood, camera phones and Facebook. So I don't blame them for not seeing this could happen. I do blame them, however, for not understanding that any time you bring the police into a situation you bump the risk of something bad happening.

True, but: They should have known this would happen once they called the goons.

They should have known, but I can suspect why they didn't -- because they actually do this all the time and don't have PR problems. Flight crews face unruly passengers reasonably often. They have training for it and procedures. And those procedures do call for getting the police, even knowing how that can go south. What those plans obviously did not account for was doing this when it was completely clear the passenger was the victim, that they only removed him because they wanted his seat. The rules for removing passengers mostly deal with safety issues. When they declare a passenger a safety risk, and the passenger makes trouble and even (rarely) causes a scuffle they are protected if the passenger was really a safety risk, or they can even come up with a credible lie why they thought he was a safety risk. No such story is possible here. Sure, the law says anybody who refuses a flight crew order can be removed from the plane. Technically it says this. In reality, it's insane to think you can remove somebody for refusing the order "leave the plane" when the order is not given for a valid reason. The law says obey, but every sense of justice goes the other way. In fact, more than that, I don't think a court would convict somebody for refusing that order, even if they are guilty, because society does not intend to grant the airlines that sort of power.

Put another way, three things are true:

1. They can't order you off the plane just to take your seat (but they didn't know that.) We don't want airlines to have that power.
2. Once somebody refuses a flight crew order, you can then order them off the plane.

As such, it's clear that "we removed him because he disobeyed our order to leave" is a loophole that would never stand up to scrutiny.

Myth: I should worry this can happen to me.

Well, I have to concede this is true -- part of this did happen to me! The first flight I took with Kathryn, the airline came up to us after we had boarded, and insisted she give up her seat for a deadheading pilot. The pilot never sat there -- instead he went up to use the jumpseat in the cockpit. We were quite angry, especially when her later flight lost an engine in the middle of the Pacific.

But a lot had to go wrong for this to happen. Here's my guess as to the list of things that went wrong:

• Something failed in the planned movement of flight crew, and they needed to get a crew to SDF for a Monday Morning flight. They looked over their options, and decided to try to get on UA3411
• They decided that very late, so the flight had already boarded full when the flight crew came to the gate and said they needed to be on that plane. (I don't know why they selected this one over the next, I presume both were full, or the next one might even have been oversold. You want to avoid the last flight in any event.)
• They tried the normal approach -- offer an incentive for volunteers. They got to $800. (UA says $1,000.) It failed. Nobody bit. This is a flight where everybody needed to get to SDF.
• They didn't know their contract well, and decided they could do involuntary bump to solve their problem. Why not, it's what they usually do, right? They got mean, declaring the plane would not fly until 4 got off.
• They really didn't know their contract well, and figured they could involuntary bump by removing passengers from the plane. They can't, but they told people they had to leave.
• Usually that works. In fact, I suspect it's worked pretty much every time for decades. Not this time. One man refuses to leave. Now they had a passenger refusing flight crew orders.
• A non-compliant passenger is something they are trained for. They follow their procedure. He won't leave. They follow their procedure and call in airport cops.
• The airport cops are thugs. They manhandle him, injure him and drag him. All recorded on camera phones.
• It explodes on the social networks. The company has no idea how to handle it, and botches that too.

Because so many things had to go wrong, the particular situation is not important. Rare things go wrong all the time. Junior staff at small airlines are not fully trained on contract nuances. Because things had never gone south like this before (and not in the way the plane was supposed to literally fly south) nobody had ever thought to write up procedures to remind gate crews that they can't remove passengers, and that they can't bump at all if it's not actually oversold.

Those of us writing so much about this online only really want to care about systemic problems. What is wrong with the system, not just one gate crew or flight crew. If there is a pattern of errors, what can be done to fix it.

Myth: That poor doctor!

I am hesitant to include this one, because I don't want to give the impression that I am defending in any way what happened to him, but it is an important fact. I am not saying anybody should be forcibly removed from a plane because the airline wants his seat. This was not just your ordinary passenger. Reports claim Dr. Dao lost his licence to practice medicine from 2003 to 2016 because he was convicted of trading prescription painkillers for sex, and his psych evaluations listed him as having anger management issues. One reason this escalated is that normally nobody dares to defy orders from the flight crew and especially from police. The orders were improper, and the bumped passengers deserve lots of compensation, but you have to attribute some portion of the blame for how far it escalated to Dr. Dao.

So, is overbooking evil or good?

The big question I have found most interesting is the subject of overbooking. Almost all airlines sell more seats on a plane than it actually has. They give you what they call a "confirmed reservation" and that name certainly makes people imagine they have a guaranteed seat on the plane. They don't, but they almost do, and that's as I will explain, a good thing for the flying public.

One basic statistic -- the no-show rate on flights is around 8%. So a plane with 100 seats, if it is considered "sold out" after 100 reservations. On average, with no overselling or standby pax, it would take off with 8 empty seats. The number is not the same for every flight. Complex algorithms predict the actual number based on history of that flight and the passengers.

Myth: The airlines primarily do this as a fraud to make money by selling the same seat twice

Turns out, when people don't fill their seat, only rarely does the airline get any money, or a profit from them. Airlines do make money from overbooking, but not the way you think. Most of those no-shows are because of late or cancelled connections. Those are money losers for the airline, big time. They have to rush to find another flight for that passenger, and get no money. Some of them are people who did free same-day changes or otherwise switched off the flight for low fee. A few have tickets with no change fees. A few more did a late flight change and paid a change fee. The change fee is sometimes as high as the ticket, but sometimes it's much less. The airline pockets the change fee, but not without cost -- the biggest one being they turned away passengers they would not have turned away because of the booking.

Update: More careful reading of United's Contract suggests both that this didn't fit the definition of an oversold flight, and that even if it did, they only have the power to "deny boarding" to a bumped passenger, not to remove them from an aircraft. If this is true, then this case is simple and much less interesting: UA/Republic should admit fault and compensate those involved and retrain staff. End of that part of the story. Later-update: This might might have involved a special "Must ride" classification put on the flight crew which changes the rule yet again.

I have a follow-on post on misconceptions and realities about these issues.

The viral video of the day is that of police pulling a main from a United Airlines flight. He doesn't want to go, and they pull him out, and bash his head on the armrest, then drag out his unconscious body. It's a nightmare for everybody, and the video sends clear chills into every viewer. (Once, after I changed my flight to fly home from Hawai`i with Kathryn, they involuntarily removed her from the plane for a crew member. I spent the flight next to an empty seat as the crew member went to the cockpit jumpseat, and she flew on a later flight that lost an engine. We've never flown on that airline again.)

In spite of that, I have some sympathy for both sides, and while clearly things went very wrong here, as even United will eventually admit, the more interesting question for me is "what should airlines do to make this work better"? I do believe that UA clearly didn't want this to happen, though their policies created a small risk that it would. I am sure they don't want it to happen again. So if you were the person writing the policy for these situations, what would you do?

The situation:

• This was UA3411, UA's 2nd last flight from ORD to Louisville. UA (or rather Republic airlines, a small regional flying under the United Express logo) had 4 flight crew who were needed for an early flight from Louisville and, I presume, had no other option for getting them there. (The next flight was obviously more oversold.) If they don't get there, and sleep the legally required amount, that flight is canceled and a whole lot of people don't fly, and a bunch of other flights are affected too. Aviation rules are strict on this.
• In an unusual situation, the four flight attendants are not expected. It is quite common for flight crew moving to their next job to be on flights and displace paying passengers, but unusual for it to be a surprise, to happen after the passengers have already boarded a full flight.
• So they ask ( as is required by law) for people to volunteer to get off in exchange for a reward. Unfortunately, all they can offer is a flight Monday afternoon. Nobody wants that, apparently, and the offer gets up to $800 plus hotel. Tickets on this 90 minute flight are only $187, but nobody wants the offer. That's also unusual.
• The law then gives the airline another option, involuntary bump. They tell the passengers they will do this if nobody volunteers. They select a pool of "low priority" passengers (those who took super-discount fares, removing elites and the disabled and a few others.) They pick 4 at random.
• 3 of those selected get off. The law requires they get a compensation of around $800 but in cash, not coupons. One, a doctor, refuses. He tells some people he has to see patients in the morning.
• They say the plane can't take off until this passenger leaves. He won't. They call the airport cops. The airport cops come to his seat to remove him.
• You can see what happens next on the video. He won't go. They physically try to pull him out. He screams and clings to the seat. They pull harder. He hits his head on the opposite armrest and is knocked out.
• They drag his limp form from the plane -- you can see that on video.
• Amazingly, he somehow gets back on the plane, bloodied and a bit confused. He keeps repeating, "I have to get home." He does not appear to be wearing leggings.

New information reveals that a whole bunch of things went wrong at once, which does not excuse police manhandling a passenger, but helps us understand why it went pear-shaped.

First, understanding overselling -- and why the flying public wants it

Most flights these days are oversold, because a lot of people don't show for their flights. The system of overselling, then calling for volunteers when too many show up makes the planes fly mostly full these days on many routes. It's a fact of flying and allowed in the law. It makes flight more efficient, perhaps 5-10% more. On competitive routes, that makes tickets cheaper for everybody. It has another benefit to the flying public -- more people get to fly on the flight they want, because the airline is willing to sell you a seat on a "full" flight, knowing that 99% of the time you and everybody else will actually get to fly. The alternative is that an empty seat flies, and you wastefully take another flight. Passengers really like more availability, though they don't directly see how it happens. The reality is many of the flights you see in your web search are technically oversold. If it is really sold out, it's actually oversold past their limit.

Airlines could elect to not oversell, or not oversell as much, but that comes with a cost. More people denied the flight they want. More expensive tickets. More emissions per passenger. The world doesn't want that, so the world allows and the law regulates, overselling.

Of course, there is a way to avoid ever being bumped. Pay more for your ticket, or be an elite flyer, as I am. (In fact, as an elite, they actually guarantee me a seat on "really, really sold out" flights 24 hours in advance, which really means they push their oversell percentage by plus-one for elites. If I do this -- I never have -- they just decide it is cheaper to pay a volunteer to get off the flight than to deny one of their elites the flight they need.)

So the most obvious solution, "Don't oversell," comes with a cost I don't think the airlines or flying public actually want. Consider it this way. A flight you need with 100 seats has had 100 bookings. The airline knows that on average 7 of them won't show up. Do you want the airline to let you "reserve" on that plane, or tell you "sorry, fly the next day?" Do you want them to only offer you a standby ticket because other people, who paid far less than you for their tickets and who barely fly on their airline, got there first? (And yes, those people who buy late pay a premium.) The airline hates taking off with an empty seat, but you hate being told you can't get on a flight that ended up with empty seats even more.

Airlines are getting quite good at it. In 2015, only .09% of passengers were bumped, and only .01% involuntarily.

The public wants bumping for flight crew, too!

Turns out, it's in the public interest that flight crew needed for another flight have higher priority than we do, even to the point of removing us from planes we already boarded. That may not be allowed, but one has to consider the difference between one person removed (voluntarily or not) with compensation and the very large group of people who will have their flight cancelled (sometimes with no compensation) if the flight crew doesn't get there, properly rested and ready. You don't want to be either, and utilitarianism is not always the right philosophy, but here the numbers are overwhelming. One guy doesn't fly or 70 people don't. So we want a system where that can happen, but smoothly and ideally voluntarily.

Understand involuntary bumping

Usually, the system of offering fat compensation -- $800, a hotel and meals for a $180 flight is a pretty good deal -- works fine. There are people who actually relish it. I met one guy who says he deliberately tries to get bumped the day before Thanksgiving -- when the offers get very high. But nobody was taking it. Most would miss a day of work, which is not an easy thing to do.

The law then allows the airlines to do an involuntary bumping. They have an algorithm that picks people and they are "denied boarding." The law specifies compensation. In this case 4 times the ticket price and other compensations. And this is cash, not flight coupons. Cash is worth a lot more.

This law is one of the culprits here. The law effectively puts a cap on the offer you will get. The airlines, in a move they thought at first was rational, don't want to offer you a lot more than the price the law defines for an involuntary bump. Why give a passenger $2,000 when you can do it for $1,000 under the law. Well, one reason is bad PR -- which is true in spades here.

The airlines don't want to do this. About 1 in 1,000 passengers are bumped, and 1 in 10,000 are involuntarily bumped, and has been going down as they get better at working their systems. But it happens.

Without the involuntary rule, the airline might have considered the next solution...

Make better offers for voluntary bumping

This problem would have been defused if they had kept increasing the offer until somebody took it. (Those who took it early will of course be upset, but that's how it goes.) While there is a practical limit, a volunteer should be found long before it.

They could also consider other things that are not money. Often bump offers come with things like first class upgrades which can be cheap for the airline and very nice to the passenger. They could offer a very coveted thing to some passengers -- elite qualification. At the extreme, if they offered 20,000 elite qualification miles or a full-tier bump in elite status, I could see even elite passengers jumping up to volunteer. We don't usually. We know we will never get involuntarily bumped. We usually have places to go. But we crave that elite status so much that some people fly "mileage runs" -- flights to nowhere just to accumulate miles -- to keep and increment it. If UA said, "get off this plane and we'll make you 1K" they would have had a line out the door of volunteers.

A new report from Navigant Research includes the chart shown below, ranking various teams on the race to robocar deployment. It's causing lots of press headlines about how Ford is the top company and companies like Google and Uber are far behind.

I elected not to buy the $3800 report, but based on the summary I believe their conclusions are ill founded to say the least.