This weekend I went to Pomona, CA for the 2015 DARPA Robotics Challenge which had robots (mostly humanoid) compete at a variety of disaster response and assistance tasks. This contest, a successor of sorts to the original DARPA Grand Challenge which changed the world by giving us robocars, got a fair bit of press, but a lot of it was around this video showing various robots falling down when doing the course:

What you don't hear in this video are the cries of sympathy from the crowd of thousands watching -- akin to when a figure skater might fall down -- or the cheers as each robot would complete a simple task to get a point. These cheers and sympathies were not just for the human team members, but in an anthropomorphic way for the robots themselves. Most of the public reaction to this video included declarations that one need not be too afraid of our future robot overlords just yet. It's probably better to watch the DARPA official video which has a little audience reaction.

Don't be fooled as well by the lesser-known fact that there was a lot of remote human tele-operation involved in the running of the course.

Check out my Gallery of Photos from the DARPA Robotics Challenge Finals.

What you also don't see in this video is just how very far the robots have come since the first round of trials in December 2013. During those trials the amount of remote human operation was very high, and there weren't a lot of great fall videos because the robots had tethers that would catch them if they fell. (These robots are heavy and many took serious damage when falling, so almost all testing is done with a crane, hoist or tether able to catch the robot during the many falls which do occur.)

We aren't yet anywhere close to having robots that could do tasks like these autonomously, so for now the research is in making robots that can do tasks with more and more autonomy with higher level decisions made by remote humans. The tasks in the contest were:

• Starting in a car, drive it down a simple course with a few turns and park it by a door.
• Get out of the car -- one of the harder tasks as it turns out, and one that demanded a more humanoid form
• Go to a door and open it
• Walk through the door into a room
• In the room, go up to a valve with circular handle and turn it 360 degrees
• Pick up a power drill, and use it to cut a large enough hole in a sheet of drywall
• Perform a surprise task -- in this case throwing a lever on day one, and on day 2 unplugging a power cord and plugging it into another socket
• Either walk over a field of cinder blocks, or roll through a field of light debris
• Climb a set of stairs

The robots have an hour to do this, so they are often extremely slow, and yet to the surprise of most, the audience -- a crowd of thousands and thousands more online -- watched with fascination and cheering. Even when robots would take a step once a minute, or pause at a task for several minutes, or would get into a problem and spend 10 minutes getting fixed by humans as a penalty.

Four wheels good, two legs bad?

One of the great debates in robotics is whether the humanoid form makes sense. It is popular in fiction, of course, but it's much harder to attain. You need to balance, you need to be able to stand on one foot, you have most of your weight up high, and to really walk like humans you even balance on your toes. It uses far more energy to travel and is slower. Legs can handle many more types of terrain, though the 4 legs of robots like BigDog (or mountain goats, the inspiration) seem able to do more. Rolling is vastly easier and more efficient, but can't do in as many places -- though tracked rollers are able to cover fairly rough terrain and even climb stairs.

A number of the robots in this contest took a hybrid approach. They were roughly humanoid (or simian) but had tracks or wheels on their feet and sometimes their hands or knees. This allowed them to switch to a 4-wheeled mode when moving on a flat surface for any distance. This made them much faster, particularly because any walking for now is quite slow due to the risk of falling.

Strangely, the wheeled robots had an advantage and the debris task. While they could not use the wheels to go over the cinder blocks in most cases, the alternative -- the field of debris -- turned out to have debris that was so light that the big heavy wheeled robots could just bulldoze through it, pushing it aside. (Though one robot did get a wheel caught.) In theory, the contest was supposed to involve the robots using their arms to clear the debris. The winning team even complained this was too easy, because they had worked on doing the harder task but would have been crazy to waste time doing it.

Most of the top teams, except the 2nd place IHMC crew, used wheels in some way. Of course, one thing that strongly discouraged wheels was the fact that DARPA helped several teams by providing them with an Atlas humanoid robot from Boston Dynamics (makers of BigDog et al.) The Atlas robot, which would have cost a fortune to make, gave teams the ability to not worry as much about hardware but did constrain what they did. The winning team from Korea (KAIST) had their own hardware division to make their own Robot, as did a few other teams. Some teams collaborated on making the hardware platform, since building a custom robot is so expensive it makes a great deal of sense to share. It's so expensive, in fact, that even with the $2 million prize, building your own robot would likely mean still losing money.

The fastest robot on Day 1, Nimbro, had 4 combination wheel/legs and was very non-humanoid. Watch its time-lapse.

A few of the top robots had fairly non-humanoid forms. The 3rd place Tartan Rescue robot named Chimp, as well as the 5th place Robosimian from JPL had more ape-like forms, walking or rolling on 4 legs, and doing that to climb the stairs and get out of the car. There was even some argument that we should not try to make our robots too humanoid, because that makes people see them as human replacements, rather than as tools.

Remote operation and autonomy

It will disappoint many to learn that there was still lots of human involvement in the running of the course. These robots had the autonomy to do very basic tasks, like walking or rolling, or rotating an arm, or grasping, but all high level decisions were done by a team in a garage on the other side of the fairgrounds. They could not see the video feed, and to force them to make more autonomy for their robots, their data connections to the robots were deliberately seriously degraded once they opened the door and went into the room. At that point they were reduced to a 9600 bps feed, plus 1 second bursts at random every 30 seconds of the high speed feed. This meant it was not possible, as some had tried in the trials, to use the robots more like radio controlled waldo devices.

What happened varied from team to team and task to task. In most cases though, teams would look at the high-res images and 3D scans that would come from the short bursts, and then tell the robot to move to a certain spot or put their hand in a given spot. They would then wait up to 30 seconds to get another burst and figure out what to do. Some robots were much better. For example, in the wall drilling task, some robots, once told where to cut the circle, could do that on their own.

Most of the delays that made the progress so slow involved this loop back to the humans. As these contests progress, and there is more autonomy, the robots will get faster and more interesting to watch. I think even the 9600bps channel is too much for a real test, because it was (mostly) up all the time. Some teams were able to think back to the days when we considered 9600bps fast and build interfaces that worked with that. Others seemed to live and die by the short high-speed (300mbps) bursts.

To make all this happen, they used network stressing equipment to create the degraded connections, but they also banned all 5ghz wifi at the event to make sure that they only got artificial problems with the network, not real ones. To screw them up, packets sometimes arrived out of order, and of as in all networks, some may have been lost.

Once again, the message is not to be too disappointed in the amount of remote control, but look at the trend, and how much less remote control was needed in the finals, just 18 months after the trials. Likewise consider the move from needing a tether due to constant falling to having to move on their own, with serious risks from falling. One robot which fell broke a fluid line and was out of competition. One robot -- the 3rd place Tartan Rescue team from CMU -- actually had their 4 legged CHIMP able to get up again, avoiding the 10 minute reset penalty -- they were the only team to get all 8 points on day 1.