Building a house organizing robot with image search

There are many fields that people expect robotics to change in the consumer space. I write regularly about transportation, and many feel that robots to assist the elderly will be the other big field. The first successful consumer robot (outside of entertainment) was the Roomba, a house cleaning robot. So I've often wondered about how far we are from a robot that can tidy up the house. People got excited with a PR2 robot was programmed to fold towels.

This is a hard problem because it seems such a robot needs to do general object recognition and manipulation, something we're pretty far from doing. Special purpose household chore robots, like the Roomba, might appear first. (A gutter cleaner is already on the market.)

Recently I was pondering what we might do with a robot that is able to pick up objects gently, but isn't that good at recognizing them. Such a robot might not identify the objects, but it could photograph them, and put them in bins. The members of the household could then go to their computers and see a visual catalog of all the things that have been put away, and an indicator of where it was put. This would make it easy to find objects.

The catalog could trivially be sorted by when the items were put away, which might well make it easy to browse for something put away recently. But the fact that we can't do general object recognition does not mean we can't do a lot of useful things with photographs and sensor readings (including precise weight and other factors) beyond that. One could certainly search by colour, by general size and shape, and by weight and other characteristics like rigidity. The item could be photographed in a 360 view by being spun on a table or in the grasping arm, or which a rotating camera. It could also be laser-scanned or 3D photographed with new cheap 3D camera techniques.

When looking for a specific object, one could find it by drawing a sketch of the object -- software is already able to find photos that are similar to a sketch. But more is possible. Typing in the name of what you're looking for could bring up the results of a web image search on that string, and you could find a photo of a similar object, and then ask the object search engine to find photos of objects that are similar. While ideally the object was photographed from all angles, there are already many comparison algorithms that survive scaling and rotation to match up objects.

The result would be a fairly workable search engine for the objects of your life that were picked up by the robot. I suspect that you could quickly find your item and learn just exactly where it was.

Certain types of objects could be recognized by the robot, such as books, papers and magazines. For those, bar-codes could be read, or printing could be scanned with OCR. Books might be shelved at random in the library but be easily found. Papers might be hard to manipulate but could at least be stacked, possibly with small divider sheets inserted between them with numbers on them, so that you could look for the top page of any collected group of papers and be told, "it's under divider 20 in the stack of papers."

The robot would of course do silly things like put away garbage, and put away things you want left out. One could go through the bins and toss out the trash, or go into the visual index and hit "delete" or "recycle" on certain items. Aside from removing them from your index, the robot could really "empty the trash," going to the bins and finding those items that you deleted, and moving them to a physical trash bin. (In case of mistakes you still might want to scan this bin before telling the robot to truly throw it out, but with things like exact weight measurements and detailed scans, the mistakes should be few.)

Another option for many items would be to apply cheap RFID tags to them. This would allow fully reliable identification. (There are not so many privacy implications for RFIDs on items that stay in your house, and if you want to RFID tag your clothing, it's possible to build tags that only respond to an authorized reader that knows their key.) You could also have small tokens to put near items to say "don't clear away items near this. The robot could also be trained with many items to know not to put them away.

The first big challenge is making sure the robot doesn't break things picking them up. That means it has to not squeeze too hard yet also not drop items. Some items may present real challenges, and would mean the humans would have to be careful not to leave out things that might be damaged by the robot. But then again, humans are easier to train about this than robots. Stuff sitting in a pile presents a problem, and so cleaning a house that has accumulated clutter could be difficult, but a house that had regular robot tidying would not accumulate too many of those. If presented with the house in a clean state, the robot's laser scanners could build a map of all the things that are left out in the clean state so it knows not to try to put away the lamps and coffee table books and art pieces.

One could also do a much simpler, but less useful robot that is only able to examine, photograph and pick things up which are isolated on a table or floor area. To make use of this robot, you, your family members or even housekeeper would pick up all the clutter and put it on the table, taking care not to put fragile items the robot can't pick up on the surface. The robot could then process them into bins. If the bins were on wheels, or part of the robot, the robot itself would not need to be something that was able to navigate the house. Indeed, one of the robot's pick-up devices could even be something akin to a thin tray (like the tray used with brooms) that can pick up anything from underneath, spin it for the camera and laser, and gently tilt it into the bin.

Some special attention would be needed in the kitchen. You don't want to put dirty dishes or smelly trash in the bins. It's possible that developments in digital "noses" could help here. Big items are their own issue, unless you have very large amounts of bin space. The robot does not have to climb stairs, but it would need to be light enough that the owners could move it to different floors it has to tidy. It would need bins on each floor.

A particularly popular application might be toys and the other things children leave out. These tend to not be fragile, to be brightly coloured, and easy to RFID tag even if the tag is ugly.

Today we have often taken to using search engines and full indexing to find things on our computers (and on the internet) rather than file hierarchies and organized directories. How much might we be able to do that for the vast mountains of stuff in our lives?


I would hand over my life savings for a robot that could sort Thomas the Tank Engine trains, Lego Duplo blocks, and Hot Wheels, into bins as my children are not capable of this feat.

For kids' toys there are a few issues. The kids may or may not be able to readily use the visual search engine to find their toys. On the other hand, they might be quite tolerant of the toys just being in a big collection of bins, since they will just throw them into the boxes they are given today. On the third hand, because parents would indeed be a top market for house cleaning robots, training the robots to identify major categories of toys (or RFID on the larger pieces) could make a lot of sense. I'm pretty sure a robot could learn to tell dolls from toy cars from lego.

What are you smoking, BT?

Another impractical, overly technical non-solution in search
of a problem.

If you're talking about robots to assist the elderly, a
house-cleaning robot won't be needed -- disabled elderly
people don't have messy houses. This is how it will be
done: all relevant belongings will be identified,
catalogued, and stored in a robot-accessible location,
and of course all this information will be loaded into the
robot. Using an appropriate interface, the user will make
requests of the robot such as "fetch X" or "store Y." It's
a simple, straight-forward application of existing automated
warehouse techniques to the home.

Although of course with your system a whole new class
of "the robot ate my homework" excuses is created.

Add new comment