Novel Interfaces for Tele-Operated Robotics

I've been thinking about the various ways to control a teleoperated robot. Autonomous robots can be given task level commands: "clean the floor", "sweep this building".

But teleoperation implies a lower level of user control, making it a harder problem. Live video feeds stream over often unreliable networks, while the operator tries to control the robot.

High level systems haven't yet been deployed. Direct a robot to a particular by clicking on a map, and command it to grasp an object by clicking on it in the video stream. The mapping, obstacle avoidance, navigation, and manipulation are all essentially automatic.

iRobot has a research project called Sentinel. It uses sensors that aren't on any deployed systems, and is able to receive such commands. The controller is a tablet with a map-based interface that can be used to control multiple robots. The operator can still drill down and control each robot, but with the autonomy in the robots, why would they want to. This is the future though. Right now, multiple people are needed to control a single robot. That ratio needs to be inversed and Sentinel can do it.




Deployed systems are unfortunately dumber and thus harder to control.

Take the PackBot EOD. It has a 3 link arm with over 10 degrees of freedom. You can't control this adroitly with a simple controller. The solution is a set of "pucks" which are high degree of freedom joysticks. They can be pushed forward & back, side to side, lifted, pushed down, and twisted. Different user modes allow this to control the chassis treads, chassis flippers, the arm, and the manipulator.

You can see a video overview of the PackBot EOD here:



Interestingly, this allows for experts to complete amazing feats, while novice users complain about the complexity. Embedded video training has been one suggested solution.

Soon, ground robots are going to have a controller that looks very familiar:


Kids that join the armed forces already know how to use it -- they've applied it to many very complicated games. This is a good direction, especially considering the chaos of many vendors providing many different controllers.


AnyBots is working on a humanoid robot. They intend it to replace human laborers, with remote workers providing the brains. Watch this video, which shows a panoply of screens used by the operator. Note the use of accelerometers and multiple joysticks.



They will need to add autonomy to make grasping tasks easier. I also worry about the lag in the video. You need enough pixels to convey the environment, but you also need to present the scene and relay back commands fast enough to avoid faults.


The Nintendo Wii is a new gaming platform that uses gyros to note sweeping motions. The applications to robotics are pretty obvious. I've seen it used to control a PackBot EOD arm with impressive results. Here is a video from US Mechatronics, where they're playing Riil Wii Tenniis.




Finally a note about UAVs. UAVs are easier to control than ground robots. There are very few obstacles, no manipulation, and localization is easy. Still, teleoperation can be difficult. Task level commands, such as "patrol on this GPS waypoint circuit", are often used. Automated landing and takeoff are common, even on aircraft carriers. One lacking component is situational awareness. Luckily, the operators can be in Missouri while bombing Afghanistan, so comfy chairs and multiple monitors abound. Here is a view of Raytheon's Universal Control System.



But can it play Doom?

This just in

Smoking is bad for a robot's health. So says this anti-smoking Star Wars PSA.

In far, far away galaxy, robots program YOUR thoughts.

Robotics Startups

I found a link to Willow Garage, a robotics startup.

Here are a few other startups that come to mind.

Sebastian Thrun's stealth startup Vutool, in SF area.

Aaron Edsinger's startup in SF, Meka Robotics. He made Domo.

Kiva Systems in Boston. They are going to explode, and you won't even see their robots. Better warehouse access is huge.

Q Robotics, also in stealth mode in Boston. Some very smart people there.

Anybots looks good, but I don't know their plan.

I'll mention US Mechatronics if only in hope that they go back to their sentry gun roots.

Also, note a pattern on the locations. Am I just biased towards New England and the San Francisco area? Probably.

Box Clustering

This video is of 36 Create robots pushing around some boxes. I love the emergent behavior of the very neat clustering.

Robot Tribute to Steve Irwin

Here we have the two most popular bomb disposal robots, the PackBot and the Talon, humping each other, with a mock nature documentary voice over in classic Steve Irwin style.

Note which robot is bitch :)

Local Ads in Street Views and Map Search (Day #3, Idea Week)

Google street views and MS Live Local are interesting platforms, but the content to make these map views meaningful and useful just isn't yet online. Searches degrade into a list of store fronts, with no indication of quality. It's 1997 all over again.

When I walk down the street, I pass lots of businesses. Those with store fronts such as cafes, pizza joints, and clothing stores are the kinds of businesses which I may be interested in while walking in the area, while others - mostly without storefronts, such as lawyer's or accountant's offices - are services which I probably wouldn't be interested in while on a casual stroll. These are the types of things in an area I'd first check out while online. However, it's not just dentists I'd like to read about online first. When I'm hungry I don't always want to take a stroll around town to find something to eat. I'd like to be able to go online and see what's available in my area before leaving the house.

Stores should plaster their Street View storefronts with ads and coupons.

Currently, google does have some coupon support. Look at this search for Pizza in Arlington.

Live Local is a bit better, bringing ads from Superpages.

But this whole system could be much better. Most small stores don't have the knowledge or resources to fully utilize the internet. For example, note that in the google link above, only Domino's Pizza has a coupon. Only the most corporate (and debatable worst) pizza knows enough to get online.

It isn't just about making money from ads or making more business. Right now, street views are interesting, but useless. There are large network effects where local search is worth much less without good information coverage. There is no data contained in a view like this.

That Street View is pointing straight at a Sushi restaurant, right after I did a search for "Sushi, SF CA", but there is no markup on the image at all. I should be able to see a picture menu, specials, and photos of the decor.

A company could be built whose sole purpose was to call these places or knock on their doors, explain to them the potential of advertising online, and help them get started.

One way to think of an idea is to imagine the future and build it now. In the future, you'll be able to take an image of a location using a cell-phone camera, and ask a service to send you the additional digital data on that location. As long as the local stores have something on the internet, GPS and image analysis can get that data and pass it to the customer.

The technical part is allowing for overlay of information on maps and street views. The harder part, that requires smart business and real employees, is going to these businesses and getting them to put the information online.

Server-side video processing of life-casting robots (Day #2, idea week)

iRobot just announced a new robot, the ConnectR. It is a roomba chassis with two cameras and a wifi connection that lets it stream video. Users control a robot remotely, and watch a video stream with 2-way audio.



I love iRobot's history of openness. The roomba API is completely open: anyone with a serial cable and some know-how can control it. The ConnectR should eventually be open as well.

I'd like to see this taken to the next level with live-casting from a robot. Justin.tv meets ConnectR.

The robot would either have to be managed by a human when in public, or it could wander alone in a private space. Community voting could control where the robot goes and whhere it directs its gaze. Ideally, the robot would be placed where average people can't be, like back stage at a concert or fashion show. Dangerous places are also ideal. Let the robot go where the humans shouldn't.


Robots with internet enabled cameras can do more than normal robots. Server-side processing means the robot doesn't need to have expensive hardware on board for the intelligence.

Obstacle avoidance, mapping, localization, face & pedestrian detection, object detection, object tracking & motion modeling, etc. can all be done using today's technology with a single camera stream. Some processing would need to be local given the constraints of network bandwidth, but plenty could be offloaded. It is also a good model for premium services: pay more to get faster connections, fast processing, and more capabilities.

Automated surveillance with alerts for intruders could be a killer-app for wifi/webcam robots. Today's choices for home security are immobile, cost thousands, or both.


Server side image processing itself is a viable idea. It's the next step in online video. Today, we just stream compressed pixels. Tomorrow, we'll calculate and stream information about the scene. You can send a flickr image stream to a service which finds faces, builds a corpus of data to identify the people from context, and performs face recognition. Face recognition companies like Animetrics could be tapped to do the hard part.

Motion detection for surveillance applications could all be online, with companies like Intellivid and ObjectVideo already having optimized the image processing component.

Street Views in maps can be combined with GPS tagged digital camera shots to build super high resolution aerial imagery, and eventually 3D.

Services like Fauxto which aim to be photo-shop online could build an interesting api where any image can be sent with instructions for processing.

The point is that all of these services require sending video and images to a server, where some intelligent processing occurs. Often times, the processing will involve the same software modules, so each problem is not unique. Tap the long tail of software development and allow 3rd parties to build their own processing streams that live on your servers. This could be made simple and standard using tools like Python with modules for image processing like OpenCV, PIL, NumPy, and SciPy.

More armed robots

Foster Miller has upgraded their armed robots. These are sure to make lots of people fear a robot uprising and at least a few to honestly say, "I, for one, welcome our new robotic overlords."

Wired's DangerRoom takes a closer look at the new models, along with this short video.



We really, really need to get reliable target tracking before I'd trust tele-operated firing of this system. The lag of remote control and the difficulty of communications mean that the operator should be focused on policy and not low-level motion control of a remote pan-tilt mount.