Industry – Augmented Reality

Pipes and conveyer belts that talk? Valves and bolts with Internet addresses? Factories that harvest maintenance information on each and every part to build expert knowledge for on-the-spot repairs? It's all possible thanks to a developing technology called Speech-Enabled Augmented Reality.

A new technology called Speech-Enabled Augmented Reality (SEAR) developed by Siemens Corporate Research (SCR), in Princeton, New Jersey, holds the potential for a radical improvement of this entire process. Based on a system now being tested at SCR, here's how SEAR would work in an industrial installation: A maintenance engineer working in a remote section of a chemical processing facility receives a message on his PDA, or on a wearable or mobile computer that a pump in section X is malfunctioning. The engineer's movements are tracked by combining inputs from infrared beacons located in each area of the facility and a three degree-of-freedom inertia tracker in his Compaq iPAQ Pocket PC. About twice a second each beacon transmits a unique ID, which is detected by the PDA's IR port. This causes a VRML (Virtual Reality Modeling Language) browser to produce a VRML model (a realistic-looking 3D image) of the corresponding scenery file and viewpoint on the PDA's screen. The engineer thus sees where he is standing in regard to his surroundings and an arrow indicates where he should go next.

Since the PDA has access to the facility's geographically referenced and digitized database—the most important and complex part of installing SEAR—the engineer's target object is known to the system. As a result, the PDA can use written or voice commands to direct the engineer to that object. Once in the area of the object, visual markers provide more precise location information.

Each marker has a unique combination of dots, making it look like two sets of tic-tac-toe boards. Held at the same angle as the engineer's head, a camera in the PDA orients the mobile device based on any markers in its vicinity. "This solution allows us to calculate the exact position and orientation of the user," explains SCR's Nassir Navab, Ph.D., who led the development of the location detection technology. "Even if the object in question does not have a marker, the system can still identify it because it knows the user's location and orientation and is constantly comparing it to a 3D model of the facility."

Once the problem object is located, "the system starts downloading the associated information into the cache memory of the engineer's mobile computer. The information appears on his mobile automatically, including the kinds of questions he can ask," says Navab, adding, "If, for instance, he is standing in front of a pump, it knows which pump and displays questions about pressure, contents, temperature and maintenance history." Depending on the underlying database, there is no limit to the number of questions that can be asked.

3D Voices: Once a question such as "What is your current pressure?" is asked, "SEAR uses Siemens' Very Smart Recognizer voice recognition engine to process the information. The resulting data passes through a wireless access point to a database server. The server searches its database for the appropriate information, which is transmitted to the PDA. The PDA then generates a 3D voice into the user's headset that sounds as if it is coming from the object in question," says SCR's Dr. Stuart Goose, who developed the patented voice control system. "The ability to ask questions verbally and to receive spoken responses is important in situations in which the user needs to have both hands available," explains Goose.

Does SEAR technology require a major overhaul of a facility's information infrastructure? Not necessarily. Many facilities already operate on SAP software that's based on object inventory numbers and associated databases, and some power plants have 3D models that are connected to such a database. Furthermore, many factories already run on WinCC, a Siemens automation program that controls programmable logic controllers. "The thing is," says Navab, "that today this information is shown in an abstract version on monitors. But the point is that the information exists, and that in such facilities, every object is already in the database. So in order to be efficient, our system has to talk to the WinCC and SAP systems."

And once that "talking" gets started, a lot can change, particularly in terms of maintenance and repair scenarios. Not only is each and every object in a SEAR facility in the database, but each could also have its own Internet address. As a result, information regarding each part could be harvested and compared with information from identical or competing parts in similar facilities. Maintenance and repair data could be compared and expert systems developed based on the resulting data. "By the time SEAR becomes commercially available in four or five years," says Navab, "it may be possible for an engineer confronted with a difficult problem on the factory floor to interrogate the part's maintenance history, send a multimedia e-mail to the person who most recently serviced it, ask an expert system for an opinion and, based on the responses, click the part's Web address and order a replacement. It can all happen on the spot with zip lost time."

The applications for such a system are virtually limitless. They range from asking your PDA to direct you to the nearest flower shop on your way to a date, to complex and as yet unexplored scenarios in which multiple users, such as firefighters or security personnel, visualize each other's movements as they move toward a common target. "This is a generic technology," says Navab. "All we can say now is that it is very likely to make many processes much more efficient."

Arthur F. Pease

ARVIKA: Putting Data in the Picture