Reality, Only Better

Reality, Only Better

Computing: Superimposing computer graphics on the real world, instead of displaying them on screens, has many potential uses

FIRST, catch your cockroach. The clinical psychology department at Universitat Jaume I in Castelló, Spain, paid its cleaners to capture live specimens. A team of computer-imaging specialists filmed the creatures, digitised images of their scurrying and teeming, and displayed the images—not on a computer monitor, but on see-through goggles. To the wearer, the virtual roaches then look as though they are really in the room. Next, university psychologists set about therapeutically frightening patients who have a fear of insects. “They put a foot on the ground and the cockroaches start climbing over it,” says Cristina Botella, who led the researchers. “The computer just pumps them out.” In November Ms Botella presented her team's findings at the Association for Behavioural and Cognitive Therapies conference in Philadelphia. The treatment worked very well.

For some things, it turns out, computer graphics can be much more effective when viewed not on screens, but superimposed on the real world. The technique is known as “augmented reality” (AR) or, less frequently, as “augmented vision”, because the real world is augmented with virtual text or graphics. Much AR technology remains in labs, but research funding in both the private and public sectors is increasing, and all kinds of eclectic and ingenious applications are emerging in fields as diverse as medicine, warfare, manufacturing and entertainment.

Consider the task of locating veins, a crucial step in procedures such as inserting intravenous drips, injecting medicines and drawing blood. Last year Luminetx, a medical-equipment firm in Memphis, Tennessee, began selling an AR machine called the VeinViewer. It shines near-infra-red light at the patient's skin, and because blood vessels absorb such light, a digital video camera that captures the reflected light can work out the precise location of veins to a depth of almost 1cm. A projector then shines a map of the vein network directly on the skin. The process takes place in real time, so the luminous map changes as the patient moves. “It's like Superman vision—you can see under the skin,” says Kasuo Miyake of Clínica Miyake, a clinic that performs vein-related procedures in São Paulo, Brazil. Dr Miyake says his VeinViewer boosted referrals by 20%, cut costs by 30% and reduced the need for anaesthesia.

AR can also be used in surgery. A big benefit is that surgeons do not need to keep looking up and down, switching their gaze between the patient and displays on nearby equipment; instead, the information can be directly overlaid, in effect giving the surgeon X-ray vision and other superhuman powers. Even so, there are very few AR operating rooms, says Henry Fuchs, an expert in AR medicine at the University of North Carolina at Chapel Hill. Dr Fuchs says that although early (and sparse) evidence suggests that AR surgery is more accurate, uptake is slow for several reasons. The equipment is expensive, training is time-consuming and involves ditching some skills surgeons have worked hard to attain, and new surgical techniques are more vulnerable to malpractice lawsuits.

Augmented enjoyment
The technology also has less serious uses, however. YDreams, a marketing and digital-media firm in Lisbon, Portugal, has developed an AR sightseeing viewer called VSS. The first such machine, bolted atop a battlement on the 12th-century Pinhel Castle in north-eastern Portugal, delights tourists who tilt it up, down and around for an augmented view of the castle and its surroundings. Place names and explanatory text are superimposed over objects seen through the viewer's screen, and animated graphics show how some structures were built or destroyed. The number of visitors has doubled since the viewer was installed in July 2006, says Isabel Almeida, who manages the castle.

“Head-mounted displays could show soldiers the locations of friendly forces, as in a video game.”In France the Parc du Futuroscope, an amusement park near Poitiers, is building a €7m ($10m) safari attraction that will be devoid of animals. Instead, passengers riding on a small train will look through hand-held AR binoculars that will superimpose frolicking 3-D virtual animals over the real decor. “It's pretty close to being a magic show,” says Bruno Uzzan, the boss of Total Immersion, the company that is developing the attraction. Video games would also seem a perfect fit for AR technology, and many game studios are investing in development. But very little AR gaming technology has hit the market. Gamers demand extremely fast and rich graphics, and so far the hardware is too expensive to make AR a mass-market proposition. If the future is any guide, however, prices will fall once AR is adopted by more serious users—such as the armed forces.

Virtual reality has proved to be immensely useful in military training. But virtual worlds—those created entirely by computer, and viewed on screens—cannot be used to improve live-fire training, long a priority of America's Marine Corps, says Bob Armstrong, until recently deputy director of the Marine Corps' Training and Education Technology Division. “We end up shooting at piles of tyres or old vehicles,” he says. “We wanted to inject a thinking, moving target into the live-fire environment.” His team, working closely with organisations including the Office of Naval Research and various defence contractors, managed to do just that using AR technology.

During battle, forward observers identify targets and direct artillery, mortar and aviation fire. With the Marine Corps training system, forward observers wear a head-mounted display with a see-through visor; objects displayed on the visor appear to be part of the real world. Mr Armstrong, now director of technology at the Virginia Modelling, Analysis and Simulation Centre at Old Dominion University in Norfolk, Virginia, says the Marine Corps' AR live-fire training system works exceptionally well. Instructors use tablet PCs to move the virtual targets while trainees shoot at them, and the virtual targets can even hide behind real terrain or buildings.

On the battlefield, AR could have an important role in disseminating tactical intelligence. Soldiers with head-mounted displays might, for example, read street names superimposed on the ground, follow colour-coded arrows for patrols or retreats, and see symbols indicating known or potential sniper nests, weapons caches and hiding places for booby-traps. The displays could also show the locations of friendly forces and levels of ammunition and other supplies, as in a video game.

Mark Livingston, head AR researcher at the Naval Research Laboratory in Washington, DC, says his team is developing “3-D ink” writing methods that will allow soldiers to paint virtual symbols or text onto the real world, so that other soldiers who arrive at the same spot later can see them. He remarks, only half jokingly, that young soldiers who are used to video games are better equipped to handle this visual “information overload”.

Augmenting productivity
Industrial engineers call them discrepancies, deviations, clashes or conflicts. Variations between a structure's digital architectural models—the virtual renderings produced by computer-assisted design (CAD) software—and the structure itself are common. Finding and mapping them is important, because accurate CAD models are needed to operate, maintain, repair and insure buildings. And some discrepancies require rebuilding, so the sooner they are found the better. But current checking methods, generally involving laser scanning, are expensive and require lengthy set-up. A new method, using AR, is on the way.

Siemens, working with the Technical University of Munich, has prototyped AR discrepancy-checking software for industrial plants. Engineers superimpose the original CAD models over actual buildings to determine which bits of the model need to be updated, or which parts of the building need to be rebuilt. The process is less complicated than laser scanning, so discrepancy checks can be more frequent, making rebuilding less expensive. Mirko Appel, an engineer and senior project manager at Siemens, estimates that the software will reduce the cost of constructing a typical medium-sized coal-fired power plant by more than $1m. Areva, a French nuclear giant, used the system to check a European plant for discrepancies in September. And OMV, an Austrian energy company, has tested a similar system developed by the Upper Austria University of Applied Sciences.

Volkswagen (VW) uses AR discrepancy-checking software to verify the conformity of components supplied by subcontractors. Christoph Kohnen, a spokesman, says the technique is “tremendously faster and cheaper” than previous measuring methods. The carmaker, which has teamed up with Kuka, a robot-maker, to improve the system, is introducing the technology in some 40 factories worldwide. And it has devised other uses for AR. To study crash tests, VW superimposes an image of an intact car over the wreckage of a crashed vehicle. To speed up prototype construction, it uses AR to superimpose luminous instructions directly on the tools and prototype components in front of workers. AR can also help to design production lines. Metaio, an AR developer based in Munich, does brisk business applying AR to “interfering-edge analysis”. Its systems move virtual models of prototype machinery and products around manufacturing plants to determine how existing equipment would have to be moved or modified.

AR is still an immature field compared with virtual reality, which has now entered the mainstream in the form of video games, online virtual worlds and computer-animated films and special effects. The additional technologies required to take virtual images and integrate them into the real world still have a long way to go: most AR technology is still expensive, fragile and unwieldy, though researchers are doing their best to change that. But given a few more years, it is not hard to imagine where all this might lead: imagine satellite-navigation systems that appear to paint the road yellow to show a driver which way to go, mirrors that let you try on different outfits or haircuts, or glasses that turn the whole world into a backdrop for a video game. Why settle for reality when you can augment it?