The Economist: Tomorrow’s soldiers will have their reality augmented

The Economist: Tomorrow’s soldiers will have their reality augmented

Success or failure in war often hinges on how much soldiers know about the enemy and the areas in which it operates. Tactical intelligence of all sorts helps. Locations of culverts where bombs may lie hidden. Spots from which snipers have scored kills. Water sources likely to have been polluted by agricultural runoff after heavy rain. Identities of locals suspected of aiding insurgents. Armed forces compile such intelligence and store it on computers. But making full use of it in the heat of battle has never been easy. This is now poised to change, thanks to display technology known as augmented reality (AR).

AR is the art of superimposing computer graphics on a view of the real world. It is popular in applications ranging from video games to selling furniture. America’s army would like tactical intelligence pertinent to a soldier’s mission to pop up similarly on a transparent visor attached to his helmet, no hands required. And for this capability, it is spending big. In March it announced a deal with Microsoft to build such a system. This could, over the course of a decade, cost a staggering $21.9bn.

The army has dubbed the kit IVAS (Integrated Visual Augmentation System). David Marra, who runs Microsoft’s end of the project, describes it as a holographic computer. The displays produced, he says, appear “locked to the real world”, even as a wearer moves and shifts gaze.
IVAS pulls off this wizardry by crunching and synthesising several types of data. A gps receiver locates the wearer within centimetres. Instruments fitted with accelerometers and gyroscopes provide information on how he is moving around. Cameras track eye movements. IVAS must also be aware of a soldier’s environment. This relies on lidar, an optical equivalent of radar. An array of sensors record the time it takes infrared laser pulses bounced off nearby objects to return. That allows those objects’ distances to be calculated. Machine-vision software that recognises those objects then keeps track of how they move. Mr Marra describes the process as a “continuous rendering of the xyz co-ordinates of everything”.

IVAS must calculate with extraordinary speed where on a headset’s visor to display graphics. A latency of just seven milliseconds risks causing vestibular ocular discomfort, a type of dizziness that has long plagued the development of realistic displays of augmented and virtual reality. In most circumstances, Mr Marra says, IVAS operates well within that limit.

Theatre of war

To build the system, Microsoft has modified an AR headset called HoloLens that it has so far sold mostly to businesses and research outfits. The militarised version of this has been “ruggedised” and souped up with a computing and battery “puck”, a bit bigger than a smartphone, that the user carries on his chest.

Tactical intelligence can be uploaded before an operation, with updates transmitted wirelessly as needed. AR text and graphics guide soldiers through unfamiliar terrain, highlight the whereabouts of friendly forces and mark the enemy’s known and suspected positions. The headsets will also employ facial-recognition technology to append information on possible persons of interest who come into view. As Susan Fung, the army’s deputy head of IVAS technology at Fort Belvoir, in Virginia, puts it, soldiers freed of the need to look down at a screen will be able to “focus on moving and engaging targets”.

IVAS will also exchange data with Azure, Microsoft’s computing cloud. This will permit additional features, such as language interpretation, to be included. Production of the headsets, which weigh about a kilogram, has begun. The first of an expected 120,000 or so units are to be deployed this year.

Others besides the army are also interested. America’s marine corps is a partner in the IVAS programme. Undisclosed allies are seeking to join. And modified HoloLenses may also see use on warships. Britain’s Royal Navy has paid $25.5m to bae Systems, a local defence giant, to adapt them to show pictures currently displayed on screens on the bridge to officers elsewhere on a vessel.

A sight to behold

This is heady stuff. Even so, enhancing combat operations with AR will remain, for some time, beyond all but the most technologically sophisticated armies. Marcel Baltzer, of the Fraunhofer Institute’s campus in Wachtberg, who co-chairs an AR-research team for Germany’s armed forces, believes that even the European armies most advanced in the art (which are, by his reckoning, those of Britain, Germany, Lithuania, the Netherlands and Norway) will need another decade. Using AR for training, and for the maintenance and design of military hardware, he adds, is easier and will become common sooner.