Building the military metaverse

The US Army’s synthetic training environment (STE) might deliver the first true iteration of what recent media buzz has labelled the “metaverse”. While companies such as Meta have appropriated the term “metaverse” for marketing purposes, military organisations are using the same or similar underlying web and VR technologies to address very specific needs. This means that they might deliver on the promise of their version of the metaverse well ahead of the commercial sector.

Definitions of the metaverse are evolving, but I envision it as a persistent, online and immersive virtual world supporting thousands of simultaneous players and potentially millions of artificially intelligent entities.

Military use cases for the metaverse include large-scale training and mission rehearsal. It’s not abstract and distant like something from the movies – the STE is a tangible solution already under development. It represents a step-change from a traditional federated set of local simulations to a single cloud-based solution with a high-fidelity and continuously updated digital twin of planet Earth.

The US Army states that the STE provides a “collective, multi-echelon training and mission rehearsal capability for the operational, institutional and self-development training domains”. To put it from a gaming perspective, it essentially wants less Counter-Strike and more Eve Online, but an Eve Online to support both first person (“soldier view”) and third person (“commander view”). They definitely want to transition from hundreds to thousands of players (trainees) to achieve the vision of STE, which will mark a definite step-change from the current approach.

Currently, simulation centres support specific organisational units, mostly independently. They might train a few hundred soldiers at a time, or train a brigade staff, but it’s rare for all these different users to train simultaneously in the same scenario. And while different simulation centres might use the same simulation software and aim to meet the same doctrinal training objectives, the actual training scenarios can vary widely, especially at the lower echelons.

Constructive simulations with AI-controlled entities and virtual simulations with human-controlled entities are currently integrated using common interoperability standards. Still, the in-game result varies widely. It is often apparent which entities are simulated in the tactical training game and which are in the constructive simulation, which can negatively affect training. It is difficult for a trainee to hit an AI-controlled tank that is “rubber-banding” due to its position being updated every five seconds on the separate constructive simulation.

Terrain is another limiting factor with the current approach. Terrains for simulation are typically developed centrally and provided to users for download. Each simulation system has its own terrain format that rarely correlates perfectly. Terrain updates can take weeks to months and many simulation systems cannot even work with the high-fidelity 3D data that is readily available from companies such as Maxar.

How STE will fix the problems

STE will address these issues through two significant efforts: converging constructive and virtual simulation through simulation services running on an appropriate web architecture; and providing a digital twin of Earth that is constantly updated using the latest 3D data sourced from satellites, UAVs and other similar sources.

Ultimately, every corner of the Earth will be available for training based on the latest 3D data. Management of scenarios and parametric data will be centralised, ensuring that all units train the same way, data on human performance is effectively collected and machine learning is used to better inform AI and training.

Cloud scalability will enable simulations of entire megacities and represent second and third-order effects. For example, consider an attack on a major city and the effects on traffic flows and, in turn, the difficulty of moving forces through that city. All of this can be simulated, training the command staff to the actual human operators driving the vehicles or operating the avatars in that highly complex environment. For the first time, real-world scale will be possible in a converged constructive and virtual simulation.

Militaries across the world have shifted their focus toward training for large-scale, multi-domain operations. While there will remain a place for siloed training games and simulations to train specific tasks, realistic collective training requires an accurate representation of the broader battlespace. For the first time, the individual members of entire Brigades will be able to “play out” large-scale manoeuvres that would be cost-prohibitive in the real world. In a simulation, they can conduct their training anywhere on Earth. Modern military procedures involve participation from multiple, vast domains – land, air, sea, space and cyberspace – and this also drives demand for the next generation of scalable simulation.

Why the military metaverse matters

Unlike commercial metaverses, mission rehearsal in the military metaverse will demand the most accurate digital twin of the Earth ever created.

My first experience with mission rehearsal was in 2005, working with a simulation centre in Australia to deliver meaningful training to soldiers. Preparing for this event in 2005 was similar to the same process today: identify the training needs, identify the simulation software best suited to meet these needs, build out the terrain, build out the scenarios and then deliver the training. While we certainly delivered effective training (and the Australian Army went on to make game-based simulation a significant part of its training capability), it took many months to prepare for the event. Most of this time was spent building terrain. I find it disappointing that almost 20 years later, it still takes too long to acquire terrain data and build terrain for military simulation. The same siloed approach with many different simulations requiring different terrain data is still the norm worldwide.

And this is why initiatives such as STE are so exciting – finally, simulation systems will be able to access suitable terrain on demand.

It’s easy to forget what it was like to use paper maps to navigate a city or a forest. While the commercial space has readily embraced streaming map data from the cloud, the military simulation industry has been slow to adopt the same approach.

This is not without good reason – military simulations are tremendously complex. Terrain needs to be heavily attributed for many reasons. Artificially intelligent forces must navigate on foot or in vehicles, so the terrain must include pathing data. Vegetation must provide cover and concealment and should be configured to destruct if run over by a heavy vehicle or impacted by an explosion. Buildings need to be enterable and complex urban features such as subways should be represented.

Gamers will undoubtedly be familiar with this level of terrain detail in games such as Grand Theft Auto V. That had a budget of $265 million and took years to make. But the military also needs all of the same level of detail but matching the real world exactly and created automatically, not by hand.

Initiatives such as STE matter to geospatial professionals because military simulation drives the requirements for attributing high-fidelity 3D terrain data. 3D terrain data is derived from photogrammetry and Lidar, typically from sensors on satellites or UAVs. The result is stunningly realistic but while the raw 3D data might be suitable for flight simulation, it is not suitable for ground training, where a much higher level of fidelity is required.

A best-of-both-worlds approach is needed to attribute the 3D data and replace individual features with high-fidelity alternatives that still correlate perfectly with the source 3D data. This is no easy task, but it’s vital to ensure that the resulting terrain can support training in multi-domain operations, especially mission rehearsal.

The process for building terrain for the military metaverse must be automatic, highly customisable and support regional rulesets. Specific to the military metaverse, it will be critical to ensure that terrain is suitable for the full range of military simulations and image generators that might need it. This is a vital distinction from the gaming and commercial space, where the terrain is typically rendered in a single engine (eg Cesium or Unreal). The military metaverse and the broader metaverse must stream high-fidelity terrain to runtime engines in many formats, all of which must be perfectly correlated. The military will continue to use many different simulation hosts and AI models, and these other simulation systems require different terrain data. A constructive simulation may only need terrain elevation, building footprints and roads, but a 3D training game will need 3D models of every object in the scene, down to blades of grass.

New platforms such as Mantle Enterprise Terrain Management will be central to the success of the military metaverse. This provides an automated and end-to-end solution for ingesting, conflating, streaming and exporting high-fidelity 3D terrain data to many runtime engines. Developers have made progress on the automation of terrain attribution, with companies such as Blackshark.ai now offering realistic 3D terrains based upon little more than high-resolution satellite imagery. Companies such as Maxar lead the way with data collection and are central to relevant efforts such as One World Terrain.

As long as media interest remains sceptical, those discussing metaverse technology may generate eye-rolls and dismissals. Yet, simulations were designed to digitise the environment, enhance learning and accelerate training transfer in real life – and the metaverse represents an essential step toward this goal. Geospatial professionals must monitor military metaverse progress since it is at the forefront of industry transformation.

Pete Morrison is co-founder and chief product officer at BISim (www.bisimulations.com)