Augmented Reality (AR)
Tim Cook, CEO of Apple, has repeatedly stated that AR is currently the most promising technology. According to him, augmented reality is an idea as grand as the creation of a smartphone.
The technologies of augmented reality are relatively new. The terminology has not taken its final shape yet. It is stated in detail in the article "Augmented, virtual and other realities". Wikipedia gives the following definitions.
Augmented reality (AR) is the result of introducing any sensory data into the field of perception in order to supplement information about the environment and improve information perception.
The term "augmented reality" was supposedly suggested by a researcher of Boeing Corporation Tom Caudell in 1990s.
There are several other definitions of augmented reality. In particular, in 1997, researcher Ronald Azuma defined it as a system that:
- Combines virtual reality and reality.
- Interacts in real time.
- Works in 3D.
In the concept of Paul Milgram and Fumio Kishino, augmented reality is part of mixed reality, which is also called hybrid reality.
This concept was proposed in 1994. But, starting in 2016, Microsoft began to actively use the term "mixed reality" to promote its HoloLens product on the market. And now some experts (and equipment suppliers) subdivide the terms as follows.
Augmented reality (AR) is projecting any digital information (images, video, text, graphics, etc.) over the screen of any devices. As a result, the real world is complemented by artificial elements and new information. It can be implemented using applications for ordinary smartphones and tablets, augmented reality glasses, stationary screens, projection devices and other technologies.
Mixed reality (MR) is projecting three-dimensional virtual objects or holograms on the physical space. It allows moving around a virtual object, inspecting it from all sides and, if necessary, inside. As a rule, it requires special equipment (glasses or helmets).1
It is these definitions that we will use in this article; it focuses mainly on augmented reality.
How does AR technology work
The general scheme for creating augmented reality in all cases is the following: the camera of the AR device captures the image of a real object; the software of the device identifies the resulting image, selects or calculates the visual addition corresponding to the image, combines the real image with its addition and displays the final image on the visualization device.
We will look at the technology of creating augmented reality in more detail using the example of using it to diagnose or control industrial equipment.
To work with AR in production, a smartphone, tablet or smart glasses with a video camera and appropriate software are used. If the camera lens is directed to an object (a piece of equipment), the software recognizes it either by a predetermined marker, or after analyzing the shape of the object.
Recognizing the object, the software connects to the three-dimensional digital twin of the object located on the enterprise server or in the cloud.
Then the AR device loads the necessary information and superimposes it on the image of the object. As a result, an employee of the enterprise sees partly physical reality, partly digital one on the screen (or through glasses). At the same time, the operator who manages this piece of equipment and the technician-repairman, looking at one object, will see different augmented reality in accordance with the functions performed. The repairman can see data on the operating time or, let us say, the operating temperature of a particular unit of the serviced equipment. The AR device operator can help control the object through a touch screen, voice, or gestures. When the employee moves, the size and orientation of the AR display is automatically adjusted, unnecessary information disappears, and new information appears.
A three-dimensional digital model is created either with the help of the CAD system (usually at the design stage of the object), or by digitizing this piece of equipment. This digital twin collects information about the state of the object, obtained from itself, from information systems and external sources. With its help, the augmented reality software scales and accurately places relevant data on or around the image of the object.2
An example of using AR in the IT-Enterprise SmartEAM solution at an INTERPIPE STEEL company
Devices implementing AR
Devices capable of creating augmented reality can be divided into the following groups.
Mobile devices. These include tablets, smartphones, glasses and, in perspective, augmented reality lenses.
The specialized software must be installed on tablets and smartphones. For example, augmented reality browsers, such as Wikitude, Layar, Blippar, or special applications (in particular, City Lens for Windows Phone) can be installed on smartphones and tablets. These browsers can show attractions, shops, cafes, rental centers, service points, etc., as well as other useful functions that are closest to the user's location.3
Augmented reality glasses are a separate full-fledged device developed directly for use with AR. For the most part, they are able to project holograms and information into real space, but do not tie them to physical objects. In fact, this is just a screen before your eyes. The most well-known glasses are Google Glass (in 2018, versions 2.0 and 3.0 were available for regular users; the version of 2017, Google Glass Enterprise Edition, was available for companies). Vuzix Blade, Epson Moverio, Sony SmartEyeglass compete with them. Compared to Google Glass, these and other augmented reality glasses are cheaper and more accessible. Ordinary users can buy them on official websites.
Microsoft HoloLens, Magic Leap One and Meta 2 glasses are already mixed reality glasses, i.e. they allow working with virtual objects tied to the real world.4
Lenses for augmented reality are still the technology of the future. Developers seek to transform lenses into a transparent screen containing a control system, a miniature camera, an antenna, LEDs, and other optoelectronic components. In particular, Samsung has already filed a patent application for "smart" contact lenses. Google is also working in this direction. However, such devices will enter the market no earlier than in 5-10 years.5
Stationary devices. It can be a TV, a computer screen, or a gaming computer like Kinect. An already augmented image is displayed on the TV screen (this is often the case during football and hockey matches). An example for a computer is a Google map in "Satellite" mode, when the names of streets and sights are superimposed on the satellite image. Sometimes large-format screens are used, as well as projection systems capable of superimposing images not only on screens, but also on any surfaces.
Special tools. These include, for example, specialized helmets of military pilots. Important information is given to the helmet's glass, and the pilot can perceive it without looking at the dashboard, thereby saving precious seconds. Many such systems allow realizing target detection by turning the head or moving the pilot's eyeballs. The helmet of the fifth-generation F-35 fighter pilot uses so advanced technology that the pilot can even see through the opaque body of the aircraft. It is the most expensive helmet in the world. Its cost exceeds $ 400,000. And British engineers have developed a helmet for military pilots with an already built-in night vision system.6
A smart motorcycle helmet can display motorcycle speed, route, text messages and more. A similar technology is also used to display information on the windshield of a car.7
Augmented reality in the industry
Over the past 20 years, Boeing has been looking for a system that can reduce the time it takes to produce cable harnesses and eliminate errors in their manufacture. Aircraft on-board systems contain many components that are interconnected by wires and cables. For example, their total length in a Boeing 747 aircraft is 250 kilometers.8 Laying and connecting the wires is done according to a special pattern, after which they are fastened into bundles, and connectors are installed on the ends of the cables. Such work takes a long time and is fraught with errors. In early 2014, the company introduced an augmented reality solution on the Google Glass glasses platform. Due to the introduction of AR technology, it was possible to reduce production time by 25% and reduce the number of errors by 50%.9
The Lockheed Martin company uses augmented reality technology when assembling an F-35 aircraft. As the main platform, Epson Moverio BT-200 AR-glasses are used, equipped with motion and depth sensors. When a technician mounts a brake component on the chassis, with glasses he sees all the data about where and in which order to assemble and connect the cables. According to NGRAIN, which implemented this system, the software allows engineers to work faster by 30% with an accuracy of 96%.10 (It is worth noting that Lockheed Martin also successfully uses VR technology; see the article "Virtual reality" for more information).
The Fiat Chrysler Automobiles concern (FCA) has applied the OPS Solutions projection AR-system in its work. Now, at each stage of the assembly process, workers receive visual information about their next step.
In 2015, the AGCO machine-building enterprise (USA) equipped the sites with large displays, which displayed the three-dimensional composition of products and a complete set of documentation required for fast and high-quality assembly of products (tractors and other agricultural equipment). In 2017, the company switched to using Google Glass glasses, thanks to which quality control was accelerated by 20%.
PVAITV and MibiPV portable virtual visualizers, designed specifically for engineers and IT-specialists, allow scanning equipment and detecting errors/breakdowns that need to be fixed. The program indicates where the damaged connector or plucked cable is.11
At a plant in Florida, while assembling wind turbine, General Electric workers contact experts through augmented reality glasses, show the assembled equipment in sight and get answers to questions from turbine experts using the same glasses. The analysis shows a 34% increase in productivity compared to previous equipment assembly technologies.12
In addition to the increasingly active use in industry, augmented reality is used in computer games, marketing (in particular, in street marketing, when the big screen with AR is exposed in a public place), in fashion, social networks, medicine and surgery, tourism, press, museum business – and the list of examples of the use of AR is constantly updated.13
Financial prospects (AR as J-technology)
According to the Digi-Capital consulting firm, in 2017, investors invested over $ 3 billion in VR and AR startups in the United States. As a result, the market significantly exceeded forecasts. However, at the same time, the VR/AR market is very unstable and is "warmed up" by one-time large deals, and it is mainly venture capital investors. At the same time, a significant amount of investor funds falls to the Magic Leap startup. For several years, the project of augmented reality super-secret glasses has received over $ 2 billion from Google, Alibaba, Qualcomm and Saudi funds. What is the reason for such generosity? The founder of the company, Roni Abowitz, after a series of studies, concluded that the brain did not use all the data that entered it through the visual system. Therefore, for the new technology, it was not possible to use the entire light field, but only selected bits of information that would be correctly interpreted by the visual cortex. Then, you could get rid of the displays and rely directly on the eyes of users. Therefore, investors knew what they were investing. Development lasted for 7 years, orders for the first samples for developers at the price of $ 2,295 began to be taken in August 2018.14
As for the market as a whole, it is also unstable so far. So, if in July 2017, IDC analysts believed that the global sales of goods and services related to augmented and virtual reality technologies (AR/VR) would increase from $ 11.4 billion in 2017 to almost 215 billion in 2021 and on average the market volume would grow by 113.2% annually,15 then in 2018 their valuations changed. According to new forecasts, from 2017 to 2022, the global market for technologies of augmented (AR) and virtual (VR) reality will grow by an average of 71.6% per year. The total is $ 106 billion in 2022, and not $ 215 billion in 2021.16
Digi-Capital analysts give slightly different estimates: by 2020, the VR market value will be $ 30 billion, and AR – $ 90 billion. However, Jesse Schell, CEO of Schell Games and professor at Carnegie Mellon University, believes that this is "a very wrong prediction". He believes that the market will develop very slowly. Therefore, start-ups should take into account that the harsh times will run over time for a longer period than planned. Shell believes that by 2025, the AR profit share will be 15% of the VR market revenue, that is, approximately $ 1.1– $ 3.3 billion. The augmented reality market will still be small, and difficulties await it. According to Shell, it will take many years before technologies develop enough to create augmented reality glasses that would not be different from ordinary glasses.17
However, Tim Cook, CEO of Apple, has repeatedly stated that AR is currently the most promising technology. According to him, augmented reality is an idea as grand as the creation of a smartphone. Periodically, information about Apple’s work on augmented reality glasses appears on the network, the estimated release date is the end of 2019.18
Pavel Bilenko, founder of the TEKNER engineering center, believes that AR is one of the J-technologies with accelerated diffusion. The rate of technology diffusion is the time it takes for technology to become actively used by most consumers. Over the past 110 years, the diffusion rate of consumer technologies has grown rapidly, and now the development curve of some of them is shaped like the letter J, i.e. for a few years after birth, technology becomes a key competitive advantage of the company.
Another important fact, speaking about the unconditional prospects of AR – the creation of the Augmented Reality for Enterprise Alliance (AREA) in 2015. This alliance includes such large companies as Bosch and Boeing. The goal of the Alliance is free (for the US market) and open exchange of best practices, lessons and technological resources that will help enterprises effectively implement AR.19 On April 11, 2017, the participants of this alliance have developed key industry guidance documents. The documents were developed with the assistance of UI Labs, Lockheed Martin, Caterpillar and Procter & Gamble.