There are two distinctive visual technologies in the works – Virtual Reality which is completely immersive and AR, Augmented Reality that tries to match the best of real world and add some virtual elements. Now, Tim Leland a Vice president of Product management, who is responsible for Adreno GPU part has shared his cunning plan for Augmented and eXtended Reality.

It seems that after Pokemon Go explosion, the industry realised that there is  a place for both the Virtual Reality and Augmented reality approach.

Augmented reality is versatile and can address many different markets that include kids gaming and entertainment, learning and exploration, family or business communication, professional communication and industry applications.

eXtended and Augmented reality

Kids can chase the virtual characters in their homes and fields and into the busy roadways or down mine shafts seeing them via wearable glasses or in Pokémon’s case via phone’s display. Young adults could be exploring different cities, such as Rome, and the AR could add the missing parts of the colosseum, add some additional visual graphics and information over the actual objects.

Families in the future might be virtually sitting next to each other with the power of Augmented Reality. Qualcomm beleives that Augmented and Virtual will merge to XR, eXtended reality. This could be as game changing for the future of communication as video chat are that started with Skype.  

Professionals can work on projects. For example architects could build a model even though teams might be sitting in different buildings or countries.

Fitness enthusiast could have a virtual trainer in front of them commending them how to get rid of the unnecessary weight. We would like to add general gaming as Pokemon Go was proof that it can be done, and we dobelieve that there will be more similar games that will mix reality with augmented reality. These gamers can be anything from action games, puzzle strategies to sports gamers.

Early days of XR

Leland has pointed out that Augmented and eXtended Reality is in its infancy and, like smartphones, it will take a few years for get things better. You can expect that XR is going to get better with every SoC and the Snapdragon 835 is a good example. There is more GPU power in Snapdragon 835 and at the same time SoC will need less power for the same GPU / AR related tasks. This is of course a combination of a next generation Adreno and a 10nm, among some new transistors spent to make VR and AR better. The next Snapdragon and Adreno will be even better the Snapdragon 820 was designed with AR / XR and VR in mind. 

As a glimpse to the future Tim mentioned a first responder glasses. The way Qualcomm sees this product is a synergy of lot of cameras, sensors, speakers and microphones is a stylish design. AR glasses would come with a new optics and projection technologies with a durable, semitransparent AR displays. These glasses can track users eyes and would come with ambient light sensor as well as optoelectric night vision and thermal imaging sensors. Think Predator movie.

These glasses should come with passive and active cameras at the front with fish eye and telephoto lenses as well as rear camera. First responders would need to be aware of their whole surroundings.

Qualcomm being a modem company would add a 4G, 5G or similar connectivity too as the responders will want to  transmit and receive video and some other useful data. Tim mentioned bone conduction transducers, inertial, haptic, environmental and health sensors. The glasses could even come with ultra-bright LED lights that would eliminate the needs for external flashlights.

The glasses would come with a set of high sensitivity audio microphones and tracking and recording cameras. One would also get pair of directional speakers as a part of it. If you managed to get to this point, we know that you would be interested in owning such a device. It just sounds too cool as it can give you the Predator vision, record stuff and sense more than you need.

I am not sure how would you power them, but since we are still some time away from having such a device, this is something that they will work on. Of course the SoCs in the future will pack more performance per miliWat and there will be some improvements on the battery side too. These glasses could teather with phones that will rapidly continue to be faster and needing less of an energy.

Another application of eXtended Reality would include techniques of properly lighting virtual objects realistically and dynamically. The goal is to implement proport XR environment processing. Virtual objects look real and can be correctly placed in the scene. Dynamic lightning would be correct for the environment and solid objects would look solid. Materials can be rendered physically correct making the overall interactivity smooth.

Intelligent and faster interaction between many different sensors and rendering systems will make this possible. New computer vision and global illumination algorithms will dynamically render and overlay realistic AR objects. Correct lightning considers the position, intensity and orientation of all light sources.

Machine learning should solve some of the problems . Visual intelligence will help to see and understand the world around you. One can use computer vision and machine learning to understand the objects in the real and virtual world. The goal is to properly align and be aware of the proportions of objects. If the system understands the proportion and is aware of the size, direction and distance between objects, it will be able to create much better Augmented reality scenes and applications.

Machine learning can help understand visual processing, audio processing and other sensor data more intelligent. This is still in the future, but the first wave of devices is happening now, and if compared with the smartphone market, we are roughtly at the iPhone 2G / 3G era when it comes to XR and AR right now. Things will rapidly improve with every single generation.