When Lenovo announced the Google Tango-enabled Phab2 Pro at their Lenovo Tech World event earlier this month, we noticed that it ran on an unfamiliar Qualcomm Snapdragon 652 SoC. At MWC Shanghai this week, Qualcomm is shedding a little more light on that SoC and their involvement in Google’s Tango technology.
As a refresher, Google Tango is Google’s new augmented reality initiative. It’s a step up from regular augmented reality — usually, we see augmented reality programs drape some information over the camera’s feed on a smartphone display. This is done using augmented reality markers that you have to place yourself — the Air Hogs drone flying game we looked at during E3 is a good example. Sometimes, more sophisticated programs can use GPS to come up with contextual information based on your location, like showing a restaurant’s hours when you point the camera at it.
Tango, on the other hand, actively analyzes and processes details in your surroundings. Tango uses Google’s impressive image database to use machine learning algorithms to detect that a table is a table (sounds easy, but it isn’t). With sensors unique to Tango-enabled devices, Tango can then take measurements or let you alter the real-world environment on your smartphone’s screen — a good example is the Lowe’s app that lets you place furniture in a room or see what a new paint job would look like on the walls.
Tango also introduces positional tracking — whereas a smartphone used to know only when it was being tilted or rotated, Tango-enabled phones know how far they’re being moved in one direction. That allows Tango to display that table on screen, then change the viewing angles as you walk around it. The hardest thing for you to do is get used to the idea of walking around an empty space to view all angles of a virtual table.
It’s no easy task for a processor, though. Right now, only two chipsets are capable of handling the load, and they’re both from Qualcomm. Besides the 652 that we saw on the Phab2 Pro, Qualcomm is announcing that their current flagship SoC, the Snapdragon 820, can also run Tango. The Snapdragon 810 was originally slated to run Tango, but only ever powered developer models that never made it to market. The 652 is Qualcomm’s top SoC in their second tier of chipsets, and was built to be optimized for Tango — no simple process, as it turns out. Here’s where things get technical.
I talked with Seshu Madhavapeddy, vice president of product management at Qualcomm, about what exactly goes into making Tango work. It starts with hardware. The typical smartphone has four sensors — a gyroscope, an accelerometer, and the rear and front cameras (the latter of which isn’t used in Tango). Tango-enabled smartphones add two more sensors — another rear camera with a fisheye lens and a depth sensor. Data is continually taken in through all five sensors — the gyroscope, the accelerometer, the rear camera, the rear fisheye camera, and the depth sensor — before being analyzed by Tango and made useful.
The difficult part, and the part that requires the Snapdragon 652 and 820, is synthesizing that data. As Madhavapeddy pointed out to me, it’s not as simple as kicking everything to the CPU and GPU — they’ll get overloaded and won’t have enough resources to run Tango apps themselves. So, the Qualcomm SoCs do some load sharing. A dedicated sensor processor handles the gyroscope and the accelerometer, while a feature tracker within the digital signal processor handles the new information from the fisheye camera. The rear camera feed is sent through a dedicated image processor, while information from the depth sensor gets sent to a dedicated Neon processor within the CPU. The processed data is eventually kicked up to a core dedicated to Tango within the CPU, which uses less than 10 percent of the CPU’s capacity. The take-home is that all of this data analysis takes up less than 10 percent of the CPU’s resources and none of the GPU’s resources, allowing the phone to run apps while making Tango work in the background.
Things get even more complicated. All of these sensors take in data at different rates, but the data needs to be synchronized. If the SoC combines data from the depth sensor with data from the camera that came in a few milliseconds later, the end result could look pretty bad on screen — virtual tables floating when they should be placed on the ground. Before data is sent to any processor, input from all five sensors is timestamped to an accuracy of nanoseconds, so the SoC knows to put data that came in at the same time together. Only then is everything analyzed, synthesized, and put on screen — a lot of work to see how that new recliner is going to look in the living room!
As we thought when we first heard about Tango, Qualcomm’s in-depth involvement in the project means that Tango has a good chance of becoming a standard feature in smartphones over the next couple years. Considering most 2016 Android flagships are already out on the market and lack the depth sensor and fisheye camera, we don’t expect to see very many Snapdragon 820 phones running Tango this year. But, when Qualcomm inevitably refreshes its high-end line for 2017, we’re thinking more premium smartphones than not will take advantage of it.
That said, we might not necessarily see it on all flagship phones. The augmented reality features of Tango work best on a large screen, which is part of the reason why it launched on the Phab2 Pro, a 6.4″ device. It might not be suitable for phones closer to 5″, but for the larger devices like the Galaxy Edge, we’re thinking Tango could either become standard or available on a special edition. But, we’ll have to wait until next year to find out for sure!