The Updated Qualcomm Spectra ISP is Going to Take AR and VR Into the Future Next Year

An overhauled depth sensing method will bring improvements to smartphone photography, too.

When it comes time to look at what’s next for Android smartphones, we look to Qualcomm. Virtually every flagship Android smartphone sold in the U.S. (save for the Huawei Mate 9) runs on a Qualcomm Snapdragon SoC, and that determines a lot more than just processing power. Like we saw in the 835 chipset, those Snapdragon systems help determine how smoothly 3D games play, how fast your network connection is, and how good your pictures look.

Today, Qualcomm is giving us a little preview of what we can expect from their next generation of chipsets by talking about their new Spectra ISP. We took a look at the Spectra ISP around this time last year, because it’s pretty important for camera quality. At that time, Qualcomm was introducing their dual camera system with complementary monochrome and color sensors, called Clear Sight. Clear Sight also enabled depth sensing — like most dual camera smartphones, the two cameras are used like two eyes, with the brains of the phone calculating how far away things are by measuring how offset objects are in the two images.

Next year, Qualcomm will be introducing something entirely different. I got to speak with Qualcomm reps about the upcoming update to the Spectra ISP and the new camera modules that will complement it. The highest-end module will have you seeing triple — well, it’ll look like your phone has three rear cameras, anyway.

Qualcomm is introducing something called active depth sensing, and for starters, it should make sure the subject of your bokeh shot gets outlined properly. The three units on the module are a regular camera, an infrared sensor, and the all-important infrared illuminator. The IR illuminator shines tightly packed little dots of light — about 10,000 of them — at distances between 10 and 16 feet. Because the dots are IR, you won’t see them, but the IR sensor will. The IR sensor and the Spectra ISP can scan for all those dots, then measure how small they appear or how they’ve been stretched around objects. That information gives the processor more exact information about depth — it’ll make more sense if you watch the video below, which shows a comprehensive depth map created by the camera module.

While I was talking to the Qualcomm reps, they mentioned one big problem with the current method of depth sensing — plain backgrounds. If you take a picture of something against a plain background, the image processor won’t see any difference between the two images, which can throw off depth of field effects. By actively using a matrix of IR dots, the processor won’t make this mistake — it’ll see exactly how small those dots are in the background, telling it exactly how far away the background is. That will result in clearly defined objects and faces when you’re using bokeh or portrait modes.

But, the IR illuminator does have limited range. The new modules won’t improve landscape shots in and of themselves, but Qualcomm is trying to make some separate improvements here, especially in low light. The new Spectra ISP will let these new modules take data from multiple frames for both still pictures and video, allowing the smartphone to stitch together a finished product that compensates for hand movements or a lack of light. It’s not as dramatic of a change as active depth sensing is, but the effects should be noticeable.

This won’t just be interesting for pictures — there’s a ton of potential for VR and AR, too. Current VR headsets like the Oculus Rift and the HTC Vive require separate towers that need to be set up in the room — by sensing the light coming from those towers, the headset can track your head movements and your body’s position. By using active depth sensing, those separate towers shouldn’t be necessary (and in fact, they already aren’t — HTC just announced a standalone Vive for China running on a Qualcomm Snapdragon 835 SoC). By using that IR dot matrix to create a continuous depth map, future VR headsets on their own should be able to track not only your body’s position, but even your hands — in theory, if the processing happens fast enough (and game developers take advantage of it), you’ll be able to interact with virtual objects naturally without need for separate VR controls or gamepads. Imagine being in a VR game and being able to pick up a box or flip a switch simply by making those movements, and you’ll get just a taste of what this should make possible.

There’s a little something planned for the front of the smartphone, too. Qualcomm will be offering IR iris scanning modules, which can scan your eyes to unlock your phone. It won’t be like the retinal scanners of science fiction that you have to get right up next to, either — it’ll be more like the IR cameras on Windows PCs that use facial recognition for logins. We’ll have to see how fast iris scanning works in practice — Windows Hello facial recognition, while not the same technology, can be slow and inconsistent — but if it’s implemented well, it could mark the end of fingerprint scanning. Current fingerprint scanners either have to take up space or be placed on the back of a phone, and figuring out how to put a fingerprint sensor under a touchscreen is going to require quite a bit more R&D dollars. Iris scanning could end up being a solution that takes up much less space, taking away what’s become a design constraint.

The new camera modules and the new Spectra ISP will debut in Qualcomm’s next-gen Snapdragon chipsets, which are still unannounced. We’re expecting to hear more about them before the year’s up, so there will be plenty more to discover in the near future.