What it is: Apple acquired the German augmented reality company Metaio, which was recently granted a patent that gives clues on Apple’s augmented reality plans.
If you’ve heard of Pokemon GO, then you already had a glimpse of the potential behind augmented reality. The basic idea is to show an imaginary object overlaid across an image of a real location. In Pokemon GO, you could aim your smartphone at a park bench and see a Pokemon character looking back at you.
The real trick to adding virtual objects to a real environment is making that virtual object look like it’s actually in the real world. It’s easy to plop a cartoon image over a real image, but to make that cartoon image look like it’s actually standing next to a chair with its feet on the floor requires high-definition cameras, powerful onboard image processors and advanced positioning and location hardware. Not surprisingly, most of the required tools have just recently reached the consumer marketplace thanks to iPhone.
To do this accurately, a camera must gather spatial information including the relative positioning of floor and wall planes using depth mapping, radar, stereo cameras or other techniques. The reason why Apple is quietly working on augmented reality is to get this process as accurate and flawless as possible. That means improving both software algorithms and hardware cameras. When you hear of rumors about a 3D camera coming to the iPhone, it’s likely for augmented reality and less about capturing 3D images just for the sake of capturing 3D images.
Even more challenging is removing real objects from an image. Just as Photoshop allows you to digitally erase people from a picture, so can augmented reality remove an image from reality. The hard part is filling in the background so it’s not obvious that there’s something missing in the image.
So augmented reality involves both adding new items to an image and taking real items away from a real image. Such processing will likely require a dedicated graphics processor, which could be another reason why Apple recently dumped Imagination Technologies’s graphics processors. Apple needs specific features from a graphics processor and they need to optimize it for augmented reality.
Augmented reality is coming this year or the next. Apple is just waiting for hardware to catch up to their software. When augmented reality arrives, expect this feature alone to drive sales of the iPhone and iPad.