Top Bananas

What it is: Washington University computer science graduate Nathan Gitter has released Rainbrow, an eyebrow-controlled arcade game for iPhone X.

The facial recognition features of the iPhone X means that it can analyze your facial features. Now a programmer has developed an arcade game called Rainbow, which lets you control the game completely through manipulating your eyebrows.

While this may seem like a trivial use of facial recognition, it’s far more important in that it creates a new type of user interface. With PCs, the user interface required icons and pull-down menus that a user could choose through a keyboard and a mouse. With smartphones and tablets, the user interface displayed menus and icons that a user could tap. The iPhone soon added voice control and 3D Touch that let you hold and press down on the touch screen surface. Now with Rainbow, the user interface has been extended to facial recognition.

This can be crucial for helping the disabled, but it also has uses for hands-free computing, which is what HomePod promises (whenever it arrives). By combining both voice and facial gesture commands, the future user interface will continue moving away from keyboards and mice to more intuitive, natural interfaces. Now you can speak and manipulate your face to control a computer.

Microsoft’s Kinect helped pioneer gesture recognition and then Apple acquired PrimeSense, the company that developed the technology behind Kinect. Now Apple is using PrimeSense’s technology for facial recognition and gesture recognition.

User interfaces continue to evolve. The Apple Watch helped make touch a two-way interaction. Instead of just tapping a touch screen, Apple Watch users can also receive feedback through the tactic engine that can tell you whether to turn left or right by giving you different types of taps on your wrist.

Combine tactic feedback with voice recognition and facial recognition and the world of computer user interfaces will continue to evolve, first with games and then with more commercial, business related uses, but it almost always starts with a game. Once people see the potential beyond having fun (think of the original SpaceWar video game that demonstrated the potential of cathode-ray tube terminals instead of punch cards to control a computer), they’ll soon recognize unique ways to use the latest form of user interaction.

Today you may touch your smartphone and tablet to control it. Tomorrow you may speak to it and gesture with your face or hand around it.

To learn more about using facial gestures to control an iPhone X, click here.

What it is: Android is the most popular mobile operating system in the world, closely mimicking iOS but offering additional features of its own.

At Apple’s recent event, they announced the new iPhone 6s and iPhone 6s Plus. Beyond a better camera, two main features of the new iPhone 6s models are Live Photos and 3D Touch.

Live Photos simply captures a few seconds of an image so when you browse through your pictures, they move slightly and can even store and play sound. That way you can get visual and audio clues to help you understand why you might have captured a particular image. Live Photos is a nice feature but it alone is no reason to update to an iPhone 6s.

For Android to keep up, that means Google will need to add similar Live Photos features and then get Android manufacturers to offer that feature in their next phone models. That should be relatively simple to do. What will be more difficult to mimic will be Apple’s 3D Touch.

3D Touch requires the use of a haptic engine. Apple designed the iPhone 6s models to allow 3D Touch and designed its included apps to take advantage of this additional way of interacting with the user interface. Instead of just touching or swiping across a two-dimensional flat surface, 3D Touch can detect hard presses. In the demo of 3D Touch, Apple showed that 3D Touch can simplify wading through one screen after another and then backing out again. Instead, one hard press on an icon that display a popup menu. Basically 3D Touch provides shortcuts.

From a software perspective, adding 3D Touch is fairly trivial. What will cause Android manufacturers problems is adding haptic engines to their next generation smartphones and tablets. They’ll need to wait for Google to add 3D Touch capabilities in the next version of Android, and then they’ll need to find a similar haptic engine to use.

Unfortunately for Android manufacturers, the addition of a haptic engine adds costs, and Android smartphones and tablets already have such razor thin margins that raising the cost to build one means even lower profits. If Android manufacturers don’t standardize on a haptic engine supplier, every Android manufacturer’s devices may provide different physical feedback despite using the same version of Android. This will create varying degrees of usability among Android manufacturers.

If a Samsung smartphone doesn’t offer the same haptic feedback as an HTC smartphone, that could cause people to remain loyal to HTC or another manufacturer. However it also means that Android developers will need to take advantage of similar 3D Touch capabilities in their apps, and they’ll need it to work reliably on various Android devices using different haptic engine parts. In other words, this could get messy and create less than optimal user experiences across different Android devices.

Android has to play catch up to iOS but the implementation will be key. Can Google get rival Android manufacturers to cooperate and create identical user experiences among Android devices? If so, then no manufacturer has an advantage. Or will some manufacturers keep the best parts for themselves to differentiate their products from their rivals, only to create drastically different Android user experiences as a result?

Android may be the most popular mobile operating system in the world, but it has a lot to worry about if it hopes to keep up with iOS in regards to 3D Touch. With 3D Touch, Apple has combined their hardware and software capabilities to create an integrated experience. With Android’s less than tightly coupled links between hardware and software, mimicking iOS’s 3D Touch remains chancy at best. Maybe Google and Android manufacturers can make 3D Touch work just as well on Android as it does on iOS, but don’t bet on it.

Most likely Android will sink further behind iOS and struggle to keep up while falling steadily behind anyway.

Tag Archives: 3D Touch

Controlling a User Interface with Your Face Gestures

Android Needs to Catch Up