Apple Watch air gestures released! Here’s the story behind it | Exclusive

Apple officially launched watchOS 10.1 last night, bringing Double Tap to the two new Apple Watches – you can control the screen from the air with a tap of your fingertips: cut songs, answer calls, and reply to WeChat messages . Two weeks ago, I made an overseas phone call with the product manager of Apple Watch and talked about the story behind this powerful gesture.

Hard Philosophy is Ai Faner’s column reporting on hardware design. We hope to strip away the outer covering of technology and parameters and explore the origins of design and humanities.

"Tap each other twice with two fingers", the new function of this year's new Apple Watch is not a sudden inspiration of the product manager, but comes from what every user can relate to – "those moments where you can only click the watch button with your nose."

In an exclusive interview with iFan'er, watchOS software engineer David Clark bluntly stated that there are always some unsatisfactory interactions with Apple Watch:

The phone call comes while you're walking your dog. You're rushing to your next meeting, Mac in one hand, when you suddenly receive a message from a colleague. The timer on your watch goes off while you're baking, but your other hand is covered in egg wash and you don't want to touch it.

Eric Charles, Apple Watch product manager, said that these hesitant moments actually cost users more time to think about how to operate. The team wanted to provide a reliable and intuitive way to ensure everything runs smoothly without the need for additional operations when it counts.

Pay attention to the gesture interaction of Apple Watch, because everyone is experiencing "situational disabilities"

The prototype of "Two fingers tapping each other" can be traced back to 2015, less than a year after the release of Apple Watch.

In a patent document titled "Motion and gesture input from a wearable device," Apple attempts to explore the possibility of using tendon sensing and skeletal movement as input commands.

Six years later, these interactive explorations took the lead in giving birth to assistive touch (AssistiveTouch) in watchOS 8. Users can control the watch in the air through four gestures: pinch, pinch twice, make a fist, and make a fist twice.

As part of the accessibility features, the problems that gesture control had to solve at the time were:

How should a user control Apple Watch if he only has one hand?

One of Apple's design principles is to consider everyone's perspective. They believe that people with disabilities should not be treated differently based on their size, and that people with disabilities should have equal rights to use products.

But in the official interaction design document, Apple’s definition of “disability” is unusual :

Everyone can experience a disability. In addition to the disabilities that most people experience as they age, there are also temporary disabilities—such as short-term hearing loss due to infection—and situational disabilities—such as the inability to hear on a noisy train. Qing – These can affect everyone at different times.

The phone can be controlled with one hand, but the Apple Watch was designed to be fixed on the user's wrist from the beginning. When you touch the screen, both hands are actually occupied.

In a sense, every Apple Watch user only has one hand, and they all experience "situational disabilities."

To solve this problem, there are only two options: voice and gesture control. Gesture control based on physical movements is obviously more natural. It does not rely on the buttons and controls of the interface, and is especially suitable for devices with narrow screens like the Apple Watch.

In fact, the first-generation Apple Watch supported “raise your wrist to wake”, which established the basic paradigm of gesture interaction.

Apple has studied a variety of hand movements, including some complex gesture operations.

"Tap each other twice with two fingers" finally stood out from various gesture solutions because it met two criteria:

  • Simple and intuitive
  • clear purpose

The former requires no learning threshold, and the purpose clearly requires that the action must represent the user's intention and not be the result of an accidental touch.

For the same reason, "Tap each other twice with two fingers" can only be activated when the screen is on. Eric explains that if the user can't see the screen, they may not know what the response was.

A small bug in the heart rate sensor unexpectedly gave rise to the "tap each other with two fingers"

"Tap each other twice with two fingers" seems simple, just put two fingers together, but the working principle behind it is quite complicated.

"Tap each other with two fingers" is based on an algorithm built by machine learning, and its data comes from three sensors: accelerometer, gyroscope and optical heart rate sensor. The sensor will detect the characteristics of small movements and changes in blood flow when the hand and finger are tapped twice, thereby accurately obtaining the user's true input intention.

The optical heart rate sensor is an indispensable part. Although it is not designed for gesture control, the team found in years of research that the user's daily wrist movements will change the contact between the wrist and the heart rate sensor, even the smallest gap. All will have an impact on the data obtained by the heart rate sensor.

His arsenic, my honey. This unexpected discovery allowed the team to reveal the connection between gesture operations and heart rate sensors. Without these gaps, the "tap two fingers to each other twice" gesture may not be possible.

As David said:

The same sensor, the same data, we can achieve completely different purposes.

Unlike the gesture control of the auxiliary function, which runs on the CPU, "Two fingers to double tap" calls the quad-core neural network engine of the S9 chip, which improves the accuracy of algorithm detection by 15%.

This can explain why the old Apple Watch does not support "double tap with two fingers", and the auxiliary touch gestures are also much less reliable.

The team encountered many challenges during the development process. For example, how do you ensure that every user gets a consistent experience when faced with subtle differences in body types, limb lengths, and even gestures? How to overcome interference caused by Bluetooth wireless signals and LTE?

David mentioned that the team collected a large amount of data from within the company to train the machine learning model. The model will remove unnecessary noise and ensure the reliability of gesture detection in various scenarios, especially in non-stationary scenes. , can accurately detect gestures.

What you can do with "Two fingers tapping each other" depends on the Primary Action

After solving the algorithm problem, Apple still needs to design a mechanism at the software level to determine what "two fingers tapping each other" can do.

In my experience with Apple Watch Series 9/Ultra, I mentioned that the operation object of "tap each other twice with two fingers" does not have a predetermined direction. Unlike pinch to zoom, which always results in a zoom effect. And double-clicking each other with two fingers will bring different results in different apps.

According to the information processing theory in human-computer interaction, during the process of interacting with the interface, there will be continuous two-way processing of information between the user and the machine. The established feedback will help enhance the satisfaction of the operation and lower the threshold for use, and vice versa. Affect efficiency and experience.

Apple's solution is to set up logical Primary Actions for different Apps in different scenarios. For example, when a call comes in, the most logical action is to answer it, and to click again is to hang up. For WeChat notifications, the most logical way should be to reply, so tapping with two fingers can directly activate the voice-to-text function, and then tap with two fingers again to complete the sending.

Our principle is that you don't need to think too much, you just act when you need to.

After a month of use, I found that compared to gestures inside the app, instant gesture feedback for notifications is more practical, especially in high-frequency application scenarios such as phone calls and WeChat. The former is just the icing on the cake, while the latter is a timely help.

This also leaves developers with greater room for imagination. When I asked developers how to use Double Tap to improve the software experience, David told me the answer that was on my mind: respond to notifications.

Developers can leverage existing notification APIs to place the most important operations in the right place. In this way, when a notification arrives, the user can directly click with two fingers to complete the operation.

The problem solved by "tap each other with two fingers" is exactly the original intention of Apple Watch development

Wired magazine documents the little-known story of Apple's development of the Apple Watch. After vacillating on product definition, the team finally realized the reason for the existence of Apple Watch – smartphones are ruining your life.

Kevin Lynch, technical lead for Apple Watch software, said people carry their phones and look at the screen frequently. People scoff at those who sit with their phones at the dinner table, only to reach into their pockets every time their phone dings or vibrates.

The team ultimately decided to build a device that you neither would nor could use for hours on end, one that filters out most of the information and gives you only the most important information.

If creating products that make users addicted is the basic logic of commercial companies, then Apple Watch may be the first anti-attention Apple device.

On the developer website, Apple emphasizes that the core of the Apple Watch experience lies in Moment. In a previous interview, Kevin Lynch even set a time limit for this Moment: shortened from the past 10 seconds to 2 seconds.

The birth of "Two fingers to each other" is a leap forward for Apple Watch to get rid of interference. In many scenarios, I no longer need to use my other hand to control the watch, nor do I need to take out my phone all the time for intensive message notifications.

Apple Watch may not be able to completely free our hands, nor will it be the ultimate weapon against information overload, but it provides a new perspective in a way that makes a difference: when the torrent of information comes with anxiety, in addition to turning a deaf ear, we also How to get along with machines in a calm way.

From being a bystander and recorder of technology, to becoming a practitioner of how technology affects lifestyles.

# Welcome to follow the official WeChat public account of aifaner: aifaner (WeChat ID: ifanr). More exciting content will be provided to you as soon as possible.

Ai Faner | Original link · View comments · Sina Weibo