ST is showcasing its first AI algorithms utilizing the embedded NPU of the Snapdragon Wear Elite, a premium Intelligent Wearable platform for smartwatches, pins, and other personal AI form factors.
The Snapdragon Wear Elite development platform uses our LSM6DSV32X, a six-axis inertial sensor with a machine learning core and finite state machines to run AI applications at a fraction of the power consumption a similar algorithm would need on an MCU. To showcase what this sensor can do on the Snapdragon Wear Elite, ST built multiple applications, including a walking speed estimator that runs on Qualcomm Technologies’s eNPU. We also implemented fall detection and worn/not worn recognition using algorithms that run on the machine learning core (MLC) and finite state machine (FSM) inside the LSM6DSV32X.
More than a milestone, these are concrete ultra-low-power examples of what developers can run today on the Snapdragon Wear Elite, enabling them to develop increasingly feature-rich products faster and, consequently, reduce their time to market. Some of these applications, or variations thereof, are already available on our GitHub repositories showcasing application examples for the LSM6DSV32X’s machine learning core and finite state machines. What we also want to show is how Qualcomm Technologies’ ultra-low-power AI frameworks cohesively interact and stack with ST’s on-sensor edge AI and embedded solutions. Let us, therefore, explore the three AI applications we ran on the Snapdragon Wear Elite and what developers can expect.
Application 1: Walking speed estimator #
Why does inference time matter?
Let’s start with numbers. According to industry conventions, a walking speed estimator makes roughly one prediction every 250 ms, or about four per second, which offers a good compromise between accuracy and the computational resources assigned to that task. ST’s walking speed estimator running on Qualcomm Technologies’ eNPU performs an inference in 31.2 ms on average, as measured across 16 executions on the Snapdragon Wear Elite platform. In other words, since prediction operations take only 12.5% of the time they normally do, the application has roughly 219 ms for sensor acquisition, pre-processing, and any other task the AI pin or other wearable will need to accomplish, thus offering huge efficiency gains.
Digging further into these numbers, we see that traversing the Qualcomm® AI Engine Direct graph takes 28.7 ms out of the 31.2 ms. Out of that 28.7 ms, 25.1 ms is accelerator time, with about 88% being graph traversal. Put simply, the eNPU is mostly doing arithmetic with very little time needed to move data around. It speaks to the platform’s efficiency and its ability to run a neural network without having data transfer being a bottleneck. Furthermore, the compiled model buffer took only 1.22 MB and executed only 75.3 million operations per inference. At four predictions per second, that’s barely 0.015% of the available TOPS capacity, enabling it to run on something as small as an AI pin.
What are some of the challenges in estimating walking speeds?
As the name implies, the walking speed estimator detects whether someone is walking, their direction, and the overall motion and energy spent. It’s a hard problem to solve because the same number of steps per minute can offer very different results depending on the speed and gait. As a result, the AI algorithm must distinguish between walking, running, someone simply swinging their legs, and general fidgeting. Additionally, the signals captured by the IMU look very different from an AI pin in a pocket to a smartwatch on a wrist, meaning that the algorithm must account for a wide range of conditions, including the many ways the user wears the product.
An accurate walking speed estimator must use accelerometer magnitudes, gyroscope magnitudes, and signed acceleration if it’s going to figure out travel direction and motion energy. However, a battery-constrained wearable cannot spend a vast amount of time and resources gathering and processing this information. By using the embedded pedometer on the LSM6DSV32X, ST only runs computations on the Snapdragon Wear Elite’s eNPU when the sensor starts to detect walking, not simply motion.
Afterward, a convolutional neural network and bidirectional long short-term memory (CNN-BiLSTM) network running on the eNPU can pick out local gait signatures and the evolution of gait events across a complete walking sequence. Beyond the usual gait metrics (swing time, stance time, gait cycle time, initial support time, etc.), we have also started implementing additional comprehensive gait metrics such as double support.
How are developers benefiting from our collaboration?
A walking speed estimator is a complex machine learning algorithm that requires massive amounts of data to be useful. Thanks to the use of our sensor and the optimizations afforded by our use of Qualcomm Technologies’ eNPU, we were able to run this on a system with energy and computation constraints. This application is the result of a collaboration announced last March between ST and Qualcomm Technologies, and shows how our teams harness the unique abilities of our sensor, but also master AI algorithms, enabling us to take advantage of Qualcomm Technologies’ eNPU to create a unique demo application that can now serve as an object lesson for an entire community of developers.
Application 2: Worn/not worn #
How do multiple AI algorithms inform each other?
As observed above, knowing when to apply power optimization features is critical, but it is also challenging. An AI pin will detect movement, whether the user affixed it to a piece of clothing or left it in a bag. Consequently, knowing if the user is wearing the item is fundamental. However, a worn/not worn algorithm doesn’t simply detect a pose, but multiple consecutive states, such as jogging, walking, or other various daily activities. ST not only developed a worn/not worn algorithm, but we are running it on our LSM6DSV32X by using three finite state machines and one MLC decision tree that all talk to each other, enabling us to reliably detect gestures and motion signatures while handling false positives.
Under the hood, the mechanisms are pretty common. Qualcomm Technologies’ low-power Sensing Hub and its eNPU stay in a low-power or sleep mode as long as possible to conserve battery. When ST’s on-sensor worn/not worn algorithm on the LSM6DSX32X detects that the pin is worn, it raises an interrupt on INT1. Practically, that means the Sensing Hub only runs computations if the user is actually wearing the product and walking (detected by the embedded pedometer), thus only actively using the eNPU when needing to run the walking speed estimator. Having an accurate system is paramount, as false positives will waste the little battery life available, while false negatives will prevent the walking speed algorithm from running when needed.
What can advanced development tools do?
The algorithm has 95% accuracy in our internal tests, with no user calibration, thanks in part to the decision tree optimization feature in MEMS Studio called Automatic Feature and Filter Selection. As the name implies, our software tool can automagically determine the best filters, features, and window length to apply to a dataset based on available resources and then train a decision tree based on the resources available. When running on the LSM6DSV32X, the algorithm only requires µA to run, and it will work regardless of the platform since everything happens on the sensor. By contrast, the classical embedded software on an STM32 microcontroller would consume milliamperes and would need to be ported to fit the device driving it.
Application 3: Fall detection #
What happens at the algorithm level?
Avid readers of the ST Blog will remember that fall detection was one of the earliest AI applications we ran on our LSM6DSV32X because its 32g accelerometer opened the door to detecting the broad range of motion needed to determine whether something or someone is only moving or truly falling. The algorithm looks for three phases in that order: free fall, impact, and stabilization into stillness. In a nutshell, the machine learning core of the sensor classifies the window, while the finite state machines determine the type of movement that just took place and whether they match the three phases defined by the algorithm.
What does a great dataset look like for fall detection?
We trained the ST algorithm using a dataset taken from 1,326 participants wearing sensors at 43 body locations in unrestricted orientations (140 days of recording), measuring 174 everyday activities labeled as non-falls, such as jumping, throwing a ball in sport, flopping onto a sofa, alongside falls in every direction, from chairs to standing up, and with a prevention attempt mid-fall. The solution running solely on the ST IMU achieves 99% accuracy on data unseen during training, while needing only 15 µA. For further power efficiency, teams can run the algorithm using only the machine learning core of the LSM6DSV32X. This approach offers accuracy that’s 96% as good as when the application uses the machine learning core and finite state machines.
We would also like to acknowledge the ST engineers who worked on the development/enablement of the features mentioned in the blog post:
Engineers
- Sneha SHAHI
- Ashish BHARGAVA
- SIDDHARTH
- Swapnil Sayan SAHA
- Krishna Chaitanya PALLE HAYAGREEVA
Intern
- Allen CHELLASAMY
Snapdragon and Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries. Qualcomm, Snapdragon and Snapdragon Wear are trademarks or registered trademarks of Qualcomm Incorporated.