Wearable units, from smartwatches to health trackers, have grow to be ubiquitous, constantly capturing a wealthy stream of knowledge about our lives. They document our coronary heart charge, rely our steps, monitor our health and sleep, and rather more. This deluge of data holds immense potential for personalised well being and wellness. Nonetheless, whereas we will simply see what our physique is doing (e.g., a coronary heart charge of 150 bpm), the essential context of why (say, “a brisk uphill run” vs. “a aggravating public talking occasion”) is usually lacking. This hole between uncooked sensor knowledge and its real-world which means has been a serious barrier to unlocking the complete potential of those units.
The first problem lies within the shortage of large-scale datasets that pair sensor recordings with wealthy, descriptive textual content. Manually annotating thousands and thousands of hours of knowledge is prohibitively costly and time-consuming. To resolve this, and to actually let wearable knowledge “converse for itself”, we want fashions that may study the intricate connections between sensor alerts and human language straight from the information.
In “SensorLM: Studying the Language of Wearable Sensors”, we introduce SensorLM, a household of sensor–language basis fashions that bridges this hole. Pre-trained on an unprecedented 59.7 million hours of multimodal sensor knowledge from over 103,000 people, SensorLM learns to interpret and generate nuanced, human-readable descriptions from high-dimensional wearable knowledge, setting a brand new cutting-edge in sensor knowledge understanding.