HandyTrak: Recognizing the Holding Hand on a Commodity Smartphone from Body Silhouette Images

HandyTrak: Recognizing the Holding Hand on a Commodity Smartphone from Body Silhouette Images
Hyunchul Lim, David Lin, Jessica Tweneboah, Cheng Zhang

UIST'21: ACM Symposium on User Interface Software and Technology
Session: Motion Tracking

Abstract
Understanding which hand a user holds a smartphone with can help improve the mobile interaction experience. For instance, the layout of the user interface (UI) can be adapted to the holding hand. In this paper, we present HandyTrak, an AI-powered software system that recognizes the holding hand on a commodity smartphone using body silhouette images captured by the front-facing camera. The silhouette images are processed and sent to a customized user-dependent deep learning model (CNN) to infer how the user holds the smartphone (left, right, or both hands). We evaluated our system on each participant's smartphone at five possible front camera positions in a user study with ten participants under two hand positions (in the middle and skewed) and three common usage cases (standing, sitting, and resting against a desk). The results showed that HandyTrak was able to continuously recognize the holding hand with an average accuracy of 89.03\% (SD: 8.98\%) at a 2 Hz sampling rate. We also discuss the challenges and opportunities to deploy HandyTrak on different commodity smartphones and potential applications in real-world scenarios.

DOI:: https://doi.org/10.1145/3472749.3474817
WEB:: https://uist.acm.org/uist2021/

Talk Recording of the UIST 2021 Papers Program