Intention Prediction for Encountered-type Haptic Interaction for VR-based Immersive Telemanipulation

Abstract

This thesis presents a learning-based system designed to enhance immersive 3D user interaction with robotic systems. To achieve a truly immersive experience, the feedback provided by an encounter-type haptic robot must be both precise and accurate. This requires accurately predicting the contact points between the user and virtual objects. By anticipating user intentions in advance, the system can provide faster and more accurate responses.

To accomplish this, we introduce MiXR-Interact, a dataset providing motion-tracking data for users' interactions in mixed reality (MR) environments, focusing on tracking their gaze, upper body movements, and hand gestures. The dataset is based on the Meta Quest Pro headset, offering an easy-to-use resource for researchers and developers working in MR and HRI. The dataset includes three core interactions: pushing, pointing, and grasping. Each interaction is performed in six distinct directions to capture diverse movement trajectories relative to the user's body. In addition, to precisely track contact points during interactions, 17 key contact points are defined for each direction and are labeled. These contact points are used as reference markers to accurately localize and quantify the joint-to-object contact points for each interaction type and direction. Moreover, this thesis evaluates MiXR-Interact using metrics such as trajectory similarity, joint orientation, and joint-to-contact alignment.

Based on MiXR-Interact, we developed an LSTM-based neural network to predict user intentions. The classification model identifies the interaction type, while the regression model estimates 3D contact points. Performance evaluations and different studies demonstrate strong predictive capabilities. In conclusion, this work introduces a novel dataset and develops deep-learning models for predicting human intentions from behavioral features and MR environment data, improving the accuracy of encounter-type haptic feedback in MR.

This thesis presents a learning-based system designed to enhance immersive 3D user interaction with robotic systems. To achieve a truly immersive experience, the feedback provided by an encounter-type haptic robot must be both precise and accurate. This requires accurately predicting the contact points between the user and virtual objects. By anticipating user intentions in advance, the system can provide faster and more accurate responses.

To accomplish this, we introduce MiXR-Interact, a dataset providing motion-tracking data for users' interactions in mixed reality (MR) environments, focusing on tracking their gaze, upper body movements, and hand gestures. The dataset is based on the Meta Quest Pro headset, offering an easy-to-use resource for researchers and developers working in MR and HRI. The dataset includes three core interactions: pushing, pointing, and grasping. Each interaction is performed in six distinct directions to capture diverse movement trajectories relative to the user's body. In addition, to precisely track contact points during interactions, 17 key contact points are defined for each direction and are labeled. These contact points are used as reference markers to accurately localize and quantify the joint-to-object contact points for each interaction type and direction. Moreover, this thesis evaluates MiXR-Interact using metrics such as trajectory similarity, joint orientation, and joint-to-contact alignment.

Based on MiXR-Interact, we developed an LSTM-based neural network to predict user intentions. The classification model identifies the interaction type, while the regression model estimates 3D contact points. Performance evaluations and different studies demonstrate strong predictive capabilities. In conclusion, this work introduces a novel dataset and develops deep-learning models for predicting human intentions from behavioral features and MR environment data, improving the accuracy of encounter-type haptic feedback in MR.