Camera-based Hand Tracking as Unrestrained Input System
Disciplines
Computer Sciences
Abstract (300 words maximum)
The use of hand motion-based input systems has become increasingly popular in recent years, with applications ranging from virtual reality gaming to hands-free computing. However, existing systems are often limited by physical constraints, such as the need for gloves or full-body VR markers. In this paper, we present the development of a hand gesture-based input system that is not limited by these physical constraints.
The Novel Game Design lab team has been researching on a way to use the built in camera on the Quest 2 VR headset and the sdk that Meta has provided for the public. These built-in cameras are designed to capture images of the user's hands, which are then interpreted by the algorithms to be hand and finger movements. The system does not require the user to wear any additional equipment or markers, making it more comfortable and realistic to use.
In order to evaluate the effectiveness of our system, we made a game where the player is a wizard attempting to conjure spells with hand gestures. Initial testing indicates that the input system is effective. It’s been very responsive and accurate.
However, there were some limitations regarding the position of the hand. The lack of a camera behind the VR headset meant that it was not possible to detect finger/hand gestures outside of the camera’s peripheral view which resulted in occasional missed detections or incorrect tracking of hand movements.
Despite this limitation, our findings suggest that the use of hand gesture-based input systems with the Quest 2 has significant potential in a range of applications. The system is able to function without the need for additional equipment or markers, which would often limit several players from being able to use a hand gesture-based input system. Future work could focus on addressing the limitations of the system by adding/developing a camera that faces the other half of the view (the back) that the current system isn't able to see.
Academic department under which the project should be listed
CCSE - Software Engineering and Game Development
Primary Investigator (PI) Name
Henrik Warpefelt
Camera-based Hand Tracking as Unrestrained Input System
The use of hand motion-based input systems has become increasingly popular in recent years, with applications ranging from virtual reality gaming to hands-free computing. However, existing systems are often limited by physical constraints, such as the need for gloves or full-body VR markers. In this paper, we present the development of a hand gesture-based input system that is not limited by these physical constraints.
The Novel Game Design lab team has been researching on a way to use the built in camera on the Quest 2 VR headset and the sdk that Meta has provided for the public. These built-in cameras are designed to capture images of the user's hands, which are then interpreted by the algorithms to be hand and finger movements. The system does not require the user to wear any additional equipment or markers, making it more comfortable and realistic to use.
In order to evaluate the effectiveness of our system, we made a game where the player is a wizard attempting to conjure spells with hand gestures. Initial testing indicates that the input system is effective. It’s been very responsive and accurate.
However, there were some limitations regarding the position of the hand. The lack of a camera behind the VR headset meant that it was not possible to detect finger/hand gestures outside of the camera’s peripheral view which resulted in occasional missed detections or incorrect tracking of hand movements.
Despite this limitation, our findings suggest that the use of hand gesture-based input systems with the Quest 2 has significant potential in a range of applications. The system is able to function without the need for additional equipment or markers, which would often limit several players from being able to use a hand gesture-based input system. Future work could focus on addressing the limitations of the system by adding/developing a camera that faces the other half of the view (the back) that the current system isn't able to see.