Turning is character turns left/ right. Rotation is
character rotates his body left/right or up/down to
observe the scene. This action changes eye view
to make decision of other actions.
thumb ﬁnger is worn marker for determining
the exact location, and recognizing of complex
hand gestures as well.
In addition to character
to perform other actions in the game such as,
shooting, and selection.
Firstly, we deﬁne some parameters later used
in the process of navigation. Root is player’s
Fig. 1: Kinect skeleton.
neck joint (Fig. 1) on player’s.
Rootdepth is the
the Kinect sensor for a while to obtain some
distance from Kinect sensor to root.
measurements such as, root position and hand
is the distance from Kinect sensor to the wrist
joint of the hand.
HandX is the wrist joint in
calculate character’s speed and acceleration.
horizontal direction in acquired image.
hand marker tracking,
we use emguCV
is root in horizontal direction.
and POSIT algorithm
 to determine marker
and its’ rotation.
Then, above deﬁned actions
OrthVector is a vector orthogonal to
Finally, we perform navigation
straightVector on the left.
estimation by recognized actions.
OrthSpeed are speed vectors in movements.
We deﬁned three types of navigation: straight
movement, turning (left/right) and rotation
corresponding to the following actions:
4.1. Parameter setup
Straight movement: move the wrist towards
or away from the root. The change of the
distance from the wrist to the root decides
the direction (forward/backward) and speed
At the beginning process, player is required
to stand in front of Kinect to collect initial data
about root and wrist joint position. Player simply
need to be in the pose such that Kinect can see
his root and wrist joint position. Player wears a
Turning: change the wrist to left or right
while distance between the wrist and root
does not change.
Rotation: rotate the wrist. The changed
marker in front of ﬁst. In order to track rotation
easily, we should use marker in the form of
checker board. We deﬁne some stand distances
by the initial depth data as follows:
angle decides the speed of rotation.
InitDistStraight = HandDepth RootDepth
These actions are determined by comparing
the position of wrist with root and rotation of
InitDistOrth = HandX RootX
marker glued on a thumb ﬁnger. In detail,
navigation technique includes three main steps
(Fig. 2): parameter setup, hand marker tracking,
and navigation estimation. In parameter setup,
players are asked to stand or sit in front of
HƯỚNG DẪN DOWNLOAD TÀI LIỆU
Bước 1:Tại trang tài liệu slideshare.vn bạn muốn tải, click vào nút Download màu xanh lá cây ở phía trên.
Bước 2: Tại liên kết tải về, bạn chọn liên kết để tải File về máy tính. Tại đây sẽ có lựa chọn tải File được lưu trên slideshare.vn
Bước 3: Một thông báo xuất hiện ở phía cuối trình duyệt, hỏi bạn muốn lưu .
- Nếu click vào Save, file sẽ được lưu về máy
(Quá trình tải file nhanh hay chậm phụ thuộc vào đường truyền internet, dung lượng file bạn muốn tải)
Có nhiều phần mềm hỗ trợ việc download file về máy tính với tốc độ tải file nhanh như: Internet Download Manager (IDM), Free Download Manager, ... Tùy vào sở thích của từng người mà người dùng chọn lựa phần mềm hỗ trợ download cho máy tính của mình
Kinect Based Character Navigation in VR Game
| Thể loại:
| Lần tải: 0
| Lần xem: 4
| Page: 8
| FileSize: 0.00 M
| File type: PDF
Kinect Based Character Navigation in VR Game. In this paper, we propose a technique of Kinect based character navigation in VR Game. As environment is supported by Kinect, we implement character navigation by player’s hand actions. Besides, we use markers to increase the effectiveness of hand action recognition. In our experiments, character’s rotations reaches 70% in accuracy. Straight movement and turning reach 98% in accuracy..
VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30
Kinect Based Character Navigation in VR GameI
Thi Chau Ma, Minh Duong Hoang
VNU University of Engineering and Technology,
E3 Building, 144 Xuan Thuy Street, Cau Giay, Hanoi, Vietnam
VR game is a hot ﬁeld because it provides practical experience for the player. However, in the game, players are usually required attaching special equipments. Those equipments are often very expensive. Moreover, many devices make players less ﬂexible because they are attached to players. VR game using Kinect is a good choice because Kinect is aordable and not attached directly to the player. In this paper, we propose a technique of Kinect based character navigation in VR Game. As environment is supported by Kinect, we implement character navigation by player’s hand actions. Besides, we use markers to increase the eectiveness of hand action recognition. In our experiments, character’s rotations reaches 70% in accuracy. Straight movement and turning reach 98% in accuracy.
Received 30 November 2015, revised 09 January 2016, accepted 14 January 2016
Keywords: Kinect application, hand recognition, VR games.
1. Introduction We are interested in games using Kinect
In the last two years, in Vietnam, the game attracted a lot of investments and researches. According to , Vietnam is the largest market in Southeast Asia Game. However, restrictions still exist, obviously, the solutions are still awaited. An important factor that make the game to be attractive is the ability of character movements. To control the movement of the characters, players have to touch and navigate specialized devices to control the movements of the characters. These devices were typically gamepad, keyboardist, Wiimote. Besides, with technology advances, in virtual reality games, players no need to touch game devices. In these games, body actions or speech were used to control the character movement through the special devices, such as Kinect, Virtuix Omni.
I This work is dedicated to the 20th Anniversary of the IT Faculty of VNU-UET.
Corresponding author. Email: email@example.com
because the type of these games gives players the ﬂexibility and comfort while playing without exposure Kinect. Moreover, Kinect relatively low cost comparing with other devices such as, head mounted devices or data gloves. Kinect-supported games can be classiﬁed into two main categories. In the ﬁrst type, entire bodies of players are identiﬁed to contribute and play games like audition games , or collision ball games . In the second type, hand gestures are recognized to control games . In our research, we want to make use of hand actions obtained by Kinect to navigate character in order to give players the freedom and comfort in playing games. However, the accuracy of the hand action recognition and the number of hand actions based on Kinect were limited because of identical when recognizing dierent bone joints. In this paper, we propose a solution of Kinect based character navigation in VR Games. We recognize hand action to navigate characters in
24 T.C. Ma, M.D. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30
VR Games. Marker is glued to the hands of to see the above Kinect applications are based on
players to obtain more number of hand actions and increase accuracy in hand action recognition.
recognition of human skeleton.
Another type of Kinect applications is based
The use of glued makers does not reduce the on hand actions . In ,
degree of ﬂexibility of players.
2. Related Works
With the available and aordable Kinect, Kinect and its applications appear a lot in the life. First of all, we mention the applications in the medical. KinectprovideaSoftwareDevelopment Kit (SDK) which gives access to skeletal tracking data and can be used directly in rehabilitation game developments. So, there are lots of physical therapy and rehabilitation applications  using Kinect. These applications are classiﬁed into 3 groups. Fall detection and Fall risk reduction, Stroke Rehabilitation and exercise games. In , authors presented a low-cost system to surgical training for computer assisted surgeries by utilizing dual sensors and marker based tracking components. Markers were patterns in the form of checkboard. By using markers, they implemented calibration of dual infrared sensors and point correspondence establishment in a stereo conﬁguration. Then they accomplished 3D localization by multiple back projection lines. The system was eectively thecombinationofinexpensiveelectronicdevices and traditional tracking algorithm.
Secondly, in the VR environment, a hot and exciting ﬁeld, Kinect attracted much attention. Kinect is one of easy ways of 3D reconstruction. In the environment, data recognized by Kinect is
authors accurately reconstructed complex hand poses across a variety of subjects using Kinect. The system allowed continually recovering from tracking failures. Matthew Tang  showed a novel method for recognizing hand gestures using Kinect sensor. In the approach, they recognize ’grasp’ and ’drop’ gestures with over 90% accuracy. In her thesis , Yi li used K-means clustering and Graham Scan algorithm to recognize nine popular gestures and counting numbers. Kinect data range was between 0.5m to 0.8m. Hands were distinguished from the background by depth information.
In Kinect applications in particular and recognitions in general, Marker is an eective tool. Marker is a very simple sticker, usually in the form of checker board pattern. Marker is popular in modeling by its simple manipulation and eciency. Checker board pattern has been using in camera calibration  to reconstruct 3D model from 2D images. Checker board pattern is also use in Kinect calibration  to align depth camera and RGB camera. Marker help robot to detect path to navigate.
In this paper, we want use hand actions from Kinect to control the movement of character in VR game. We utilize marker to detect hand actions easily and accurately.
3. Proposed method
divided in two types: Human skeleton and hand In this paper, we propose a Kinect-based recognitions. RGBDemo  is an open source navigation technique in VR game. In the software providing a simple toolkit to manipulate game, we use a Kinect sensors to obtain Kinect data. In this software, 3D model of an player’s hand actions. The players navigates object would be rebuilt when the object are taken characters using actions only on one hand. The with markers. 3D builder app  and Kinect character’s actions include straight movement, sensor are used to create a 3D reconstruction turning (left/right) and rotation. We chose these for a 3D printer. Ultra Seven  is a game action in implementation because hand gestures used Kinect in AR environment. In the game, are meaningful to people. So, learning and the player is turned into a character and shot using hand gestures are easy. Moreover, in the his power. The character was modeled game game, most free parts of body but hand makes player by body recognition with Kinect. Easy player more comfortable. In technique term,
T.C. Ma, M.D. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30 25
the problem is more focus. Meanings of these actions are described clearly as follow. Straight movement is character goes forward/backward. Turning is character turns left/ right. Rotation is character rotates his body left/right or up/down to observe the scene. This action changes eye view to make decision of other actions. The player’s thumb ﬁnger is worn marker for determining the exact location, and recognizing of complex hand gestures as well. In addition to character navigation, the player can also use the hand
to perform other actions in the game such as, shooting, and selection.
Firstly, we deﬁne some parameters later used in the process of navigation. Root is player’s neck joint (Fig. 1) on player’s. Rootdepth is the distance from Kinect sensor to root. Handdepth is the distance from Kinect sensor to the wrist joint of the hand. HandX is the wrist joint in horizontal direction in acquired image. RootX is root in horizontal direction. StraightVector is a vector showing straight direction of the character. OrthVector is a vector orthogonal to straightVector on the left. StraightSpeed and OrthSpeed are speed vectors in movements.
We deﬁned three types of navigation: straight movement, turning (left/right) and rotation corresponding to the following actions:
Straight movement: move the wrist towards or away from the root. The change of the distance from the wrist to the root decides the direction (forward/backward) and speed of movement.
Turning: change the wrist to left or right while distance between the wrist and root does not change.
Rotation: rotate the wrist. The changed angle decides the speed of rotation.
These actions are determined by comparing the position of wrist with root and rotation of
Fig. 1: Kinect skeleton.
the Kinect sensor for a while to obtain some measurements such as, root position and hand depth. These measurements are utilized to calculate character’s speed and acceleration. In hand marker tracking, we use emguCV  and POSIT algorithm  to determine marker and its’ rotation. Then, above deﬁned actions are recognized. Finally, we perform navigation estimation by recognized actions.
4.1. Parameter setup
At the beginning process, player is required to stand in front of Kinect to collect initial data about root and wrist joint position. Player simply need to be in the pose such that Kinect can see his root and wrist joint position. Player wears a marker in front of ﬁst. In order to track rotation easily, we should use marker in the form of checker board. We deﬁne some stand distances by the initial depth data as follows:
InitDistStraight = HandDepth RootDepth
InitDistOrth = HandX RootX
marker glued on a thumb ﬁnger. In detail, navigation technique includes three main steps (Fig. 2): parameter setup, hand marker tracking, and navigation estimation. In parameter setup, players are asked to stand or sit in front of
HandRangeStraight = Min(HandDepth)
HandRangeOrth = Min(HandX)
26 T.C. Ma, M.D. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30
into body parts, this information is pooled to generate the position of 3D skeletal joints. The meanshiftalgorithmisusedtocomputemodesof probabilitydistributions. Byusingthisalgorithm, the densest region of body part is extracted. For more detail information, please read .
4.2.2. Marker tracking
The marker detection has three main steps. First, the contour detection algorithm is used to detect all quadrangle. To normalize the marker image, the input image must be unwarp by using perspective transformation. The estimated transformation will transform the marker the square form to extract data from it. In the
Fig. 2: Overview method process. 4.2. Hand and marker tracking
implementation, we use simple marker 5x5 bit. For each normalized quadrangle, the image is dividedintocells. Thedataofeachcellisdecided by the number of white pixels; if the number of white pixels is greater than the number of black pixels then the cell’s value is 1, otherwise,
Hand tracking is processed by using Kinect the value is 0. After comparing the candidate
SDK. To detect marker, we process RGB image from Kinect to get contours ﬁtting for 4 vertices. After that, camera is calibrated in stardard form, and data of marker is extracted in the form of a 2D matrix.
4.2.1. Skeleton tracking
The skeleton tracking feature of Kinect is used to get hand position. This process has two stage: ﬁrst computes a depth map (using structed light), then infers body position (using machine learning). At ﬁrst stage, the depth images are all computed by the PrimeSense hardware built into Kinect. Pixels in a depth image indicated calibrated depth in the scene, rather than a measure of intensity of color. Depth images oer several beneﬁts, such as working in low light levels, giving a calibrated scale estimate, being color and texture invariant and resolving silhouette ambiguities in pose.
The depth image received from previous stage
is transformed to body part image by using
marker’s value with the real marker’s value, the marker is recognized.
4.3. Navigation estimation
4.3.1. Forward, backward
When playing, player do hand actions. While distance between wrist joint changes, the character moves straight away. If the distance increases, character moves forward. Otherwise, character moves backward. The estimataion of straight speed is shown in Algorithm 1.
At each frame, the depth distance between Hand and Root depthDistance is calculated and the dierence between current pose and initial pose rawVertical is obtained. If rawVertical is greater than minimum distance minDeltaStraight, rawVertical is converted into range [0;1] and assigned to scaledVertical. Otherwise, scaledVertical is 0. After that, the speed vector of forward movement is calculated.
4.3.2. Left, right
randomized decision forest. This tree is learned We use horizontal distance between root from 100,000 depth images with known skeleton and wrist joint to control character turning and theirs derivations. After classifying image (Algorithm 2).
T.C. Ma, M.D. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30 27
Algorithm 1: Estimate forward/backward
1 depthDistance RootDepth - HandDepth
2 rawVertical (depthDistance InitDistStraight) 3 if rawVertical > minDeltaStraight then
4 scaledVertical rawVertical / HandRangeStraight 5 else
6 scaledVertical 0
7 StraightSpeed StraightVector playerMaxS peed scaledVertical
Algorithm 2: Estimate Left/right movement
1 orthDistance HandX - RootX
2 rawHorizontal orthDistance InitDistOrth 3 if rawHorizontal > minDeltaOrth then
4 scaledHorizontal rawHorizontal / HandRangeOrth 5 else
6 scaledHorizontal 0
7 OrthoSpeed OrthoVector playerMaxS peed scaledHorizontal
The horizontal distance between Hand and translation matrix HomoMat is calculated by and Root orthDistance is obtained and the coplanar POSIT algorithm. After extract yaw,
dierence between current pose and initial pose rawHorizontal is calculated by subtract current orthDistance to InitDistOrth. If rawHorizontal is greater than minimum distance minDeltaOrth, rawHorizontal is converted into range [0;1] and assigned to scaledHorizontal. Otherwise, scaledHorizontal is 0. After that, the speed vector of left movement is calculated.
Player rotates the wrist to control character rotation. The marker is rotated follow the hand. By using Coplanar POSIT algorithm , the estimated angle can be inferred into 3 axis angles (ie. yaw, pitch, roll). The estimated roll angle is used to rotate left, right, and the pitch angle decide to rotate player up or down. After that, each frame add amount of angle to user’s quaternion to change the rotation angle of player (Algorithm 3).
pitch, roll angle, the markerState is checked. If marker is in FRONT state and roll angle is
greater than minimum angle to rotate minAngleX, axisX is obtained by normalize roll angle into range [0;1]. If marker is in LEFT, RIGHT state,
the axisX is respectively 1 and 1. The up/down rotation is measured by using pitch angle instead of yaw angle because of the instability of yaw angle estimation. If pitch angle is too small then player does not rotate. Because the value of pitch is the same when marker rotate up and down, the value of roll and yaw is checked to infer whether player want to rotate up or down. If roll and yaw less than 90, player wants to rotate up; and otherwise, player rotates down. After normalized axisX and axisY, the rotation speed of each frame angleX and angleY are calculated.
Firstly, the left/right rotation is measured. To prove the eectiveness of the proposed By tracking marker, the marker rotation state method, we experimented on several routes markerState is estimated. The estimated rotation (Fig. 3). These routes are in the shapes of lines,
28 T.C. Ma, M.D. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30
Algorithm 3: Estimate rotation
1 axisX 0, axisY 0
2 markerState Extract(markerData) % detect marker on thumb ﬁgure %
3 HomoMat CoPOSIT(source;des) % extract homography matrix %
4 (yaw; pitch;roll) Extract(HomoMat) % determine yaw, pitch or roll %
5 if markerState==FRONT then
6 if Abs(roll) > minAngleX then 7 axisX roll=maxAngleX 8 else
Fig. 3: A test map.
9 axisX 0
11 if markerState==LEFT then 12 axisX 1
14 axisX 1
15 pitch pitch+90
16 if pitch < minAngleY then 17 axisY 0
19 if roll < 90 and yaw < 90 then
20 axisY Abs(pitch)=maxAngleY 21 else
22 axisY Abs(pitch)=maxAngleY
Fig. 4: A test.
backward-left, etc. According to table 2, the accuaracy of up/down rotation is lowest because of shadow and errors of estimation algorithm.
Each test is deﬁned by a sequence of
movements and rotations. An error is deﬁned
23 angleX 24 angleY
axisX damplingTimeFrame axisY damplingTimeFrame
as follow: ”when user sends control a signal, the responding result is not corrected or the responding time is over a time threshold, then
an error is occurred”. In order to calculate
triangulations, rectangles, hexagons. Each route estimatedminimumnumberofmovementactions (ie. forward, backward, left, right) and rotation (ie. left/right, up/down) actions to complete each test (Fig. 4). We compare the time to complete test of this method to keyboard/mouse method (Table1). Though,ourproposal’sruntimehasnot reached those of using keyboard/mouse. In small games including mainly character’s movement, our runtime is acceptable. The proposed method works well with simple movement action, such as forward, backward, left, right; however, the accuracy of movement is decrease a bit with
the accuracy, we experimented each test several (around 20) times. The experimenters do the sequence of actions as deﬁnition of the test in order to reach the ﬁnal destination. For each sending signal, if the responding movement/rotation is incorrect or the responding time is too long (the time threshold is 1 second), then it is counted as an error. The accuracy is calculated by the percentage of the number of corrected actions overall the number of all actions,wherenumberofcorrectedactionsequals to the subtraction between the number of actions and the number of errors.
We applied Kinect based character navigation
compound movement actions like forward-left, in Maze Game and Haunted House Game. In
T.C. Ma, M.D. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30 29
Maze, we navigate character ﬁnding his way to some realtime games such as ﬁnding short route get destination. In Haunted House Game, we in Mazes, ﬁnding way to escape the haunted navigate character escaping from a ghost’s chase. house.
In the near future, we study deeply recognition
Table 1: Comparison of detection time reductions
to enhance the accuracy of rotation, especially the accuracy
Movement Test Rotate Left/Rightt Rotate Up/Down Triangle
Movement / Rotation 12 / 0
0 / 2
0 / 2
2 / 2 4 / 4 8 / 8
Time (keyboard/ our method) 55:3s=38:2s
44:8s=21:1s 25:6s=16:5s 34:0s=20:1s
of rotate up/down. We also think about how to improve running time to apply our method in many kinds of VR games. Beside that, more gestures can be deﬁned to control not only the navigation but also game actions (ex. jumping, shooting, interacting, etc.) by using hand ﬁnger. Moreover, by using Kinect, the action in game can be deﬁned by body action; this makes the games more interactive and interesting.
This work has been supported by VNU
Table 2: Accuracy of character navigation
University of Engineering and Technology, under Project No. CN.15.02
Route name Movement Rotation Error
Movement 12 0 3% Test
Rotate 0 2 2% Left/Rightt
Rotate 0 2 30% Up/Down
Triangle 2 2 4% Rectangular 4 4 2% Hexagons 8 8 3%
In the paper, we proposed a technique to navigate characters by hand actions in VR game. A Kinect was used to obtain the hand actions. In this technique, we also suggested using the marker attached to the player’s hand to increase the accuracy of hand action recognition. So, we not only take advantage of low-cost device but also enhance ﬂexibility for players. accuracy of straightmovementisupto98%. Rotateleft/right test also gives high accuracy, but rotate up/down have low accuracy. We deploy our solution in
 D. C. (series), Mtv games, Microsoft Game Studios, 2011.
 Kinect adventures, . Microsoft Game Studios, 2010.  http://www.metacritic.com/game/xbox-360/angry-
 A. M. L. D. C. V. L. M. Khademi, H. Mousavi Hondori, S. C. Cramer, Comparing direct and indirect interaction in stroke rehabilitation, .in Proceedings of the Extended Abstracts of the 32nd Annual ACM Conference on Human Factors in Computing Systems 2014 1639–1644.
 L. D. S. C. M. Khademi, H. M. Hondori, C. V. Lopes, Comparing ’pick and place’ task in spatial augmentedrealityversusnon-immersivevirtualreality for rehabilitation setting, .inAnnual International Conference of the IEEE Engineering in Medicine and Biology Society 2013 4613–4616.
 H. M. Hondori, M. Khademi., A review on technical and clinical impact of microsoft kinect on physicaltherapyandrehabilitation,JournalofMedical Engineering 2014.
 W. L. Hongliang Ren, A. Lim, Marker-based surgical instrument tracking using dual kinect sensors, IEEE transactions on automation science and engineering. 2014.
30 T.C. Ma, M.D. Hoang / VNU Journal of Science: Comp. Science & Com. Eng., Vol. 32, No. 2 (2016) 23–30
 T. Sharp, et. al., Accurate, robust, and ﬂexible real-time hand tracking, Best of CHI Honorable Mention Award 2012.
 M. Tang, Recognizing hand gestures with microsoft’s kinect, CS228 2011.
 Y. Li, Hand gesture recogniton using kinect, Master thesis 2012.
 Z. Zhang, A ﬂexible new technique for camera calibration,IEEETransactionsonPatternAnalysisand Machine Intelligence 22 (2000) 1330–1334.
 P. F. Sturm, S. J. Maybank, On plane-based camera calibration: A general algorithm, singularities, applications, International Conference on Computer Vision 1999.
 J. Heikkila, O. Silven, A four-step camera calibration procedure with implicit image correction, CVPR,
IEEE Computer Society 1997 1106–1112.