The Semantic Meaning of Hand Shapes and Z-Dimension Movements of Freehand Distal Pointing on Large Displays
Abstract
:1. Introduction
2. Related Work
2.1. Manipulation and Semaphores
2.2. Semaphores in Freehand Pointing
2.3. Manipulation in Remote Pointing
2.4. Z-Dimension Movements and Orientation
3. Experiment 1: Hand Shape in Manipulation
3.1. Participants
3.2. Apparatus
3.3. Design
- Target width: 10 mm, 100 mm, 250 mm, 400 mm;
- Amplitude: 20 mm, 200 mm, 400 mm.
3.4. Task and Procedure
3.5. Results
3.6. Discussions of Experiment 1
4. Experiment 2: Z-Dimension Movements in Freehand Pointing
4.1. Participants
4.2. Apparatus
4.3. Design
- Target width: 20 mm, 60 mm, 100 mm;
- Target density (i.e., the width between targets): 1 mm, 20 mm, 40 mm;
- Orientation: right, right-up, up, left-up, left, left-down, down, and right-down.
4.4. Task and Procedure
4.5. Result
4.6. Discussion of Experiment 2
5. Design Suggestions
- Unlike traditional input modalities like mouse and touch, freehand has a more powerful input bandwidth. We can express more details in a command through the semantic meaning of gestures. For distal freehand pointing, users can use different hand shapes and the Z-dimension movement to change the size and CD gains of the cursor, thereby improving the freehand pointing efficiency.
- Users prefer to complete a low-precision task with the whole hand, while using the finger in high-precision tasks on large displays from a remote distance. For distal freehand pointing, the perceptual precision of the open hand gesture and index finger gesture is different. Users can use the open hand to control an area cursor in coarse selections, while moving a pointer cursor through the index finger in high-precision selections.
- Users tended to stretch their hand to reduce the distance between the large display and themselves in high-precision tasks, especially in small target selection. Thus, one way to use hand motion in the Z-dimension is by mapping it to the precision of an interactive tool. For example, when people moved their hand forward in pointing, it may have decreased the CD gain and size of the area cursor, thereby helping them select small targets easier.
- Orientation has a significant effect on hand motion in the Z-dimension (Figure 6). We can use this relationship to predict the Z-dimension movement distance when moving in different directions. That is, when using the Z-dimension movement distance as a command attribute, the orientation has to be taken into account.
- In dragging tasks, it was found that some people preferred to keep the initial hand shape unchanged in manipulation due to the legacy of mouse interaction. Thus, designers had to be careful when using gestures to increase the expressiveness in the interaction. It is important to keep the gestures simple and easy to understand. Too complicated gestures can be confusing.
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Quek, F.; McNeill, D.; Bryll, R.; Duncan, S.; Ma, X.-F.; Kirbas, C.; Ansari, R. Multimodal human discourse: Gesture and speech. ACM Trans. Comput. Hum. Interact. 2002, 9, 171–193. [Google Scholar] [CrossRef]
- Kopper, R.; Bowman, D.A.; Silva, M.G.; McMahan, R.P. A human motor behavior model for distal pointing tasks. Int. J. Hum. Comput. Stud. 2010, 68, 603–615. [Google Scholar] [CrossRef]
- Nancel, M.; Pietriga, E.; Chapuis, O.; Beaudouin-Lafon, M. Mid-Air Pointing on Ultra-Walls. ACM Trans. Comput. Hum. Interact. 2015, 22, 1–62. [Google Scholar] [CrossRef]
- Tse, E.; Hancock, M.; Greenberg, S. Speech-filtered bubble ray: Improving target acquisition on display walls. In Proceedings of the 9th International Conference on Multimodal Interfaces, Nagoya, Aichi, Japan, 12–15 November 2007; pp. 307–314. [Google Scholar]
- Jota, R.; Pereira, J.M.; Jorge, J.A. A comparative study of interaction metaphors for large-scale displays. In Proceedings of the CHI ’09 Extended Abstracts on Human Factors in Computing Systems, Boston, MA, USA, 4–9 April 2009. [Google Scholar]
- Vogel, D.; Balakrishnan, R. Distant freehand pointing and clicking on very large, high resolution displays. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, Seattle, WA, USA, 23–27 October 2005. [Google Scholar]
- Mäkelä, V.; Heimonen, T.; Turunen, M. Magnetic Cursor: Improving Target Selection in Freehand Pointing Interfaces. In Proceedings of the International Symposium on Pervasive Displays, Copenhagen, Denmark, 3–4 June 2014. [Google Scholar]
- Casiez, G.; Roussel, N.; Vogel, D. 1€ Filter: A simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, TX, USA, 5–10 May 2012. [Google Scholar]
- Lou, X.; Peng, R.; Hansen, P.; Li, X.A. Effects of User’s Hand Orientation and Spatial Movements on Free Hand Interactions with Large Displays. Int. J. Hum. Comput. Interact. 2017, 34, 519–532. [Google Scholar] [CrossRef]
- Lin, J.; Harris-Adamson, C.; Rempel, D. The Design of Hand Gestures for Selecting Virtual Objects. Int. J. Hum. Comput. Interact. 2019, 35, 1729–1735. [Google Scholar] [CrossRef]
- Tian, F.; Lyu, F.; Zhang, X.; Ren, X.; Wang, H. An Empirical Study on the Interaction Capability of Arm Stretching. Int. J. Hum. Comput. Interact. 2017, 33, 565–575. [Google Scholar] [CrossRef]
- Goth, G. Brave NUI world. Commun. ACM 2011, 54, 14–16. [Google Scholar] [CrossRef]
- Morrel-Samuels, P. Clarifying the distinction between lexical and gestural commands. Int. J. Man Mach. Stud. 1990, 32, 581–590. [Google Scholar] [CrossRef]
- Nielsen, M.; Störring, M.; Moeslund, T.B.; Granum, E. A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. In Proceedings of the 5th International Gesture Workshop (GW 2003), Genova, Italy, 15–17 April 2003; Camurri, A., Volpe, G., Eds.; Springer: Berlin/Heidelberg, Germany, 2004; pp. 409–420. [Google Scholar]
- Wobbrock, J.O.; Aung, H.H.; Rothrock, B.; Myers, B.A. Maximizing the guessability of symbolic input. In Proceedings of the CHI ’05 Extended Abstracts on Human Factors in Computing Systems, Portland, OR, USA, 2 April 2005. [Google Scholar]
- Wobbrock, J.O.; Morris, M.R.; Wilson, A.D. User-defined gestures for surface computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA, 4 April 2009. [Google Scholar]
- Dong, H.W.; Danesh, A.; Figueroa, N.; El Saddik, A. An Elicitation Study on Gesture Preferences and Memorability toward a Practical Hand-Gesture Vocabulary for Smart Televisions. IEEE Access 2015, 3, 543–555. [Google Scholar] [CrossRef]
- Chen, Z.; Ma, X.C.; Zhou, Y.; Yao, M.G.; Ma, Z.; Wang, C.; Shen, M.W. User-defined gestures for gestural interaction: Extending from hands to other body parts. Int. J. Hum. Comput. Interact. 2018, 34, 238–250. [Google Scholar] [CrossRef]
- Wu, H.; Yang, L. User-Defined Gestures for Dual-Screen Mobile Interaction. Int. J. Hum. Comput. Interact. 2019, 1–15. [Google Scholar] [CrossRef]
- Norman, D.A. Natural user interfaces are not natural. Interactions 2010, 17, 6–10. [Google Scholar] [CrossRef]
- Chen, X.A.; Schwarz, J.; Harrison, C.; Mankoff, J.; Hudson, S.E. Air + touch: Interweaving touch & in-air gestures. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, Honolulu, HI, USA, 5–8 October 2014. [Google Scholar]
- Alkemade, R.; Verbeek, F.J.; Lukosch, S.G. On the Efficiency of a VR Hand Gesture-Based Interface for 3D Object Manipulations in Conceptual Design. Int. J. Hum. Comput. Interact. 2017, 33, 882–901. [Google Scholar] [CrossRef]
- Matulic, F.; Vogel, D. Multiray: Multi-Finger Raycasting for Large Displays. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018. [Google Scholar]
- Rempel, D.; Camilleri, M.J.; Lee, D.L. The Design of Hand Gestures for Human-Computer Interaction: Lessons from Sign Language Interpreters. Int. J. Hum. Comput. Stud. 2015, 72, 728–735. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- McNeil, D. Hand and Mind: What Gestures Reveal about Thought; University of Chicago Press: Chicago, IL, USA, 1992. [Google Scholar]
- Cochet, H.; Vauclair, J. Deictic gestures and symbolic gestures produced by adults in an experimental context: Hand shapes and hand preferences. Later. Asymmetries Body Brain Cognit. 2014, 19, 278–301. [Google Scholar] [CrossRef] [Green Version]
- Wilkins, D. Why pointing with the index finger is not a universal (in sociocultural and semiotic terms). In Pointing: Where Language, Culture, and Cognition Meet; Kita, S., Ed.; Erlbaum: Mahwah, NJ, USA, 2003; pp. 171–215. [Google Scholar]
- Foehrenbach, S.; König, W.A.; Gerken, J.; Reiterer, H. Natural Interaction with Hand Gestures and Tactile Feedback for large, high-res Displays. In Proceedings of the Workshop on Multimodal Interaction through Haptic Feedback (MITH 08), held in Conjunction with International Working Conference on Advanced Visual Interfaces (AVI 08), Napoli, Italy, 28–30 May 2008. [Google Scholar]
- Haque, F.; Nancel, M.; Vogel, D. Myopoint: Pointing and Clicking Using Forearm Mounted Electromyography and Inertial Motion Sensors. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea, 18–23 April 2015. [Google Scholar]
- Sambrooks, L.; Wilkinson, B. Comparison of gestural, touch, and mouse interaction with Fitts’ law. In Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, Adelaide, Australia, 25 November 2013. [Google Scholar]
- Jude, A.; Poor, G.M.; Guinness, D. Personal space: User defined gesture space for GUI interaction. In Proceedings of the CHI ’14 Extended Abstracts on Human Factors in Computing Systems, Toronto, ON, Canada, 26 April–1 May 2014. [Google Scholar]
- Jota, R.; Nacenta, M.A.; Jorge, J.A.; Carpendale, S.; Greenberg, S. A comparison of ray pointing techniques for very large displays. In Proceedings of the Graphics Interface 2010, Ottawa, ON, Canada, 31 May–2 June 2010. [Google Scholar]
- Baloup, M.; Oudjail, V.; Pietrzak, T.; Casiez, G. Pointing techniques for distant targets in virtual reality. In Proceedings of the 30th Conference on l’Interaction Homme-Machine, Brest, France, 23–26 October 2018. [Google Scholar] [CrossRef] [Green Version]
- Bateman, S.; Mandryk, R.L.; Gutwin, C.; Xiao, R. Analysis and comparison of target assistance techniques for relative ray-cast pointing. Int. J. Hum. Comput. Stud. 2013, 71, 511–532. [Google Scholar] [CrossRef]
- Siddhpuria, S.; Malacria, S.; Nancel, M.; Lank, E. Pointing at a Distance with Everyday Smart Devices. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018. [Google Scholar] [CrossRef] [Green Version]
- Debarba, H.; Nedel, L.; Maciel, A. LOP-cursor: Fast and precise interaction with tiled displays using one hand and levels of precision. In Proceedings of the 2012 IEEE Symposium on 3D User Interfaces (3DUI), Costa Mesa, CA, USA, 4–5 March 2012. [Google Scholar]
- Nancel, M.; Chapuis, O.; Pietriga, E.; Yang, X.-D.; Irani, P.P.; Beaudouin-Lafon, M. High-precision pointing on large wall displays using small handheld devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, 27 April–2 May 2013. [Google Scholar]
- Endo, Y.; Fujita, D.; Komuro, T. Distant Pointing User Interfaces based on 3D Hand Pointing Recognition. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces, Brighton, UK, 17–20 October 2017. [Google Scholar] [CrossRef]
- Petford, J.; Nacenta, M.A.; Gutwin, C. Pointing All Around You: Selection Performance of Mouse and Ray-Cast Pointing in Full-Coverage Displays. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018. [Google Scholar] [CrossRef] [Green Version]
- Fitts, P.M. The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 1954, 47, 381–391. [Google Scholar] [CrossRef] [Green Version]
- Cha, Y.; Myung, R. Extended Fitts’ law for 3D pointing tasks using 3D target arrangements. Int. J. Ind. Ergon. 2013, 43, 350–355. [Google Scholar] [CrossRef]
- Murata, A.; Iwase, H. Extending Fitts’ law to a three-dimensional pointing task. Hum. Mov. Sci. 2001, 20, 791–805. [Google Scholar] [CrossRef] [Green Version]
- Burno, R.A.; Wu, B.; Doherty, R.; Colett, H.; Elnaggar, R. Applying Fitts’ Law to Gesture Based Computer Interactions. Procedia Manuf. 2015, 3 (Suppl. C), 4342–4349. [Google Scholar] [CrossRef] [Green Version]
- Jones, K.S.; McIntyre, T.J.; Harris, D.J. Leap Motion- and Mouse-Based Target Selection: Productivity, Perceived Comfort and Fatigue, User Preference, and Perceived Usability. Int. J. Hum. Comput. Interact. 2019, 1–10. [Google Scholar] [CrossRef]
- Ballendat, T.; Marquardt, N.; Greenberg, S. Proxemic interaction: Designing for a proximity and orientation-aware environment. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, Saarbrücken, Germany, 7–10 November 2010. [Google Scholar]
- Vogel, D.; Balakrishnan, R. Interactive public ambient displays: Transitioning from implicit to explicit, public to personal, interaction with multiple users. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, Santa Fe, NM, USA, 24–27 October 2004. [Google Scholar]
- Harrison, C.; Dey, A.K. Lean and zoom: Proximity-aware user interface and content magnification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy, 5–10 April 2008. [Google Scholar]
- Clark, A.; Dünser, A.; Billinghurst, M.; Piumsomboon, T.; Altimira, D. Seamless interaction in space. In Proceedings of the 23rd Australian Computer-Human Interaction Conference, Canberra, Australia, 28 November–2 December 2011. [Google Scholar]
- Li, A.X.; Lou, X.L.; Hansen, P.; Peng, R. On the Influence of Distance in the Interaction with Large Displays. J. Disp. Technol. 2016, 12, 840–850. [Google Scholar] [CrossRef]
- Ren, G.; O’ Neill, E. 3D selection with freehand gesture. Comput. Gr. 2013, 37, 101–120. [Google Scholar] [CrossRef]
- Lubos, P.; Bruder, G.; Ariza, O.; Steinicke, F. Touching the Sphere: Leveraging Joint-Centered Kinespheres for Spatial User Interaction. In Proceedings of the 2016 Symposium on Spatial User Interaction, Tokyo, Japan, 15–16 October 2016. [Google Scholar]
- Guinness, D.; Jude, A.; Poor, G.M.; Dover, A. Models for Rested Touchless Gestural Interaction. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction, Los Angeles, Ca, USA, 8–9 August 2015. [Google Scholar]
- Kim, H.; Oh, S.; Han, S.H.; Chung, M.K. Motion–Display Gain: A New Control—Display Mapping Reflecting Natural Human Pointing Gesture to Enhance Interaction with Large Displays at a Distance. Int. J. Hum. Comput. Interact. 2019, 35, 180–195. [Google Scholar] [CrossRef]
- Morris, M.R.; Danielescu, A.; Drucker, S.; Fisher, D.; Lee, B.; Schraefel, M.C.; Wobbrock, J.O. Reducing legacy bias in gesture elicitation studies. Interactions 2014, 21, 40–45. [Google Scholar] [CrossRef]
- Grossman, T.; Balakrishnan, R. The bubble cursor: Enhancing target acquisition by dynamic resizing of the cursor’s activation area. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Portland, OR, USA, 2 April 2005. [Google Scholar]
- Blanch, R.; Ortega, M. Benchmarking pointing techniques with distractors: Adding a density factor to Fitts’ pointing paradigm. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada, 7 May 2011. [Google Scholar]
Pointing Metaphor | Pointing Technique (As Defined in Each Study) |
---|---|
Relative grab | Gestural pointing [30] |
Magnetic cursor [7] | |
Personal space [31] | |
Absolute ray-casting | Absolute position finger ray casting [6] |
Bubble ray [4] | |
Laser pointing [32] | |
Three-dimensional (3D) bubble cursor [33] | |
Relative ray-casting | Relative ray-casting [34] |
GyroAcc [29] | |
Relative mouse | Relative pointing with clutching [6] |
Myopoint [29] | |
Pointer-acceleration (PA)-based pointing [9] | |
Mixed mode | Hybrid technique [6] |
Laser + gyro [3] |
Hand Shape | ||||
---|---|---|---|---|
Open Hand | Index Finger | Others | Total Number | |
Width: 10mm × Amplitude: 20 mm | 0% | 90% | 10% | 30 |
Width: 10mm × Amplitude: 200 mm | 0% | 90% | 10% | 30 |
Width: 10mm × Amplitude: 400 mm | 0% | 90% | 10% | 30 |
Width: 100mm × Amplitude: 20 mm | 10% | 70% | 20% | 30 |
Width: 100mm × Amplitude: 200 mm | 10% | 73.3% | 16.7% | 30 |
Width: 100mm × Amplitude: 400 mm | 20% | 63.3% | 16.7% | 30 |
Width: 250mm × Amplitude: 20 mm | 26.7% | 53.3% | 20% | 30 |
Width: 250mm × Amplitude: 200 mm | 30% | 53.3% | 16.7% | 30 |
Width: 250mm × Amplitude: 400 mm | 20% | 60% | 20% | 30 |
Width: 400mm × Amplitude: 20 mm | 43.3% | 43.3% | 13.3% | 30 |
Width: 400mm × Amplitude: 200 mm | 46.7% | 40% | 13.3% | 30 |
Width: 400mm × Amplitude: 400 mm | 46.7% | 40% | 13.3% | 30 |
Total | 21.1% | 63.9% | 15% | 360 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, C.-H.; Wang, J.-L. The Semantic Meaning of Hand Shapes and Z-Dimension Movements of Freehand Distal Pointing on Large Displays. Symmetry 2020, 12, 329. https://doi.org/10.3390/sym12030329
Chen C-H, Wang J-L. The Semantic Meaning of Hand Shapes and Z-Dimension Movements of Freehand Distal Pointing on Large Displays. Symmetry. 2020; 12(3):329. https://doi.org/10.3390/sym12030329
Chicago/Turabian StyleChen, Chien-Hsiung, and Jian-Li Wang. 2020. "The Semantic Meaning of Hand Shapes and Z-Dimension Movements of Freehand Distal Pointing on Large Displays" Symmetry 12, no. 3: 329. https://doi.org/10.3390/sym12030329