With the rapid development of computer technology, more and more information are used electronically and saved in digital formats. It is much easier to collect, store, transfer, and retrieve information in the digital world. However, physical paper documents still play a significant role in our daily life due to the ease of use, superior readability, and availability. Some people still prefer to read a printed document in many workplaces. When people read a paper document, they may need to know more information regarding a particular object, such as a text, a paragraph, or an image. Such information might be available in the digital format. Integrating information from paper documents and electronic devices allows readers to easily access information not included in a traditional paper document. To achieve this, paper-based augmented reality which overlays digital information on traditional paper is needed.
Augmented reality technique is the integration of digital information with physical real world environments. Using augmented reality, elements in the real world are augmented by computer-generated information such as sound, video, or digital images. For instance, when a user looks at a restaurant on a paper map, with the aid of augmented reality, he/she can also read reviews, the menus of the restaurant that are available digitally. To make connections between the object on a physical paper and its related digital information, we need spatial awareness techniques. Spatial awareness is the ability to make an object aware of itself in a real-world environment. Developing spatial awareness is to identify the location of an object in relation to its own body in space. Therefore, location information is vital to spatial awareness.
The development of spatial awareness and augmented reality has shifted from traditional computers to mobile devices. Mobile devices are getting more computational power and are becoming more popular. Different methods have been applied to support spatially aware, mobile interactions, such as marker-based methods [1
] and content-recognition based solutions [4
]. Marker-based approaches convert physical information to digital content, while content-recognition based techniques require high computing power to identify objects in real environments. Marker-based systems can work with a large paper sheet without using large hardware. This methodology, however, has a high requirement of computer capability and power. It is hard to provide dynamic and simultaneous results due to the complexity of the calculation. Reilly et al. [7
] developed a marker-based mobile system to combine paper maps with electronic information. This system uses a high performance, compact radio frequency identification (RFID) reader to recognize the RFID tags on the paper maps. The problem with this approach is that the user can only obtain static electronic information from each tag on a map. Content-recognition based techniques use cameras to identify objects in real environments also requires high computing capabilities and are not suitable for smartphones.
In an augmented paper system, a user can move a spatial-aware mobile device over a paper document and acquire the digital information based on the position of the user’s focus. However, the mobile device has no ability to directly access a physical paper document. Spatially aware mobile systems must rely on other hardware to achieve the function of recognizing a user’s focus when browsing a paper document. Lee [8
] uses the Wiimote to create Multi-point Interactive Whiteboards. His invention enhanced the information in a paper document with the tablet display, however, the system reduces the portability due to the large external Wiimote hardware.
To address some of the issues mentioned above, we design a new spatial aware system called the Ultrasonic PhoneLens. It is a low-cost, portable, and high-performance solution for spatial aware systems. The major functional requirement of the system is to provide a means for users to acquire digital information regarding the texts or images in a paper based on the location of the device. Nonfunctional requirements include usability, real-time performance, portability, and accuracy. We evaluated these requirements in the empirical evaluation in Section 4
, such as comfort level when browsing a paper document, and precise detection of real position.
Ultrasonic PhoneLens integrates two ultrasonic sound sensors, Arduino board, and a mobile device. The ultrasonic sound sensors assembled on an Arduino board provide reliable distance detection in the paper-based working environment (by sending sound waves to the wood barriers set up at the borders of the paper). Arduino transmits the distance as coordinates (i.e., the location over the paper) to a mobile device through Bluetooth. Based on the coordinates detected by the ultrasonic sound sensor, digital information corresponding to the text or the image in the paper will be displayed on the screen of mobile device.
The major contribution of this paper is that we design a low-cost, mobile-based, portable augmented paper system with spatial awareness capability. The movement-based interface in Ultrasonic PhoneLens allows the user to acquire digital information related to a paper sheet through hand movements. Ultrasonic PhoneLens combines hand movements with the traditional screen touch interaction on a mobile device. The traditional screen touch devices can only allow users to operate the system within the scope of the screen. The movement of the handheld mobile device over a paper document can enhance the system usability without the limitation of small-screen devices. To our knowledge, it is the first design of a touch-based mixed-media approach that utilizes ultrasonic technologies and smartphones. We also designed a multi-step data processing method to improve data accuracy.
To evaluate the performance of Ultrasonic PhoneLens, we conducted an empirical study on browsing a paper-based architecture plan. An architecture plan for a room includes multiple layers of information such as lighting layout, electrical layout, etc. Such information are overlapped and cannot be represented in a single paper sheet. With the help of a mobile spatial awareness system, the user can easily locate information about objects, such as circuit breakers, on the plan. We compared Ultrasonic PhoneLens with Wiimote PhoneLens [9
], the first generation of the system, which used a Wii remote and Infrared Light-Emitting Diode (LED) lights to augment a paper-based workplace with digital information.
The rest of the paper is organized as follows. Section 2
reviews related work. Details of our system are explained in Section 3
. Section 4
describes the empirical study. Results are analyzed in Section 5
. Section 6
discusses our future work.
2. Related Work
Numerous studies have been proposed to augment the paper document with electronic devices. Our study is related to areas of interactive paper systems, spatial aware computing, and augmented reality techniques.
Paper systems: Many interactive paper systems have been developed to combine the benefits of paper and digital media. The Anoto technology [10
] is a pen-based interaction system which can track handwriting on a physical paper and augment paper documents with digital information. Liao [11
] designed a pen-based command system for interactive paper. He proposed pen-top multimodal feedback which combines visual, tactile, and auditory feedback to support paper-computer interaction. Hotpaper [12
] aims at augmenting paper information with multimedia annotations (such as video or audio). The system can analyze the physical information which is a captured document patch image or video frame to identify the corresponding digital information such as electronic document, page number, and location on the page. Paper Composer [13
] is an interactive paper interface for music composition. This system supports composers’ expressions and explorations in a music book by computer-aided composition tools. S-Notebook [14
] is a mobile application that connects mobile devices with interactive paper using an Anoto pen. It allows users to add annotations or drawings to anchors in digital images without learning pre-defined pen gestures and commands. These paper systems combine traditional paper with digital information. Most of the approaches, however, utilized computers or special pens. Our approach improve the portability of interactive paper system using touch-based mobile devices.
Spatial aware computing: Spatial aware computing had been applied in interactive paper systems. MouseLight [15
] is one of the examples. A spatially aware projector is made with a mobile laser projector. It can detect the position of the digital pen and track the handwriting from the end user. This application, however, is a bimanual hardware. It is very hard for users to write and operate the system at the same time. In our paper, we are presenting a new spatial aware interactive system which can be operated by a single-handed device.
Augmented reality: Camera-based approach: Currently, camera-based augmented reality technique has been widely applied in the field of digital images and traditional paper document interaction. The SESIL [16
], an augmented reality environment for students' improved Learning, is a novel approach to setting up a digital environment to perform the recognition of physical book pages and of specific elements of interest within pages. The pages in books can be captured by a camera. The system can recognize images from the camera and produce an electronic page which can produces an interaction with actual books and pens/pencils. Jee et al. [17
] designed an electronic learning system which can allow users to read 3D virtual content from traditional textbooks. This system creates a 3D modeling environment based on the content of physical book. To improve the portability, integrated cameras in handheld devices have been adopted in the field of camera-based augmented reality. Hansen et al. [18
] used integrated cameras in mobile devices to address how mixed interaction spaces can have different identities, can be mapped to applications, and can be visualized. By applying image analysis algorithms to the camera pictures, movements or actions such as rotation and marking, can be determined. These camera based approaches require high computing power to identify objects in a real environment.
Marker-based approach: A primary challenge of augmented reality is how to align digital information with the real world. To address this, a marker-based approach using visual markers are proposed. In 1998, Masutani et al. [19
] constructed an augmented, reality-based visualization system to support intravascular neurosurgery and evaluate it in clinical environments. It augments the motion pictures from X-ray fluoroscopy with 3D virtual vascular models. This technique relies on the 3D registration fiducial markers. The data adaptive reprojection technique was introduced to evaluate the reliability of the displayed fluoroscopy. It predicts the number of wrong registrations around the registered objects. The results were compiled using synthetic data consisting of fiducial marker coordinates with 2-Dimensional (2D) or 3-Dimensional (3D) errors. It was perhaps the earliest software that utilized a marker-based approach for augmented reality. As smartphone technology has exploded, marker-based augmented reality has been revolutionized simultaneously. Built-in camera recognition and detection is a recent development which takes advantage of an internal mobile camera to track the markers in a real environment. Klemmer et al. [20
] used bar-codes as markers to augment paper transcripts with digital video interviews. The system uses CyberCodes reader [21
] to identify real objects by a mobile device. Rohs [22
] proposed using 2-dimensional graphical widgets to retrieve the relevant digital information by a camera phone. The widget is a generic, reusable, directly manipulated visual code which is suitable for printing on paper. Reilly [8
] used RFID tags to create a marker-based mobile system which combines paper maps with digital information. The RFID tags placed in a regular grid set the bottom of the paper map. Wen et al. [23
] studied an indoor tracking system with infrared projectors. The projector generates the infrared markers on the workplace and a user needs to wear a tracking camera which can capture the position and orientation of each infrared marker. The infrared projector has to be installed on the ground or a wall, which limits the mobility of users. To improve the portability, we utilized the ultrasonic sensors to detect the location of an object in its environment. It is a fast, inexpensive, and a more portable approach than the previous work. Based on the location information, the system can retrieve the digital information from the smartphone. The touch based screen of smartphone allow the users easily to interact with the digital information.