A QR Code-Based Approach to Differentiating the Display of Augmented Reality Content

Abstract

The augmented reality (AR) system requires markers to recognize and locate virtual objects on the screens of mobile devices. However, both markers and objects must be registered via the online platform in advance. In addition, an AR marker can only pair with a fixed set of virtual objects, limiting the flexibility and immediacy of changing and updating these data. This paper incorporates the quick response barcode (QR code) into the AR system to address these issues. We propose an algorithm with two vital goals, including (1) generating differentiated virtual objects for different target users by using only one QR code as the marker and (2) concealing different private authentication in QR modules by applying the error correction capability. We then demonstrate the proposed approach via a simulation of two practical scenarios, the electronic catalogs for business applications, and differentiated instructional materials for digital learning. This paper contributes to AR and QR code research and practices.

1. Introduction

The rapid advance in the hardware and software of smart devices has benefited the development of augmented reality (AR) technologies. In recent years, AR can operate efficiently without heavy and expensive equipment in different scenarios, such as educational training, digital learning, product marketing, and industrial design [1,2,3]. Furthermore, by taking AR’s capability of associating virtual objects with real space, many studies have applied it to spatial visualization [4]. Examples include combining the localization technique to improve interactive indoor navigation [5], helping people with limited peripheral vision identify things in the environment [6], and enhancing rehabilitation training for older people by providing a visualized guide [7].

In short, AR enables users to see virtual objects and models on their mobile phones. Most AR technologies are marker-based, meaning marker recognition is used to locate the relative position of 3D virtual models on the screen. Common development tools include ARToolkit, NyARToolKit, and AR Tag [8,9,10], and markers have different forms, such as frame markers, dot markers, split markers, data Matrix markers, ID markers, and template markers. These markers are usually simple squares, text, or patterns in black and white or grayscale, with the advantage of allowing the system to recognize patterns accurately and quickly when the light is dark or the camera pixels are insufficient.

While AR is convenient, it has several limitations [4,11,12,13] as follows:

• The marker and the virtual object must be registered and linked in advance on the online platform to track the relative position and direction between the camera and the tag, download the linked virtual object, and display it. In this case, the marker and the object cannot change dynamically, significantly limiting the effectiveness and immediacy.
• Keeping the internet connection is necessary to access the AR system and download virtual objects in real-time.
• The content, card size, image feature points, or print size of markers must meet certain conditions to maintain the system’s stability.
• Different companies have marker patterns that are not interchangeable, making it impossible to share sources and enable compatibility.

As shown in recent studies [5,14], the trend of AR would provide real-time computed and flexible media and content to let users interact with various virtual objects. Therefore, it is vital to resolve the limitations of the marker-based AR technique.

Combining AR with the quick response barcode (QR code) can be a gateway to address the above issues. The QR code is a two-dimensional matrix developed by Denso-Wave company in 1994 [15,16]. Based on this technique, the data content of a QR tag can be encoded and decoded efficiently. Therefore, more and more research has tried to combine mobile AR with QR codes [17,18]. For example, Ken et al. [17] applied QR code as an AR-based marker to track the angle of the QR Code and its positioning coordinates and mapped 3D virtual objects to the QR marker. VisionLens [18] also utilized a similar concept, establishing a QR Code for each product. After scanning the QR code, the AR virtual object and 3D animation will appear when mobile devices point at the product. In brief, the data content of the QR code can be a link to a virtual object or a uniform resource locator (URL). Therefore, when triggering the AR system, the location information of the QR marker can be estimated immediately and decode the data content of the URL to retrieve 3D virtual objects. Moreover, this process does not require users’ pre-registration of the AR system.

Unfortunately, current AR schemes [17,18,19,20,21,22] align the same virtual object with the same marker. The AR system thus considers all users with the same maker the same role. In other words, the AR system can only display the same virtual content to various users. However, companies often need to deliver different information to different target customers in real-world business applications. Accordingly, we propose a new AR mechanism to present differentiated virtual objects for target users by using the same QR marker to deal with this issue [23]. This approach is practical and can meet different scenarios. For example, companies can display an ordinary virtual object to general customers and deliver advanced models such as recommended products or preferences to the jointed member or VIP. In addition, the proposed approach can also provide differentiated instruction for educational purposes, such as offering students of different levels different learning materials. Therefore, this paper develops a flexible and differentiated AR system.

2. Design of Differentiated AR System Based on QR Markers

Figure 1 presents a conceptual model of our approach. To show the differentiated 3D virtual objects in the AR system, we develop a secret QR tag generation procedure in Section 2.1. Then we describe how the QR tag enables AR recognition and the presentation of specific virtual objects in Section 2.2.

2.1. Secret QR Tag Generation

We designed a secret QR tag concealment algorithm to derive the marked QR tag by applying the concept of the steganography technique [23,24,25]. Let Q be a given QR tag generated by a barcode generator. Here, the data content of Q can be a link or a uniform resource locator (URL) of the general virtual object. Public users can receive the ordinary virtual object from the QR tag.

According to the QR barcode standard, let n be the number of QR data and error correction modules, as the red region in Figure 2. To distinguish the general user and the specific users, let there are m classifications of members, ID_i and s_i be the identification number and the authentication code of the i-th member classification, 1 ≤ i ≤ m. Here, the value of s_i $\in$ {0, 1} and the length of s_i is $\lceil {\log }_{2}m\rceil$.

The stream c_i corresponding to ID_i thereby can be generated by

c_i = H_K(ID_i) mod n.

(1)

Here, K is the secret key of the system/dealer, and H_K(^•) is the one-way hash function with the key K. The stream length of c_i is $\lceil {\log }_{2}m\rceil$. The c_i indicates the concealed positions of the QR modules for the i-th member. Note that the generated values of c_i are limited within n, and the values in c_i cannot be selected more than once to avoid overlapping.

Assume q_i are the selected modules in Q according to the assigned positions c_i. The authentication codes s_i for differential members thereby are concealed into the corresponding q_i modules by

q_i′ = q_i⊕s_i,

(2)

where q_i′ are labeled as the modified modules. Note that, to conceal s_i into the QR tag without degrading the readability of the content, the value of m should be restricted,

$$m\times \lceil {\log }_{2}m\rceil \le \lfloor EC/2\rfloor \times 8.$$

(3)

Here, EC be the number of error correction codewords of the given Q.

With the concealment process, the embedded QR tag Q′ consequently can be derived. Figure 3 demonstrates the result of the marked QR tag generation procedure for eight classifications of members (i.e., m = 8). The original QR tag and the marked QR tag are depicted in Figure 3a,b, respectively. The red modules in Figure 3c show the difference between Figure 3a,b.

2.2. QR Tag-Based AR Procedure

To recognize and locate the virtual objects upon the marked QR tag Q′ [17,24], the position detection patterns of the QR tag (the black and white pattern in Figure 4a) is firstly registered as the marker in ARToolKit [13]. The camera capture can detect the three-position detection patterns A₀, B₀, and C₀, as depicted in Figure 4b. To find the fourth corner, we apply the corner-based manner to find D₀ according to A₀, B₀, and C₀. Figure 4c shows the estimated fourth corner [24].

With the four corners, the camera pose can be estimated by exploiting homography between the captured image and the established feature map of Q′. Finally, we can estimate the rotation degree R and the translation T between camera and Q′.

With the marked Q′, the camera pose, such as the rotation matrix R and the translation T, can be calculated. According to the QR content of Q′, the normal 3D virtual object can be rendered on the scene by transforming the 3D virtual object with R and T. Moreover, the specific member with their member number ID_i can further extract the corresponding authentication code by the system key K automatically. The system thereby can link and render the specific 3D virtual objects for differential members.

3. Demonstration of the Proposed Approach

To demonstrate the proposed method, we evaluate the reliability by scanning the marked QR tag in different lighting conditions with different devices and software in Section 3.1 and show the usability by turning our algorithm into practical systems for meeting different scenarios in Section 3.2.

3.1. Reliability Evaluation

Different mobile devices, scanning software, and environmental illumination may influence the accuracy of QR content decoding. Hence, we assess the feasibility of our system via three different smartphones (two in the Android system and one in the iOS system) and four types of scanning software under three different lighting conditions.

In the scanning tests, we evaluate QR codes displayed both on the screen and the printed paper using two QR tags generated by our method (see Figure 5).

Table 1. Scanning tests of the Android system under different lighting conditions.

Mobile Device	HTC			Samsung
Light Intensity (lux)	Outdoor	Indoor		Outdoor	Indoor
	305	221	21	305	221	21
Media Type of QR Code	Print/ Screen	Print/ Screen	Print/ Screen	Print/ Screen	Print/ Screen	Print/ Screen
QR Code & Barcode Scanner	✓	✓	✓	✓	✓	✓
QR Code Reader	✓	✓	✓	✓	✓	✓
QR Code Reader-Scanner App	✓	✓	✓	✓	✓	✓
Scandit Barcode Scanner Demo	✓	✓	✓	✓	✓	✓

shows the scanning results via two phones of the Android system (HTC: Desire 20 Pro and Samsung: Note 20), while

Table 2. Scanning tests of the iOS system under different lighting conditions.

Light Intensity (lux)	Outdoor	Indoor
	305	221	21
Media Type of QR Code	Print/Screen	Print/Screen	Print/Screen
QR Code & Barcode Scanner	✓	✓	✓
QR Code	✓	✓	✓
QR Square	✓	✓	✓
Scandit Barcode Scanner Demo	✓	✓	✓

shows the results of the iOS system (iPhone 11 Pro MAX). We set three lighting conditions: the outdoor, the indoor with living room lights, and the indoor without artificial lights. The illuminance values captured in this test are lux = 305, 221, and 21, respectively. As shown, both two QR codes can be decoded by four types of scanning software under three sets of environmental illumination.

3.2. Simulation Results

We conducted simulations to demonstrate the usability of our system. Figure 6 and Figure 7 are our simulation results in different scenarios. This simulation uses Vuforia Developer [27] to build the AR content with the processed QR code (Figure 5b). After installing the designed AR app, users can obtain these 3D objects by scanning the QR code.

Figure 6 presents a fashion design store that provides differentiated electronic catalogs based on clients’ attributes via the same QR code. The left side of Figure 6 shows the augmented reality results that general users see when scanning the QR code. The right side of Figure 6 offers a recommended product in addition to the standard one for specific users.

In our system, when a user installs the app, he/she will get an identification (i.e., ID indicated in Section 2.1) and an authentication code (i.e., s_i) in his/her mobile device. According to the ID, the system K can automatically calculate where the s_i information should hide in the QR code. That is, in Figure 6, general users’ and specific users’ authentication information have been hidden in different positions of the QR code, respectively. The app will trigger corresponding 3D objects only when the ID matches the authentication code.

On the other hand, Figure 7 shows electronic materials for digital learning. These materials are for language learning for children. The left side of Figure 7 has only one simple model, which lets beginners recognize it quickly. Comparatively, the right side presents a complicated set of models, requiring advanced learners to identify each of them. Accordingly, with one QR code, different users can receive what they need, thus saving costs and easing the management of numerous images.

4. Discussions

The proposed system has combined confidential QR codes and AR. Hence, when users scan the marker, they are no longer limited in getting the same and fixed virtual objects. Besides, the system can present various AR effects according to different user groups or preferences.

Past studies [11,12,13] focus on improving the recognition of AR without using markers. However, these approaches still require pre-registration to match the image features and virtual objects. Furthermore, these 3D objects remain fixed and cannot update promptly after the registration. The stability and accuracy of scanning are also lower than marker-based techniques. That is why we combine the QR code-based approach with the AR system to break these limitations.

Several studies [17,18,19] have explored similar issues. However, their emphasis is on recognizing QR markers, especially positioning the AR on QR codes after calculating the coordinates. We build our theory upon these studies, but we do not just treat QR codes as markers. Instead, we further incorporate the authentication mechanism to propose the differentiated display of AR content, which is the significant contribution of this study.

Finally, research [21,22] uses multi-markers to provide different AR content in different environments. Compared to their results, our system only requires one QR code to generate various virtual objects, thus saving costs on producing extra QR markers.

5. Conclusions and Future Research

This applied research is practical and could be widely used in many cases. For example, general users can scan the original QR content data to download AR virtual objects. Companies can update these objects anytime. Specific groups or members can also retrieve different authentication codes to generate specific virtual objects. In addition, the proposed algorithm complies with the QR Code compilation and decoding specifications. Thus, users can scan it with general barcode devices without additional assistance from the computer image processing. In addition to maintaining the original readability, the proposed approach might also benefit confidentiality, security, and visual design of QR code information since it can hide data in an ordinary image. We expect this technical research can contribute to AR research and practical business or education applications.

There are several issues researchers can explore in the future. For example, since our approach provides differentiated AR content to different user groups, it requires more capacity to embed more authentication codes when user groups increase. However, higher QR code versions and more giant QR markers are usually needed to hide more information. Thus, it is worth studying how to increase the capacity of the authentication length within the general-sized QR codes. On the other hand, researchers could apply the technique of embedding authentication codes into general images in place of QR codes, providing another medium for differentiated displays.