Augmented Reality in Food Promotion and Analysis: Review and Potentials

Abstract

The paper presents the current state of using augmented reality (AR) in the sectors of food analysis and food promotion through products and orders. Based on an extensive literature review, 34 indicative augmented reality applications of various purposes, target audiences and implementations have been selected and presented. Applications are research-based, commercial, or oriented just for entertainment. Eight classification criteria are defined, especially for these applications, and used for presenting them, including content, context, execution scenario, markers, devices supported, implementation details and appeals based on evaluation, downloads, or sales. Additionally, 16 implementation and supportive platforms that have been used in the presented applications are described. The paper discusses advantages and limitations of current applications leading to proposals of further use of augmented reality in these food sectors towards a uniform handling of all parameters related to food processing, from production until consumption. These parameters include content use, design considerations, implementation issues, use of AR markers, etc.

1. Introduction and Related Work

Augmented reality (AR) has now become a widespread technology that affects a lot of activities of our lives. In food sciences, AR is also present and already employed in sectors such as food safety; analysis of food ingredients; training operations in food production companies; promotion and sales of food products, either as sole products, or as meals through restaurants. In recent years, this trend has been accelerating, with reports from food-related companies, publishers, and software developers. In a Forbes report [1], the potentials of using AR in food sector were also explored, especially during food production and safety. Zealar [2] discusses existing and potential uses of AR in restaurants and on their menus, managing food inventories and nutritional information delivered to customers about a dish. Moreover, [3] examines how AR use has been accelerated in the food sector due to the COVID-19 pandemic. In addition, [4] explores the potentials of AR in food promotion, training, and customer support. Spoon magazine Marston [5] promotes the significance of 3D models for food that customers are about to order. Customers can have a preview of how food looks on their table and view it from multiple angles and zoom levels. Augray [6] enumerates potential AR uses in the food sector, such as the display of nutritional information and recipes about a product, campaigns and promo games for new lunch propositions. Similar aspects are examined in [7]. Food flavor perception change through AR has been the focus of workshops, such as Hacking flavor perception: Design, technology, and gastrophysics [8]. Focusing on promotion, [9] examines the use of coupons, interactive menus, and packaging during food ordering. Moneyinc presents indicative applications such as Coca-Cola and Nestle cereal, along with their assumption of how AR and VR intervened in the food sector. Racketail [10] states that restaurants and the food industry have spiked their sales with the help of AR. In [11], the focus is on AR packaging, which, as stated, “creates an engaging brand experience and empowers customers to directly interact with a brand via uber-creative visual and auditory content”. Ascentpark [12] describes the advantages of using an AR-based food menu, such as increased visibility, sales and promotion boost, the active engagement of customers, and the provision of infotainment features during dining. Similar subjects are explored in [13]. Skywell software company [14] enumerates how AR benefits customers by providing them information such as country of origin with pictures of the farm or factory where the product was made, nutrition information, test results and certificates, along with the visualization of recipes. This information can lead to better shopping decisions, as stated. In [15], the prospect of AR in boosting gastronomic experiences is explored with the use of interactive menus and virtual interiors.

In this paper, the focus is on a systematic presentation of actual AR applications and their implementation in the sectors of food analysis and food promotion. The presentation of the applications intends to provide a thorough mapping of the current state of AR use in food analysis and promotion, depict limitations, and inspire future implementations.

The rest of the paper is organized as follows. Section 2 presents an introduction of augmented reality, focusing on uses and technologies employed by the current review. Section 3 clearly specifies the research domain, presents the review methodology that includes the selection of applications, the reviewing criteria, and the presentation of selected applications based on these criteria. The current state is then discussed in Section 4. Discussion is followed by the potentials of further AR use in food-related issues that combine existing or integrate new features for satisfying end users’ needs in all aspects, from food production to consumption.

2. Augmented Reality Technologies and Potentials

Augmented reality has been defined as “systems that have the following three characteristics: (1) Combine real and virtual, (2) Interactive in real time, (3) Registered in 3-D”, while generally, it is described as a state between real and virtual environments [16,17]. Applications in augmented reality are divided into two main categories: (i) image-based and (ii) location-based. Image-based applications are divided furthermore into two categories: (a) marker-based, which require specific labels (e.g., Quick Response Code) and (b) marker-less tracking, in which an image (photo of the real environment) becomes the trigger for the playback of multimedia content. Location-based applications are triggered by the user’s arrival at a certain location [17]. An AR application may be implemented from scratch as a custom implementation, by using an existing AR platform, or an underlying OS dependent toolkit. AR has numerous applications in many domains, such as culture and education [17], retail settings [18,19], and markets [20].

A lot of reviews have concluded that AR will play an important role in these domains. In [18], the concept of the ‘augmented store’ is introduced, which is a physical store enhanced to accommodate AR. The evaluation shows that consumers appreciate the ability to experience an enhanced, more immersive and “realistic” store environment arising from AR. AR stimulates brand engagement and consumers’ desire to shop. Interest in AR is expected to increase in retailing, and there is a great scope for further development [19]. The technology acceptance model (TAM) [20] has been tested in [21] for virtual try-on, with the aid of a smart mirror on young consumers in Italy and Germany. Findings across these two markets favor the employment of AR for supporting online purchasing decisions.

In the food industry, food promotion and consumption in restaurants and online orders share similar concerns with retail management, and extends them with nutritional concerns regarding food quality. In research, there are some reviews that have presented applications of AR in food industry. Rejeb et al. [22] review 51 publications for identifying the enablers of AR in the food industry, concluding that they include food process efficiencies, food decision-making, food marketing, food training, and food safety. Chang et al. [23] state that “the features of blending the real world with digital domain and even purely intimidate real world into virtual variety create more complicated business model and dynamics in the whole food chain and immigrate into multi-dimensional interaction from single dimension”. In food safety, [24] concludes that “the food industry could benefit from AR technologies through just in-time information on the location of inventory; instant recognition of different foods, tools, and preparation areas along with context-specific information; and recognition of dangerous food situations where food safety has been compromised”. In training, a representative application, ServAR [25], is presented in this review, as it also supports hospitality services in addition to training for correct food serving sizes. Crofton et al. [26] present mainly research-oriented AR applications in food safety and the perception of food images. Authors also focus on the use of biometrics and sensors in food industries. A sensor-based AR application for evaluating yogurt tasting is presented in [27]. Participants rated the acceptability of sensory characteristics of different kinds of yogurt samples, including appearance, color, aroma, taste/flavor, sweetness, sourness, mouthfeel, viscosity, aftertaste, and overall liking, while viewing them embedded on a virtual environment appropriate for every state. In nutrition, Juan et al. present in [28] an application that identifies nutritional information on packaged foods by using AR. Finally, concerning product traceability, Penco et al. [29] state findings that indicate a particular usefulness of Mobile AR for communicating nutritional information to consumers, and identifying the falsification for “Made in” products.

These reports and applications provide invaluable insights about the use of AR in various food sectors. Apart from findings and reviews, the reports refer to multiple food-related AR applications. However, they do not provide a comparative study of applications per sector and details about their implementation.

3. Materials and Methods

As seen in Section 1 and Section 2, AR may be used in a lot of food-related issues. In this paper, the focus is to present AR applications in the sectors of food analysis and food promotion from a technological point of view. Food analysis regards applications that provide nutritional information about some kind of food, either literally, or through the visual perception of food products. Food promotion concerns applications that support retail products, orders from restaurants, and consuming food in dining areas.

More specifically, this review intends to answer the following qualitative research questions:

• RQ1—Scope: How do applications support food promotion and healthier nutrition? Which are the food categories that are displayed by the applications? What is the content orientation of the applications?
• RQ2—Audience: What is the audience for the applications? To what extent do end-users embrace them? What is the appeal of the applications to them?
• RQ3—Technology and Cost: What kind of devices are employed for executing the applications? And in what kind of settings? What are the most common development platforms? What is the cost of using the applications?

Towards this direction, firstly, a search has been performed in Google Scholar for publications of the last decade containing the terms “Augmented Reality”, “application”, and “food”. In total, 17,400 results have been returned and sorted by relevance. Publications in results were characterized as suitable, and were included in this review if they contained an actual AR application implementation in the domain of interest, or contained a review with references to actual applications in the domain of interest. The inclusion stopped when no more relevant content was present in the results. The outcome was 19 publications with actual applications [25,27,28,30,31,32,33,34,35,36,37,38], and reviewing references to existing applications [3,18,19,22,26,29,39]. It became obvious that most of the actual applications from research resources are limited and rather outdated [31,32,34,35,37]. References from reviewing papers to actual applications are mostly commercial [26,33,39]. These applications are also included in this review. As an extra filtering process, in order for an application to be included in this review, it should have been presented in detail in a publication, or be available for execution. Furthermore, an application has to be evaluated, preferably from third parties, or have a measurable appeal (downloads, sales and/or user reviews—as shown in

Table 1. AR applications and their characteristics.

No.	Title	“Catch”	Context	Content	Device	Marker	Implement	Appeal/ Evaluation	Deliverable	Research/ Commercial
1	Wise snacks AR [40]	Snack/mobile	Promotion	Game	Mobile	Packaging	Zappar	No data	Free	Commercial
2	M&M’s arcade [41]	Snack/ads	Promotion	Game	Mobile	Packaging/ads	Blippar	No data	Free	Commercial
3	AR gauge meter [31]	Gauge meter	Food analysis	Nutrition	Mobile	Food picture	Custom	Pending	No deliv	Research
4	Augment app [33]	3 representations	Food analysis	Nutrition	Mobile	Packaging	Augment	No difference	No deliv	Research
5	Perceived fullness evaluation through a picture-based AR app [34]	Food size	Food analysis	Nutrition	HMD	Cookie	Custom	Consume more with small sizes	No deliv	Research
6	Projective-AR for customizing appearance and food taste [36]	Food taste	Food analysis	Nutrition	Projective	Cookies, chips	Custom	Sweetness varies with color	No deliv	Research
7	Meta cookie [35]	Food smell	Food analysis	Perceived smell	HMD	Cookie	Custom	Both olfactory and visual stimuli affect user taste	No deliv	Research
8	Bubble tea [42]	Face animation	Appearance	Entertainment	Mobile	Face	Instagram	Million impressions	Free	N/A
9	Papaya [43]	Body animation	Appearance	Entertainment	Mobile	Body	Snap	Million impressions	Free	N/A
10	Lobster [44]	Animation on surface	Appearance	Entertainment	Mobile	Dining surface	Qreal/insta	No data	Free	N/A
11	Domino’s pizza [45]	Menu+order	Order	Menu	Mobile	Dining surface	Kabaq/snap	Thousand views	Free	Commercial
12	Eats in a blink [46]	Contest/eye blink	Promotion	Game	Mobile	Face	Instagram	Over 1 million players	Free	Commercial
13	Peached [47]	Multiple transforms	Appearance	Entertainment	Mobile	Faces	Instagram	No data	Free	N/A
14	Patrón experience [48]	Making/surface	Promotion	Infotainment	Mobile	Bar surface	Apple ar	4.3/5 rating @ appstore	Free	Commercial
15	Coca-Cola magic [49]	Multiple promo	Promotion	Entertainment	Mobile	Packaging/poster	No data	Thousand downloads	Free	Commercial
16	Great american bacon cheeseburger [50]	Seasonal promo	Promotion	Entertainment	Mobile	Burger picture	Blippar	No data	Free	Commercial
17	Nestle cereal [51]	Big company	Promotion	Game	Mobile	Generic marker	Dassault	Thousand views	Free	Commercial
18	19 crimes [52]	History/bottle	Promotion	Entertainment/game	Mobile	Packaging	No data	Million installations	Free	Commercial
19	Boston pizza [53]	Test	Promotion	Menu	Mobile	Menu	Vuforia	No data	No deliv	Commercial
20	Sketch [54]	Artistic	Hospitality	Entertainment	Mobile	Dining surface	Custom	5.0 rate	Free	Commercial
21	Safe food AR [37]	Sealed food	Food analysis	Nutrition	Mobile	Packaging	Vuforia	No relevant feedback	No deliv	Research
22	Halal [30]	Religious diet	Food analysis	Nutrition	Mobile	Packaging	Vuforia	Positive feedback	No deliv	Research
23	Google glass app [55]	Google glass	Food analysis	Nutrition	Glass	Food	Glass	75.9–87.9% accuracy	No deliv	Research
24	Snack university [31]	Cardboard game	Appearance	Entertainment	Mobile	Generic marker	Ar goggles	No data	No deliv	Research
25	AR ThaiMalay [56]	Translation	Hospitality	Menu	Mobile	Label	Custom	100% accuracy on some items	No deliv	Research
26	Nesquik [57] (same as Kelloggs [58])	Jigsaw	Promotion	Infotainment	Mobile	Packaging	Blippar	20% purchased another box	Free	Commercial
27	Cadbury [59]	Advent-product	Promotion	Filters	Mobile	Packaging	Blippar	4th best-selling advent calendar in the uk in 2017	Free	Commercial
28	Boost [60]	Drink/sport/game	Promotion	Game	Mobile	Packaging	Blippar	3.5 interactions/user	Free	Commercial
29	Mcdonald’s [61]	Trays/seasonal	Hospitality	Calendar/filters	Mobile	Trays	Blippar	12,000 scans > 3 days	Free	Commercial
30	Pepsi/Subway [62]	Music	Hospitality	Game/music	Mobile	Packaging	Blippar	70,000 customers	Free	Commercial
31	Magnum [63]	Decision on flavor	Hospitality/order	Decision/game	Mobile	Packaging	Blippar	48.5% engagement rate	Free	Commercial
32	Le petit chef [64]	3d projection	Hospitality	Entertainment	3d projection	Dish	Custom	>50 restaurants	Free	Commercial
33	ServAR [25]	Training	Hospitality/Food analysis	Nutrition	Tablet	Dish	Zappar	Improved accuracy and consistency of estimating standard serve sizes	No deliv	Research
34	Kiosk [38]	Multiple plabels	Food analysis	Nutrition	Mobile	Packaging	No data		No deliv	Research

). Applications implemented by the same platform with identical functionality as others already included in the review (mainly by Blippar and Zappar) have been omitted, as they would not add some extra value in this review. This process led to 34 representative applications of various implementation platforms, target audiences, and contexts that are presented in Section 3.3.

During the analysis of the representative applications, in order to achieve a comparative review as uniform as possible, a set of classification criteria have been defined that are present in most applications, and relate to functionality, implementation, and food-related issues. More specifically, the criteria represent the content of applications and the context that is used, implementation details related to AR, functionality, appeal, evaluation, and price. The meaning of every criterion is analyzed in Section 3.1, along with references to applications presented, in order to better clarify them. The presentation of applications based on these criteria allows the generation of an expert view on these applications in Table 1.

As this review focuses on technological details, the platforms that have been used for implementing the AR applications in these sectors have been identified and presented separately in Section 3.2. There are custom implementations, employment of platforms, and/or underlying standard AR toolkits.

In the rest of the section, before the presentation of applications, in Section 3.1, the eight classification criteria are presented, and Section 3.2 enumerates the implementation platforms that have been employed in the implementation of the applications. The exemplary applications are presented in Section 3.3 and in Table 1, based on the classification criteria. Reviewing information of the applications also takes place in Section 4, and during the definition of classification criteria.

3.1. Classification Criteria

In this section, the eight classification criteria are presented according to which of the AR applications are characterized.

• Context: The first criterion reflects the context in which the application is executed, the stage during which a user engages interaction with a food product or process. Usually, an application supports one stage. A commonly used context is food brand promotion aiming at brand reputation and sales increase (e.g., Wise Snacks AR, Pepsi Max, and Nestle Cereal). Other contexts include the food order process (e.g., Domino’s pizza), or other innovative interfaces (e.g., Magnum, Eats In A Blink). Hospitality context is related to some dining service (e.g., Le Petit Chef) in a restaurant or take-away (e.g., Magnum). Food analysis context reflects apps displaying the ingredients and other nutritional information of raw food (e.g., AR gauge meter), or sealed food (e.g., Safe Food AR). Food appearance context, often combined with entertainment purposes (e.g., Bubble tea), focuses on the display of different food types and transformations (e.g., Peached).
• Content: The content of the application is reflected on the second criterion. The content may be nutritional information about a food, such as calories, fat, sugar, vitamins, etc. (e.g., AR Gauge meter and Augment app). An application may present alternative flavors, tastes, and colors, helping customers choose their favorite food. This choice is not necessarily based on nutritional facts, but on subjective tasting preferences (e.g., Magnum). A lot of food-related applications present the menu of a restaurant, either for in-dining purposes (e.g., ARThaiMalay), or for delivery (e.g., Boston Pizza) and catering. Most of promoting applications contain entertainment content including music (e.g., Pepsi/Subway), animation (e.g., Le Petit Chef) and sport games (e.g., Boost). In this criterion too, content combinations exist, such as nutritional information and games (e.g., Scott’s).
• Scenario: It describes the interaction process of a user with all application cases, such as a game (e.g., Nestle Cereal), a contest (e.g., Eats in a Blink), the ordering process (e.g., Domino’s pizza), etc.
• Device: The criterion enumerates the device with which the interaction takes place. Most commonly, a mobile device is employed (e.g., Patrón Experienc‪e). Other interesting mediums appear too, such as a head-mounted display (e.g., Meta cookie), a projector (e.g., Le Petit Chef), and Google Glass. When an application is run on a mobile device, it may require a priori the installation of the whole application (e.g., SHALAL) or the generic application of the platform that it depends on (e.g., Blippar, Zappar, and Augment).
• Marker: For activating AR, a marker is always necessary. All presented applications employ image-based markers in the form of food (e.g., Meta cookie), pictures of food (e.g., AR gauge meter), food packaging (e.g., Kellogg’s), labels in menus (e.g., ARThaiMalay), dining (e.g., Sketch), and bar surfaces (e.g., Patrón Experienc‪e), trays (e.g., McDonald’s), dishes (e.g., Le Petit Chef), faces (e.g., Bubble Tea), advertisements (e.g., M&M’s Arcade), bus stations (e.g., Coca cola) or generic markers (e.g., Snack University).
• Implementation: An IT-oriented criterion that describes the implementation process of the application. The implementation may be based on a platform such as the ones presented in the following section (e.g., Magnum with Blippar platform), or it may be a custom implementation (e.g., ARThaiMalay) that may be based on a core platform, such as Apple AR Core or Google AR ToolKit. Custom implementations are employed mostly by research products.
• Appeal/Evaluation: In this criterion, any data available for the appeal and/or evaluation of the application are presented. These may be app downloads (e.g., Coca-Cola Magic), interactions (e.g., Boost), rates (e.g., Sketch), engagement (e.g., Magnum), sales boosts (e.g., Nesquik), statistical evaluation data (e.g., Meta Cookie), and app views (e.g., Nestle Cereal). If there are no numeric evaluation results, qualitative evaluation results are provided for this criterion. This is more often in research application (e.g., Augment app).
• Price: The price that a user should pay for obtaining the app. Most apps are free to use when purchasing a product (e.g., Nesquik), being on some promotion area (e.g., Coca-Cola magic) or dining area (e.g., Le Petit Chef). Most research-based apps are not available to the public after their publication.

3.2. Implementation Platforms

Firstly, the section presents AR platforms (1–6) employed by some applications, focusing on food-related features. Then, some companies (7–11) are presented with expertise in AR-based labelling and menu applications, for which no actual case studies are available publicly. Finally, there is a reference to supporting platforms and toolkits (12–16), on which the implementation of some applications has been based.

• Zappar [65] is a platform for implementing AR applications. The platform includes ZapWorks Studio, which can be used for programming AR applications and ZapWorks Designer, where active content is placed visually on an image. The generic Zappar app should be installed on a mobile device for accessing AR applications made by Zappar tools. The app has been downloaded by more than a million users on the Play Store, but the review score is quite low (2.7/5). Importantly, the platform provides a universal AR SDK, with which Zappar’s core technologies may be integrated into another platform or tool. The platform is ThreeJS, A-Frame, JavaScript, Unity and React-ThreeJS compatible. There is no significant expertise in food industries.
• Blippar [66] is a generic platform for implementing AR applications. It provides Blippbuilder for transforming packaging, ads, magazines, and posters in AR applications, without the need for coding. More advanced AR applications are achieved with Blippbuilder Script, which supports markered and markerless tracking, face tracking, etc. The generic Blippar app is needed in a mobile device for accessing Blippar applications. According to Play Store, it has been downloaded by more than a million users, and the review score is average (3.2/5). Blippar has numerous applications in food industries, especially for promoting well-known brands, with more than encouraging engagement results. A lot of Blippar applications are presented in this paper, as each of them supports unique features.
• Augment [67] is an ARKit and ARCore compatible platform, and a mobile app that integrates, in real time, actual size visualizations of 3D models in AR. Developers can directly upload their own 3D models to be available through Augment app, or upload some 2D pictures through 3D Factory and let Augment guide them to produce a 3D model. The generic Augment app has to be installed in a mobile device for accessing Augment applications with 3D models. According to the Play Store, it has been downloaded by more than a million users, and the review score is quite good (3.9/5). Augment has some expertise in the food sector, and an indicative application that they promote is the partnership with Coca-Cola [68], which allows representatives to choose the ideal placement of a POS through AR-based actual size simulation.
• Formerly known as Kabaq, QReal [69] has a long engagement with food sector applications, although it supports AR applications for other domains too. The company has excelled in lifelike models of cuisine, and especially the presentation of dishes in AR. They state that, when customers view food in AR, it makes them more likely to order, raises check averages, and increases positive word of mouth. Qreal supports the development of social media AR experiences through AR (e.g., Lobster filter), the production of QR Codes for AR, and applications for food delivery and catering menus, as in Magnolia Bakery QReal [70].
• Dassault Systèmes [71] is a mega company with expertise in 3D design. Concerning AR and 3D representations, the company suggests the 3DEXPERIENCE platform, which allows the preparation of 3D models and their exploitations in various representations, further enhanced by the API that they provide for integration in other platforms. The company has expertise in the food sector, stating that they support consumer demand for ‘healthier’ nutrition [72]. An indicative example that they promote is the marketing strategy to launch the expanded line of Häagen-Dazs ice cream, which adopted the 3DEXPERIENCE platform.
• Vuforia [73] is a development platform for image-based AR apps. It can be combined with Unity 3D applications [74]. It provides developer studios and APIs that allow apps to be embedded in other platforms.
• Visuar Labelling [75] implements AR interactive labelling applications that can be used for advertising and providing food information. They also enable customers to interact with a product through a personalized experience.
• Arsome [30] develops generic AR applications, stating their expertise in AR-based interactive restaurant menus by presenting some demos [3].
• Juego Studions [76] provide the AR Restaurant Food menu, which is an interactive and informative mobile application. It supports a healthier lifestyle by providing, for a restaurant menu, nutritional information, calories, ingredients, cooking videos, etc.
• Jarit [77] is an AR application, which allows its users to preview their potential order in 3D. It is designed specifically for the food industry, and partner restaurants may employ it to make their dishes more realistic to customers through 3D visualizations of dishes with the use of photogrammetry. Visualizations can be displayed through the Jarit app, or embedded in other platforms.
• Menu.AR [78] is a similar application that supports displaying 3D representations of dishes in a restaurant menu in their actual size, and the way they look on all sides. For every dish, the ingredients and the size of the food in it are also displayed. The dish is displayed in a dining surface through the Menu.AR app. Users point the camera at the surface and see their order on the surface. At the Play Store, there are more than 10,000 installations, and the app has a 4.3 rate.
• Instagram filter Spark AR platform [79] is a generic AR platform by Facebook, with scripting capabilities for generating filters. It may be employed by other platforms and apps.
• Snapchat [80] is a generic AR platform by Snapchat, with scripting capabilities for implementing AR lenses, embeddable in Snapchat and other platforms.
• Apple AR Toolkit [81] is a generic AR platform by Apple for devices in the iOS ecosystem.
• Google AR Core [82] is a generic AR platform by Google with scripting capabilities. It may be exploited in the development of Android applications.
• Google Glass [83] is a discontinued AR platform and hardware by Google that has been revived as Glass Enterprise Edition [84]. It supports building applications based on Glass Android OS.

3.3. Applications

The section presents AR applications in food promotion and analysis based on the classification criteria defined in Section 3.1. In every app, there is firstly a sentence that summarizes the app’s unique characteristics. A summary of the apps’ characteristics is presented in Table 1. In Table 1, every app is also characterized as commercial or research-oriented. Four applications cannot be described as commercial or research-based. “N/A” is noted for them.

• Zappar Wise Foodsbaseball-themed AR game presented in [40] is a promotion application for a snack.
  Context: Wise snacks has partnered with major baseball teams in order to produce and promote baseball-themed chips devoted to their teams.
  Content: A baseball mini-game.
  Scenario: The game starts with an animation, and then players see an animated pitcher as augmented content. The pitcher tosses baseballs towards the player, who must hit the balls by touching them at the right moment. The player earns points for each ball hit, and the game ends after three misses. Players can record their score and take a picture with the virtual pitchers, which they can further share.
  Device: The game is accessible through a mobile device with a camera running iOS or Android.
  Marker: The packaging of the snacks includes codes to access the game. While the game is accessible through the packaging of snacks colored according to a team, the in-game pitcher appears in Wise’s brand colors.
  Implementation: The Zappar app for iOS or Android is used for implementing the game.
  Appeal/Evaluation: No specific data for this app. The generic Zappar app has been downloaded by more than a million users, but the review score is quite low (2.7/5).
  Price: Free.
• M&M’s ARcade [41] is an AR game by Blippar for promoting M&M’s Caramel activated by advertisements and packaging.
  Context: Promotion for M&M’s caramel-filled variety of candy.
  Content: M&M’s Arcade game.
  Scenario: Three vintage games, Caramel Cannon, Caramel Crawl, and Square Smash appear on the game’s home screen. In Caramel Cannon, the player uses the arrows to move the cannon left and right to shoot a falling caramel square that splashes on impact. In Caramel Crawl, the player collects parts of Caramel characters that, when completed, initiate a quick cartoon animation. In Square Smash, the player smashes as many caramels as they fall, while avoiding M&M’S characters.
  Device: The game is accessible through a mobile device with a camera running (iOS or Android).
  Marker: Advertisements in New York’s Times Square or packages of M&M’s Caramel.
  Implementation: The Blippar app for iOS or Android is used for accessing and playing the game.
  Appeal/Evaluation: No specific data for this app. The generic Blippar app has been downloaded by more than a million users, but the review score is average (3.2/5).
  Price: Free.
• AR gauge meter system visualizes the nutritional content of food [31].
  Context: A food analysis application that provides the carbohydrate percentage of a food.
  Content: The application helps users make sound nutrition choices through a visually appealing and familiar interface, displaying the carbohydrate percentage of a food.
  Scenario: Upon detecting a food based on its image, the application displays the percentage of the food’s carbohydrate on an analog Gauge meter.
  Device: The application is accessible through a mobile device with a camera running Android.
  Marker: A picture of a food initiates the application.
  Implementation: Custom development with use of open-source technologies, including Java, C++, and JavaScript.
  Appeal/Evaluation: Review pending.
  Price: No deliverable.
• AR application conveying nutritional information of drinks ranging in healthiness [33].
  Context: A food analysis application that displays a moving 3D human model, based on the nutritional ingredients of six drinks.
  Content: The application encourages users to select a drink based on its nutritional information, such as calories, fat, and sugar.
  Scenario: For each beverage, three unique moving 3D human models were developed according to whether the product is good, medium, or bad in terms of ingredients. Each model was unique for a certain product, reflecting the appearance of the most recently scanned product.
  Device: The application is accessible through a mobile device with a camera running Android or iOS.
  Marker: Packaging (bottles) of drinks is employed for detection.
  Implementation: The implementation is based on the Augment platform.
  Appeal/Evaluation: As stated in the paper, the main purpose of the implementation was to compare three different nutritional information provision methods in terms of their impact on healthy food choice: nutrition facts table, traffic light logo, and AR application. Results showed that there was no significant difference in drink choice between the three information methods. Nutritional information conveyed via the traffic light logos was perceived as more understandable, comparable, and salient than the information that was conveyed via the nutrition fact tables. The AR app was evaluated as being less useful than the other two methods. Researchers state that this outcome can be explained by the lack of detailed information that was provided in the AR app.
  Price: No deliverable.
• Application evaluating the effect of a food picture on perceived fullness [34].
  Context: Visual transformation of food that increases the size of food by 15% using AR.
  Content: Addressing nutrition issues by coping with obesity, the application encourages users to consume less food, as it visually satisfies them by increasing the volume of food by using AR.
  Scenario: The AR app changes the size of food in the visual field through a head-mounted display (HMD). The shape of the users’ hand is also deformed as if they really were handling the virtually scaled food. This gives the users the impression that there is a difference in food size, while the size of the hand remains stable. The app focuses on finger-food eating, as it can easily be compared to one’s body elements.
  Device: In this system, users wear only a video see-through HMD.
  Marker: Actual food, especially cookies.
  Implementation: Custom implementation through a system that consists of a laptop PC, a video see-through HMD, camera, and food. Firstly, the area of the food item and the user’s hand is extracted from captured images based on color. Then, the appearance of food is scaled. Next, the hand is resized by using a deformation algorithm. Finally, the scaled food and deformed hand images are overlaid over the background image.
  Appeal/Evaluation: Results indicated that subjects consumed significantly greater amounts of the cookie when they ate a visually smaller cookie relative to a visually larger cookie.
  Price: No deliverable.
• A projection-based AR app that evaluates the perceived appearance and sweetness of food products is presented by Nishizawa et al. [36].
  Context: The visual transformation of food is performed by the application that changes the color and the appearance of food by using Projective AR.
  Content: The app is used for investigating whether the evaluation of taste can be modified by the food color of a sponge cake and chips.
  Scenario: First, participants drink natural water before to wash away any taste in the mouth. Second, a castella stimulus on the dish was presented with a specific chroma controlled by the system. Then, a food stimulus was presented, and participants cut the food into bite-size pieces with chopsticks, move the castella slowly, and carry it to their mouths. After eating, participants evaluated the sweetness of the cake by using a 7-point semantic differential scale. Regarding the flavor, a potato chip was placed on the projection area. The food color was selected from 4 types. Participants ate it in one bite with black chopsticks. They were instructed to look at the stimuli before eating, and while eating. After that, participants evaluated their perception of flavor.
  Device: A projector and a camera are fixed on a stand and a table, respectively. A camera and a mirror obtain the top view of food, and a projector shows the edited food image.
  Marker: Actual food, cake, and chips.
  Implementation: Custom implementation. To modify the appearance of food in real time, the extraction of the food region is extracted from the camera, and a monochromatic mask image is created from the food binary image and projected on the food from the projector.
  Appeal/Evaluation: Results show that subjective sweetness increases with increasing chroma of castella, even though they ate the same cake, indicating that the chroma of food can influence the sweetness. The flavor of potato chips was also changed by modifying the color optically with the projector, while not modifying the food itself.
  Price: No deliverable.
• Meta Cookie’s AR system evaluates the effect of overlaying visual and olfactory information onto a real cookie [35].
  Context: Visual transformation of food is performed by the application that checks the perception of taste when overlaying visual and olfactory information onto a real cookie.
  Content: The application focuses on how users may change their smell based on food size and color.
  Scenario: A user wears an HMD with olfactory capabilities. The cookie detection unit detects the cookie that the user has chosen between two types of appearances and scents of commercially available cookies: chocolate and tea. Then, an overlaid image, along with the scent of a flavored cookie with an intensity that is determined based on the calculated distance between the cookie and the user’s nose, is generated by the olfactory system.
  Device: Users only need to wear a video see-through HMD.
  Marker: Actual food, cookies.
  Implementation: MetaCookie’s custom implementation consists of four components: a marker-pattern-printed plain cookie, a cookie detection unit, an overlaying visual information, and an olfactory unit.
  Appeal/Evaluation: Results show that olfactory stimuli affect user perception of taste, but not sufficiently enough, without the help of visual stimuli. The system can change a perceived taste, and lets users experience various flavors while eating the same cookie, with the help of visual and olfactory stimuli.
  Price: No deliverable.
• Bubble Tea [42] is an entertainment Instagram AR filter consisting of face animation into a tea drink. Robbie Conceptuel [85] has implemented many more similar apps such as Papaya.
  Context: AR-based transformation of a user’s face.
  Content: A person’s face is employed for producing visual transformations only for entertainment purposes.
  Scenario: Using a user’s face, this is transformed into a drink. When users open their mouth, then jelly bubbles pour out of it.
  Device: The filter is accessible through a mobile device with a camera running Instagram.
  Marker: The filter is based on the user’s face detected by the mobile device’s camera.
  Implementation: Instagram AR filter by Robbie Conceptuel.
  Appeal/Evaluation: A million impressions according to the artist.
  Price: Free.
• Papaya is a Snapchat filter [43], which transforms the user’s body into a tropical fruit that floats in nature.
  Context: AR-based transformation of a user’s body.
  Content: A person’s body is employed for producing visual transformations only for entertainment purposes.
  Scenario: A user’s body is transformed into a fruit that moves around natural settings.
  Device: The filter is accessible through a mobile device with a camera running Snapchat.
  Marker: The filter is based on detecting the user’s body by the mobile device’s camera.
  Implementation: Snapchat filter by Robbie Conceptuel.
  Appeal/Evaluation: A million impressions according to the artist.
  Price: Free.
• Lobster Mac&Cheese is an Instagram filter by QReal [44] where a user can place a photorealistic 3D model of a lobster with macaroni and cheese on a flat surface.
  Context: AR-based plate positioning.
  Content: Animation of a plate of lobster with macaroni and cheese for entertaining purposes.
  Scenario: A flat surface is detected upon which a plate of lobster with macaroni and cheese is placed.
  Device: The filter is accessible through a mobile device with a camera running Instagram.
  Marker: The filter is activated when detecting a flat surface.
  Implementation: Instagram filter by QReal.
  Appeal/Evaluation: No specific data.
  Price: Free.
• Kabaq has implemented a Snapchat lens [45] so that consumers may view a Domino’s pizza in AR and then order that pizza online.
  Context: Order Domino’s pizza online.
  Content: Display Domino’s pizza delivery menu.
  Scenario: The lens starts with a reflective pizza glass. Flipping the camera shows a pizza box and reveals a photorealistic pizza based on Domino’s menu and the user’s selection. Customers can order pizza directly on Snapchat using “Order now” button.
  Device: The lens is accessible through a mobile device with a camera running Instagram.
  Marker: The pizza is placed on a flat surface detected by the mobile device’s camera.
  Implementation: Implementation through Snapchat AR Lenses by Kabaq.
  Appeal/Evaluation: No specific data. A thousand views of the YouTube video.
  Price: Free.
• Instagram app “Eats in a Blink” contest by Uber Eats India was for boosting orders to their platform [46].
  Context: Increase brand recognition by exposing users to a promotion from which they could benefit.
  Content: A game and contest aiming to earn points to win free food for a year. The player with the highest points by the end of the campaign, ran in 2019, gained free food for an entire year, and a financial award.
  Scenario: Players tap on their screens to begin. The camera opens and shows an AR scoreboard over the user’s forehead. A delivery driver from Uber Eats traverses a city and the player’s aim is to blink at every green building, which means that a delivery is completed, and a point is gained. Players may share their scores for followers to see, and be tempted to enter the contest.
  Device: The filter is accessible through a mobile device with a camera running Instagram.
  Marker: No marker to start the game. Eye blinking is detected to mark successful delivery.
  Implementation: Instagram AR platform.
  Appeal/Evaluation: Over 1 million players have used the app during the weeklong campaign.
  Price: Free.
• Peached by Beth Wickerson [47] is an entertaining Instagram app where the heads of multiple viewers may be simultaneously replaced by proportionally sized peaches.
  Context: AR-based transformation of a user’s face.
  Content: A person’s face is employed for producing visual transformations only for entertainment purposes.
  Scenario: Using a user’s face, this is replaced by a peach, the size of which is analogous to the user’s face size. The filter can be performed simultaneously to multiple persons shown in the camera.
  Device: The filter is accessible through a mobile device with a camera running Instagram.
  Marker: The filter is based on the user’s face detected by the mobile device’s camera.
  Implementation: Instagram AR filter.
  Appeal/Evaluation: No data.
  Price: Free.
• An Apple ARToolkit-based AR app for the history of Patrón tequila [48].
  Context: Promotion of the Patrón tequila brand.
  Content: The app narrates the procedure of making tequila and tasting differences through an entertaining and graphically appealing setting.
  Scenario: The application is initiated on a flat surface with a single Weber Blue Agave plant. The user should plant the agave as in a field across the flat surface. When done, the field of agave flourishes, and Patrón Hacienda appears, where Patrón Tequila is produced and bottled. The experience continues with animations of Patrón on bottles, while a bartender guides through the process of tequila making and explaining tasting and aging process when tapping on a bottle.
  Device: The application is accessible through a mobile device with a camera running iOS.
  Marker: The app is designed to work on a flat surface like a bar top.
  Implementation: Apple ARToolkit.
  Appeal/Evaluation: 3/5 rating @ AppStore
  Price: Free.
• The Coca-Cola Magic app [49] is an AR app with multiple markers and experiences.
  Context: Promotion of Coca-Cola brand.
  Content: The app provides three types of experiences that are activated differently, and initiate some kind of entertainment according to the setting where the activation took place.
  Scenario: The application has been designed for the holiday season. There are three kinds of markers, leading to different experiences:

  -

  Coca-Cola Christmas bottles that trigger a snow animation around users.

  -

  Bus stations with Coca-Cola advertisements. When activated, Coca-Cola trucks pass by.

  -

  Coca-Cola sign on floors in a mall, which makes Santa appear among visitors.

  -

  Device: The application is accessible through a mobile device with a camera running iOS or Android.

  Marker: The application is initiated on a bottle, an advertisement, or a sign.
  Implementation: No data.
  Appeal/Evaluation: According to video [49], a thousand downloads and reaches.
  Price: Free.
• “Great American Bacon Cheeseburger” seasonal campaign by Kraft is an AR infotainment app [50].
  Context: “Great American Bacon Cheeseburger” campaign for promoting 14 different summer-themed products, such as cheese, pickle relish, mayonnaise, and BBQ sauce.
  Content: The app provides entertaining content such as recipes, taking photos, and sweepstakes.
  Scenario: The application is activated when pointing to the picture of a cheeseburger and the whole mobile screen is filled with it. The app provides recipes, taking photos with a country singer and sweepstakes.
  Device: The application is accessible through a mobile device with a camera running iOS or Android and the Blippar app.
  Marker: The picture of a cheeseburger is used for activating the application.
  Implementation: Blippar.
  Appeal/Evaluation: No specific data for this app. The generic Blippar app has been downloaded by more than a million users, but the review score is average (3.2/5).
  Price: Free.
• An AR game has been implemented for a computer by a big company [86] for Nestle Cereal Box [51].
  Context: Nestle cereal brand promotion through a game with an animation hero.
  Content: The app provides access to an interactive game.
  Scenario: The application is activated when pointing a generic marker on a Nestle cereal box. A field appears, and after a while, a game setting where the player should move the box in order to make a ball move through a maze towards a mechanism appears, and when the ball touches it, it triggers the jumping of the RIO bird character Blu.
  Device: A computer with an Internet connection and a webcam is required.
  Marker: The application is activated when pointing to a quite large black and white marker on the packaging of a Nestle box—it is an early AR implementation.
  Implementation: PLATFORM: Dassault Systèmes.
  Appeal/Evaluation: No specific data for this app. A thousand views on the YouTube video.
  Price: Free.
• An AR app with rich animation features, interactivity and games has been implemented for the history of 19 Crimes Wine, and is initiated by the labels of wine bottles [52].
  Context: 19 Crimes brand promotion.
  Content: The app provides access to interactive games and numerous entertainment materials.
  Scenario: The application is activated when pointing to the picture on a wine bottle. Then, a user has access to audio, video, textual material interactive material and games as follows:
  AUDIO: stories of history’s Convicts and Rebels behind the 19 Crimes and stories from Embrazen women about their achievements, and learn how to be a Gentleman.
  VIDEO: movies with The Walking Dead fights, the Barossa Valley that show why Wolf Blass is one of Australia’s most awarded winery, the art of hosting great parties and exclusive recipes using Main & Vine.
  INTERACT: Interactions with the Warden in a trial with the Magistrate, for creating artworks and Curly Tales, simulating a dip in the chilling iceberg, and learning how to check if Matua wine is chilled to drink.
  Device: The application is accessible through a mobile device with a camera running iOS or Android.
  Marker: The application is activated when pointing to a 19 Crimes wine bottle (packaging).
  Implementation: No data.
  Appeal/Evaluation: Million installations; 2.5/5 grade @ Play Store, 3.9 grade @ App Store.
  Price: Free.
• A simple testing app for the capabilities of AR menus has been implemented for Boston Pizza [53] by Patio Interactive that displays images and nutritional information about plates.
  Context: PROMOTION. Patio interactive demo for Boston Pizza menu.
  Content: MENU/NUTRTION/GAME: The app provides information about Boston Pizza plates, through an interactive setting, while having access to a simple entertaining game.
  Scenario: The user should point the Boston Pizza menu from a certain distance. When detected by the app, it initiates a 3D presentation of the menu. The user may choose plate category, and choose an item from the category and present ingredients and nutrition information. The user may also share a menu item, contact personnel, and play a simple game by popping some balls.
  Device: Mobile device with camera.
  Marker: The application is activated when pointing to the menu of Boston Pizza.
  Implementation: Unity and Vuforia.
  Appeal/Evaluation: No specific data for this app. Implemented for demo issues.
  Price: No deliverable.
• Sketch and HATO [54] have implemented an AR hospitality app with strong artistic features.
  Context: Augment the hospitality experience at Sketch London, a place in London for dining and art.
  Content: While seated, the app provides entertaining animations inspired by Turner Prize nominee David Shrigley’s artworks. The app lets users blend between the digital and real world, while being served afternoon tea or an evening cocktail.
  Scenario: Users sit at a table and around their dining area; 15 different animation stickers based on artworks, such as Shrigley’s tearful puppy to a spray, are activated as animations of characters and motifs from stickers are placed in and around the immediate surroundings of the users’ phones. Stickers interact with the objects around them, allowing users to explore their own AR world, download and further share their experience.
  Device: Mobile device running iOS with a camera.
  Marker: The application is activated when seated near a dining surface.
  Implementation: CUSTOM implementation for iOS. No reference for Apple ARToolkit use.
  Appeal/Evaluation: 5,0 rate.
  Price: Free.
• Safe Food AR application [37] implemented with Vuforia shows the usage of AR in food packaging for providing nutritional information for sealed food.
  Context: A food analysis app that provides detailed information about the origin and content of sealed food products.
  Content: It supports traceability, by showing the nutritional information and origin of sealed food product, and informing and protecting consumers in this way about the quality of products.
  Scenario: Users point to the packaging of a sealed food product. If available, the app displays overlaid the origin and content of the product. All data are stored in-app, enabling the costumer to be offline while scanning the product.
  Device: Mobile device with camera.
  Marker: The application is activated when pointing to a sealed food product.
  Implementation: Scripting was built on Mono, an open-source implementation of the NET Framework. Development employed Unity Script, C #, and Boo.
  Appeal/Evaluation: Tested only by the small group of users without having concrete feedback.
  Price: No deliverable.
• HALAL Mobile Application [30] is an AR app for supporting diet based on religious beliefs.
  Context: A food analysis app that provides information about whether a product has an appropriate halal status.
  Content: It addresses customers who need to know if a sealed food or beverage is suitable for their diet based on its nutrition information.
  Scenario: After signing in, a user may scan the logo of a product with a mobile device camera. The application displays information about the food.
  Device: Mobile device with camera.
  Marker: The application is activated when pointing to a sealed food product.
  Implementation: Vuforia is employed along with Android SDK, Eclipse, and a database containing logos of products containing the appropriate information.
  Appeal/Evaluation: A survey of 22 users revealed that users can detect the halal status of a product.
  Price: No deliverable.
• A Google Glass-based food app [55] presents consumers with nutritional information for promoting the right product choices.
  Context: A food analysis app that displays nutritional information of foods is overlaid with the use of Google Glass.
  Content: The ability of customers to visualize insightful nutrition-related information and manipulate virtual objects in AR settings aims to optimize the decision-making process and increase consumer satisfaction.
  Scenario: The user scans a food (e.g., a pineapple) with Google Glass and nutrition information is overlaid. With the proposed system, a user may see the nutrition information of multiple items registered on the corresponding food.
  Device: Google Glass and mobile device with an Internet connection.
  Marker: The application is activated when pointing to some food in a real environment.
  Implementation: The Google Glass platform is employed. The app consists of three main components: content-based food recognition; food image and nutrition information retrieval from a database with the use of JSoup; and image tracking and visualization.
  Appeal/Evaluation: The overall accuracy of image search from the real environment was 75.9%. Test with food pictures reached 87.9% accuracy.
  Price: NO DELIVERABLE.
• Snack University [32] is a cardboard game with AR features that promotes the cultural and historical breadth of Taiwan’s snack foods.
  Context: The game uses 3D animations to introduce and project Taiwanese snack foods culture to an international audience.
  Content: Players scan AR markers corresponding to snack cards with a webcam to access information about a snack. The application’s game tests players on their knowledge of Taiwan snack foods. The movement of players in the game takes place on a Monopoly-like formatted map of Taiwan by using dice.
  Scenario: As in the classic version of Monopoly, there are cards with Taiwanese snack food representations, which can trigger the 3D representation of the snack. When arriving at a point of a certain stack by rolling the dice, players may choose to buy the snack booth and take the corresponding snack booth card. Then, they use the image on the card to view the 3D video through AR goggles, thus giving them information about the snack. When players land on certain points corresponding to Taiwan landmarks, they have to answer a question about Taiwanese snack foods in order to earn some reward.
  Device: Computer with a webcam and AR goggles for viewing content.
  Marker: The game points to a generic AR marker in order to trigger the appearance of a 3D representation of a snack food.
  Implementation: ARToolkit/Flartoolkit: AR is integrated into the game via ARToolKit. NFT technology uses computer graphics techniques for determining AR markers positions. Flartoolkit is employed for publishing AR designs. The application can be embedded in HTML.
  Appeal/Evaluation: No data.
  Price: No deliverable.
• ARThaiMalay [56] supports the translation of printed Thai food menu to the Malay language.
  Context: A hospitality application that addresses the needs of tourists knowing Malay language to understand a food menu in Thai language.
  Content: Items in a food menu in Thai Language are recognized and displayed in Malay language.
  Scenario: The visitor points to the mobile device where the application is installed towards the phrase in the menu that needs translating. The translation appears over the original phrase.
  Device: A mobile device with a camera running Android without Internet connection.
  Marker: The application identifies labels with menu item phrases in the Thai language.
  Implementation: Tesseract OCR and Google Translate API for Android have been used in the implementation for translating menu item phrases stored locally in an SQLite database.
  Appeal/Evaluation: 100% accuracy for detecting some menu items. Less accuracy in other parts of the menu due to linguistic problems.
  Price: No deliverable.
• A Blippar game app for Nesquik [57] favors the completion a puzzle targeting children. A similar application is the promotion of Kellogg’s cereal by Blippar [58]. Another infotainment application by Blippar targets both parents and kids on vitamin information through games and animation is Scott’s app [66].
  Context: Promotion of Nesquik cereal boxes.
  Content: Infotainment content. Users construct 3D sea creatures from different Nesquik cereal boxes.
  Scenario: As Blippar states, a range of 3D sea creatures may be constructed from different Nesquik cereal boxes, in order to experience a variety of engaging and educational underwater missions.
  Device: The game is accessible through a mobile device with a camera running iOS or Android.
  Marker: The packaging of cereals is used for triggering the application.
  Implementation: The Blippar app for iOS or Android is used for accessing and playing the game.
  Appeal/Evaluation: According to Blippar, 20% of the application users purchased another cereal box.
  Price: Free.
• An advent calendar app for Cadbury by Blippar [59].
  Context: Promotion of Cadbury brand.
  Content: AR based on Cadbury’s traditional advent calendar.
  Scenario: Customers scan the Cadbury advent calendar with the Blippar app and a winter house appears, which may activate a new selfie filter every day throughout advent, based on festive scenes, such as Cadbury Wispa party glasses, snowball fights, reindeer antlers, and elf costumes.
  Device: The game is accessible through a mobile device with a camera running iOS or Android.
  Marker: The Cadbury advent calendar is used for triggering the application.
  Implementation: The Blippar app for iOS or Android is used for accessing and playing the game.
  Appeal/Evaluation: According to Blippar, the advent calendar debuted in 2017 as the 4th best-selling advent calendar in the UK.
  Price: Free.
• Promotion of Boost sport drink [60] through cricket, popular sport game in India. There is a similar sport-based application for Pepsi Max by Blippar [66].
  Context: Promotion of Boost energy drink.
  Content: As promoting an Indian brand, the app engages India’s favorite sport, cricket.
  Scenario: When triggered, the app initiates cricket lessons led by Indian cricket team captains. Kids can submit their best shots to Boost for a chance to meet one of the captains.
  Device: The game is accessible through a mobile device with a camera running iOS or Android.
  Marker: The bottle of the drink is used for triggering the application.
  Implementation: The Blippar app for iOS or Android is used for accessing and playing the game.
  Appeal/Evaluation: According to Blippar, a high engagement has been achieved with 3.5 interactions per user.
  Price: Free.
• McDonald’s hospitality AR app with Blippar [61] transforms tray liners into interactive advent calendars.
  Context: Hospitality app for enriching dining experience in a McDonald’s restaurant during the festive period.
  Content: The application unlocks games and filters for each day in the advent calendar.
  Scenario: When triggered through a tray liner, the app initiates an interactive advent calendar, every day of which unveils different games selfie filters, animation, and chances to win Amazon vouchers.
  Device: The app is accessible through a mobile device with a camera running iOS or Android.
  Marker: A tray is used for triggering the application.
  Implementation: The Blippar app for iOS or Android is used for accessing and playing the game.
  Appeal/Evaluation: According to Blippar, the campaign gained the success, as around 12,000 people scanned for three or more days during the festive period.
  Price: Free.
• Pepsi Max AR app with Blippar [62] focuses on music-related content.
  Context: Hospitality app that motivates Subway customers to order a Pepsi drink along with their food.
  Content: The application reveals music videos and the chance to win tickets for music concerts.
  Scenario: When a Pepsi cap is targeted, an augmented reality jukebox appears that plays video and music clips and the chance to win tickets for music concerts.
  Device: The game is accessible through a mobile device with a camera running iOS or Android.
  Marker: A Pepsi cap is used for triggering the application.
  Implementation: The Blippar app for iOS or Android is used for accessing and playing the game.
  Appeal/Evaluation: According to Blippar, 70,000 customers entered the competition.
  Price: Free.
• Magnum AR app with Blippar [63] focuses on shareable taste decision on the road.
  Context: Hospitality app that motivates customers to buy a Magnum ice cream from a store.
  Content: The application lets customers choose their flavors and order an ice cream with these flavors from a nearby store.
  Scenario: Customers have to target Magnum’s banner ads on mobile devices and experience Magnum’s different custom toppings floating around them, which they can select in order to compose their favorite ice cream and order it from nearby stores at a discount.
  Device: The application is accessible through a mobile device with a camera running iOS or Android.
  Marker: A Magnum banner add is used for triggering the application.
  Implementation: The Blippar app for iOS or Android is used for accessing and playing the game.
  Appeal/Evaluation: According to Blippar, a 48.5% engagement rate has been achieved.
  Price: Free.
• A hospitality app based on 3D projection, Le Petit Chef by Skullmapping [64], is an AR dining experience that displays 3D projections on dining tables.
  Context: Hospitality app that lets customers view an animation, showing the preparation of a plate by a small chef.
  Content: The animation is an entertaining unfoldment of the preparation of a dish.
  Scenario: A tiny chef unveils ingredients around the plate of the customer and uses them in an entertaining way for preparing a meal into the customer’s plate. This animation takes place, while the actual meals are being prepared.
  Device: Moreover, 3D projection mapping technology is used for projecting the animation on the customer’s plate.
  Marker: The animation is unfolded around the plate that is in front of a customer.
  Implementation: No data available.
  Appeal/Evaluation: According to Skullmapping, it has been installed in over 50 restaurants worldwide.
  Price: Free.
• ServAR [25] educates the estimation of the right portions of servings in a restaurant. Although it is a training application, it is included in this review, as it resembles the functionality of other hospitality apps.
  Context: Participants are educated through AR to estimate the right size when serving and estimating food portions.
  Content: Right size of servings for 9 different kinds of food.
  Scenario: A tablet computer overlays virtual food servings over a plate. Participants are asked to estimate the standard serving sizes of nine kinds of food using validated food replicas.
  Device: Tablet.
  Marker: The plate on which the virtual food is placed.
  Implementation: Zappar.
  Appeal/Evaluation: Ninety participants evaluated that ServAR is easy to use and showed potential to support optimal portion size selection, provided that some refinements are implemented. ServAR improved the accuracy and consistency of estimating standard serve sizes compared to traditional methods.
  Price: N/A.
• Kiosk [38] provides environmental, provenance and nutritional informative labels about packaged food through an AR application.
  Context: Participants are educated through AR to estimate the right size when serving and estimating food portions.
  Content: Display of environmental and nutritional information about packaged food, as well as their provenance in separate floating labels.
  Scenario: Users point to a packaged food through their mobile device and multiple labels displaying nutritional, environmental and provenance information.
  Device: Mobile device.
  Marker: Packaged food.
  Implementation: No data.
  Appeal/Evaluation: Eighty-four participants compared static and AR versions of the application. With the AR version, participants learned more about food products than with the static version.
  Price: N/A.

4. Results and Discussion

After the presentation of augmented reality applications in the domain of food promotion and analysis, a lot of resulting points can be derived regarding the purposes of applications, their audiences, the implementation procedures, and their appeal. This discussion will lead to proposals about the evolution of AR use in these domains. As a general comment, it is obvious that commercial applications (18/34) outnumber the research ones (12/34). Based on the potentials of AR in the food industry, as outlined in Section 2, this lack should ignite further research in the development of AR applications. Furthermore, the use of AR in food promotion has been accelerated due to the COVID-19 pandemic [3], which created the need for remote ordering processes and viewing plates in a menu in photorealistic way without having to visit a restaurant. Applications such as Domino’s Pizza and Magnum help towards this direction, along with platforms such as Menu.AR and the AR Restaurant Food Menu. The latter are some of the very few platforms thar are dedicated to restaurant sector and provide services for displaying a menu, viewing plates, and completing an order. In the rest of the section, results and discussion are organized by the research questions. To improve readability, the name of applications in Table 1 is employed for referencing them.

RQ1—Scope: How do applications support food promotion and healthier nutrition? Which are the food categories that are displayed by the applications? What is the content orientation of the applications?

Regarding food promotion, Blippar seems to have established a standard procedure for food brand promotion. A lot of AR applications in food brand promotion have been implemented by Blippar and the procedure usually starts from a visual, image-based marker on a package, which ignites an interactive entertainment procedure, a game, or animated content. The app provides some nutritional and other information on the product and usually ends with a contest or other reward. Additional content apart from those promoted from brand includes sports, music, and touristic information. Blippar and most companies that have produced AR applications for food promotion, (e.g., Zappar, Augment, etc.) deal with food promotion as one of the sectors they operate. Usually, they demand that their generic AR application is installed in users’ devices for them to activate the AR experience. Most applications exploit AR services by Google and Apple in their implementations.

In food analysis, the increasing need of people for healthier nutrition has inspired research on visualizing ingredients of food and providing feedback on healthy food choices. This information is present in some promotion applications (e.g., Scott’s) too. In research, food colors and sizes (e.g., Meta Cookie) and nutritional information (e.g., Augment, AR gauge meter) are evaluated on how they affect taste perception, which affects food choice. Usually, research applications are based on custom implementation (e.g., Meta Cookie) and open-source technologies, and do not provide a deliverable that may be further exploited. New technologies are not always sufficing for providing more satisfactory interactions to end users. For example, in the Augment app, it has been evaluated that the AR version of nutritional information was not preferred against more traditional visualizations, as the latter were more recognizable and understandable. As discussed in Section 5, proper app design is crucial to its success and should be in accordance with user needs.

Various kinds of food and their representation are the objects of applications, such as sealed foods (e.g., Safe Food AR), packaged foods (e.g., Nestle Cereal), food pictures (e.g., AR gauge meter), raw foods (e.g., Meta Cookie), served foods (e.g., Le Petit Chef), nonalcoholic drinks (e.g., Boost), alcohol (e.g., 19 Crimes), food based on religious beliefs (e.g., Shalal) and food labels (e.g., ARThaiMalay). Regarding the content of the applications, 20 out of 34 contain entertainment, infotainment and games, 11 applications focus on nutrition and food perception, whereas 3 of them display mainly the menu of restaurants.

RQ2—Audience: What is the audience of the applications? What is the appeal of the applications? To what extent do end-users embrace them?

Applications address mainly customers of certain branded products (e.g., 19 Crimes, Nesquik, Boost), food consumers on the move (e.g., M&M arcade, Coca-Cola magic, Magnum, Snack University), in restaurants (e.g., McDonald’s, Le Petit Chef), who are cautious about food quality (AR gauge meter, Meta cookie), or who just want to entertain themselves (e.g., Lobster, Papaya). An interesting outcome of app presentations is a trend towards multiple audiences in the same application. For example, Scott’s addresses kids for entertainment and their parents for providing the nutritional information of vitamins. On the other hand, both promotion applications (e.g., 19 Crimes, Patrón Experienc‪e) and research applications (e.g., Augment, Snack University) may offer multipurpose content, usually combining entertainment with information.

The appeal of an application, either if this is translated to number of downloads, rates in online stores, or content derived from an evaluation process, is an important medium through which end-users express their satisfaction while using the application. It is worth noting that applications that offer pure entertainment content were rewarded thanks to their attractive design (e.g., Bubble Tea), while other applications boosted sales, as a purchase was necessary to complete the experience (e.g., Nesquik). Other applications embraced and gained from a seasonal event (e.g., Cadbury). Positive feedback leads to stronger user engagement with an application, which proves its value, and indirectly with the underlying food product.

RQ3—Technology and Cost: What kind of devices are employed for executing the applications? And in what kind of settings? Which are the most common development platforms? What is the cost of using the applications? Most applications employ a mobile device for providing augmented content through the device’s screen and by exploiting its camera. The use of HMD (e.g., Meta cookie), glasses (e.g., Google Glass) and projectors (e.g., Le Petit Chef), though, provide a more immersive interaction to users, as their visual field is not limited by the display size of a device. Regarding AR functionality, almost all the presented applications exploit it by employing a visual marker just for initiating the experience, and then they provide interaction and content solely by the device that initiated the experience. Thankfully, for example, Magnum app with hovering flavor toppings and partly Meta Cookie with cookies continue their interaction by placing active content around the user’s visual field.

The environment context is also important in food related AR applications. Most applications that are executed in house are usually activated with product packages. Open air applications are more interesting marker-wise, as these may include bus stops (e.g., Coca-Cola), plates (e.g., Le Petit Chef), bar surfaces (e.g., Patrón Experienc‪e) and advertisements (e.g., M&M’s ARcade). Even more interesting interactions stem from apps that include multiple users (e.g., Peached, Snack University etc). Hospitality applications that provide ordering services while displaying entertainment content seem to have invested a lot in the artistic design of their apps, leading to attractive results (e.g., Le Petit Chef, Sketch). Attractive design, high interactivity levels and rich multimedia content ensure the high appeal of an application, as proven by the appeal of Eats in a Blink, Peached, Coca-Cola Magic, and 19 Crimes.

The most commonly used development platforms employed by the presented applications are Blippar (8/34), Instagram (3/34), Vuforia (3/34), and Zappar (2/34), and there are 7 custom implementations, mostly by research-based applications.

Regarding cost, although most applications are free, in order for users to interact with them, they should have bought a product (e.g., Kellogg’s), or be at a specific place (e.g., Le Petit Chef, Coca-Cola, Subway), and they are encouraged to stick with the brand in order to complete their experience (e.g., Eats in a Blink, Nesquik).

5. Proposals

Inspired from the review of applications related to food promotion and analysis and related work presented in Section 1 and Section 2, in this section, some proposals are presented concerning context, content, markers, devices, and the design and implementation processes of the applications. The aim of these proposals is to trigger the improvement of existing and the development of new applications for food products, which will lead to higher engagement with users and cover their food choice needs.

Regarding context, applications should have a constant, basic functionality and content regarding information and/or entertainment according to their purpose. Table 1 shows the basic functionality orientation of existing applications. This functionality could be refreshed to sustain the audience’s attention, in the form of a seasonal event (as in Cadbury and McDonald’s), if it concerns a promotion application (as in Great American bacon cheeseburger), or for distributing new scientific data on food ingredients in case of an analysis application (e.g., Boost). Additionally, most current applications cover only some parts of the food processing sequence. However, as seen in presented applications (e.g., Shalal, AR Gauge Meter, Meta Cookie, Kiosk) and reviews [4,6], consumers are concerned about the origin of the food that they consume, along with their nutritional value. Therefore, a streamlined procedure can be defined that covers all aspects of food processing until its consumption, starting from the origin of raw materials employed in food production, the nutritional information, their inclusion in a sealed food product or in a restaurant menu, and their promotion through advertisement and infotainment material (e.g., gaming, animation, contests, etc.) where the previous information is available.

Related to context, the content of these applications should provide all appropriate information, such as the nutritional value of ingredients, fair trade practices for the production and distribution of raw materials, production procedures, and qualities of provenance areas (as in Kiosk). This content may appear in sealed food promotion and hospitality applications. Food ingredients are usually related to dietary information such as sugar, salt, fat, etc. (e.g., AR Gauge meter). Even in entertainment applications, as already presented (e.g., Boost), this kind of information can be propagated. It may be provided as visual hints (e.g., hovering) throughout the application, that may be further elaborated with material inside the application or links to outside sources.

Rather unsurprisingly, most research applications (e.g., AR gauge meter, Augment app, Safe food AR, Google Glass app) propose new methods for displaying nutritional information and evaluating how food is perceived in terms of taste, smell and appearance (e.g., Me). On the other hand, most promotion applications intend to strengthen brand acknowledgement (e.g., Coca Cola magic). Some of these applications too, provide, both on sealed foods (e.g., Boost) and on menus (e.g., Boston Pizza), nutritional information and food origin. Food promotion applications should embrace the functionality and content of research applications for providing food quality and origin information along with healthy tips more. The whole app functionality should follow suggested strategies for sustainable production and consumption, such as the European Union Initiative “From Farm To Fork” [87].

Moving to technical issues, an essential feature of all AR applications is the marker. In most applications, image-based markers have been employed, while in a few older applications (Nestle cereal and Snack university), QR-code type generic markers have been used. A few applications have based their markers on parts of the real environment, e.g., real food in Google Glass app. Modern AR applications for food should emphasize marker-less solutions, so that user experience is closer to natural operations. For example, [88] concludes that the probability of recommending a displayed brand is higher under marker-less AR than marker-based. For the latter to succeed, sophisticated image recognition techniques are required in order to cope with size, shape and ambient light changes when targeting a product. As already discussed in most presented applications (e.g., Nesquik, Le petit chef, to name two differently oriented apps), makers are used for just initiating the interactive experience, which unfolds without taking into account the surrounding environment anymore. Interaction is often limited inside the application, often running on a mobile device, without providing any other augmented experience. In order to exploit the full potential of AR, applications should offer multiple markers for activating various parts of the application’s content and interactions. Additionally, content presentation should blend more tightly with the surrounding environment, allowing the user to be immersed in the content. Of the presented applications, only some pure entertainment transformation (e.g., Peached), research (e.g., Perceived fullness evaluation through a picture-based AR app) and promotion (e.g., Coca-Cola magic) applications manage to immerse their users in the experience even partially. This trend is proposed to be strengthened in informative applications. Finally, none of the presented applications exploit location-based AR. Popular games such as Pokémon Go have familiarized end users with the notion of an experience activated when in a certain place and direction. This activation type may be exploited, especially in hospitality applications inside and outside of dining and areas and food promotion points.

Devices used in AR applications should not be limited to mobile phones. Some of the research applications presented employ HMDs and projecting devices that provide a more immersive experience. Hospitality applications also rely on projection devices, resulting in a more natural experience, where users do not depend on holding or wearing device in order to experience the AR content. The rest of applications employ mobile devices (phones and tablets) and computers. The positive impact of these devices that offer more immersive experience, either on a restaurant (e.g., Le petit chef) or during research (e.g., Meta Cookie) should be favored in the future, as these devices become more common among consumers.

Concerning the design of applications, especially in promotion and hospitality areas, these should follow a uniform look and feel that enforces brand acknowledgement (as already happening, mainly in presented Blippar apps). In other words, an AR application should not be overwhelmed with interface features that are strange to the whole brand philosophy (as happening early implementations), but these should be in accordance with it, regardless of the application platform, operating system, and other technological parameters in general. Focus should be on enhancing animation, interaction, and exploiting multiple markers, as already discussed. An application should be addressed to multiple audiences as discussed in Section 4, by differentiating some of the content offered to audience categories visually and interaction-wise. As presented in some applications (e.g., Peached), group experiences, embracing multiple users, strengthen the AR experience engagement, and they should be favored. The latter can be combined with sharing options of the outcome of the experience, or for inviting other users to join it.