Towards an Accessible Platform for Multimodal Extended Reality Smart Environments
Abstract
:1. Introduction
2. Related Technologies and Conceptual Frameworks
2.1. Automation Applet Creation Platforms
2.2. Digital Technologies for Platform Ecosystems
3. Context and Considerations
3.1. Harnessing Social Resources for a Profound Digital Transformation
3.2. Characteristics of a Platform for a Digitally Enhanced Life Experience
4. The DEMOS Prototype Platform as Proposed Solution
4.1. User Experience as Creator and Explorer
4.1.1. Application Creation Interface
4.1.2. Application-Exploration Interface
4.2. The Backend Functionality and Assembled Code
4.2.1. Intra and Inter-Device Communication
4.2.2. Application Builder Compiler
4.3. The Core of the Concept and the DEMOS Platform
4.3.1. Formula-Based Processing (Numerical Node)
4.3.2. Decision-Based Processing (Nominal Node)
5. Testing the DEMOS Prototype Platform
5.1. ConfiDrive Architecture
5.2. MusExp Architecture
5.3. Application Rating after Experiencing the DEMOS Prototype Platform
6. Testing the Conceptual Framework
6.1. Experiment for Assessing the Platform’s Concept and Pattern Affordance
6.2. Survey for Characteristics of a Platform-Based Sustainable Digital Ecosystem
7. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Parmentola, A.; Petrillo, A.; Tutore, I.; De Felice, F. Is blockchain able to enhance environmental sustainability? A systematic review and research agenda from the perspective of Sustainable Development Goals (SDGs). Bus. Strategy Environ. 2022, 31, 194–217. [Google Scholar] [CrossRef]
- Anshari, M.; Syafrudin, M.; Fitriyani, N.L. Fourth Industrial Revolution between Knowledge Management and Digital Humanities. Information 2022, 13, 292. [Google Scholar] [CrossRef]
- Van Dijk, J. The Digital Divide; John Wiley & Sons: Hoboken, NJ, USA, 2020. [Google Scholar]
- Bran, E.; Bautu, E.; Popovici, D.M. Open Affordable Mixed Reality: A Manifesto. In Proceedings of the 2020 International Conference on Development and Application Systems (DAS), Suceava, Romania, 21–23 May 2020; pp. 177–184. [Google Scholar] [CrossRef]
- Bran, E.; Bautu, E.; Popovici, D.M. Towards a sustainable future: Ubiquitous knowledge mixed reality museum. Procedia Comput. Sci. 2020, 176, 2878–2885. [Google Scholar] [CrossRef]
- Bran, E.; Bautu, E.; Sburlan, D.F.; Puchianu, C.M.; Popovici, D.M. Ubiquitous Computing: Driving in the Intelligent Environment. Mathematics 2020, 9, 2649. [Google Scholar] [CrossRef]
- Nadoleanu, G.; Staiculescu, A.R.; Bran, E. The Multifaceted Challenges of the Digital Transformation: Creating a Sustainable Society. Postmod. Open. 2022, 13, 300–316. [Google Scholar] [CrossRef] [PubMed]
- Fuster Morell, M.; Espelt, R.; Renau Cano, M. Sustainable platform economy: Connections with the sustainable development goals. Sustainability 2020, 12, 7640. [Google Scholar] [CrossRef]
- De Filippi, P.; Mannan, M.; Reijers, W. Blockchain as a confidence machine: The problem of trust & challenges of governance. Technol. Soc. 2020, 62, 101284. [Google Scholar] [CrossRef]
- Nair, M.M.; Tyagi, A.K.; Sreenath, N. The future with industry 4.0 at the core of society 5.0: Open issues, future opportunities and challenges. In Proceedings of the 2021 International Conference on Computer Communication and Informatics (ICCCI), Coimbatore, India, 27–29 January 2021; pp. 1–7. [Google Scholar] [CrossRef]
- Sharma, B.; Obaidat, M.S. Comparative analysis of IoT based products, technology and integration of IoT with cloud computing. IET Netw. 2020, 9, 43–47. [Google Scholar] [CrossRef]
- Kim, J.C.; Laine, T.H.; Åhlund, C. Multimodal interaction systems based on internet of things and augmented reality: A systematic literature review. Appl. Sci. 2021, 11, 1738. [Google Scholar] [CrossRef]
- Sendari, S.; Firmansah, A.; Aripriharta. Performance analysis of augmented reality based on vuforia using 3d marker detection. In Proceedings of the 2020 4th International Conference on Vocational Education and Training (ICOVET), Malang, Indonesia, 19 September 2020; pp. 294–298. [Google Scholar] [CrossRef]
- Joshi, A.V. Amazon’s machine learning toolkit: Sagemaker. In Machine Learning and Artificial Intelligence; Springer: Cham, Switzerland, 2020; pp. 233–243. [Google Scholar] [CrossRef]
- Torres, D.; Dias, J.P.; Restivo, A.; Ferreira, H.S. Real-time feedback in node-red for iot development: An empirical study. In Proceedings of the 2020 IEEE/ACM 24th International Symposium on Distributed Simulation and Real Time Applications (DS-RT), Prague, Czech Republic, 14–16 September 2020; pp. 1–8. [Google Scholar] [CrossRef]
- Liu, L.; Bahrami, M.; Chen, W.P. Automatic generation of ifttt mashup infrastructures. In Proceedings of the 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), Melbourne, VIC, Australia, 21–25 September 2020; pp. 1179–1183. [Google Scholar]
- Kim, J. Advertising in the Metaverse: Research agenda. J. Interact. Advert. 2021, 21, 141–144. [Google Scholar] [CrossRef]
- Tanwar, S. Blockchain Revolution from 1.0 to 5.0: Technological Perspective. In Blockchain Technology; Springer: Singapore, 2022; pp. 43–61. [Google Scholar] [CrossRef]
- Kubek, M.; Unger, H. WebEngine Version 1.0: Building a Decentralised Web Search Engine. In The Autonomous Web; Springer: Cham, Switzerland, 2022; pp. 35–49. [Google Scholar] [CrossRef]
- Wang, Q.; Li, R.; Wang, Q.; Chen, S. Non-fungible token (NFT): Overview, evaluation, opportunities and challenges. arXiv 2021, arXiv:2105.07447. [Google Scholar]
- Hirniak, J. Research-Driven Cardano DEX White Paper v1. 2021. Available online: https://docs.maladex.com/whitepaper.pdf (accessed on 19 July 2022).
- Vallas, S.; Schor, J.B. What do platforms do? Understanding the gig economy. Annu. Rev. Sociol. 2020, 46, 273–294. [Google Scholar] [CrossRef]
- Comunità, M.; Gerino, A.; Lim, V.; Picinali, L. Design and evaluation of a web-and mobile-based binaural audio platform for cultural heritage. Appl. Sci. 2021, 11, 1540. [Google Scholar] [CrossRef]
- Xu, M.; Ng, W.C.; Lim, W.Y.B.; Kang, J.; Xiong, Z.; Niyato, D.; Yang, Q.; Shen, X.S.; Miao, C. A full dive into realizing the edge-enabled metaverse: Visions, enabling technologies, and challenges. arXiv 2020, arXiv:2203.05471. [Google Scholar]
- Van Zanten, J.A.; Van Tulder, R. Beyond COVID-19: Applying “SDG logics” for resilient transformations. J. Int. Bus. Policy 2020, 3, 451–464. [Google Scholar] [CrossRef]
- Maavak, M. Horizon 2030: Will emerging risks unravel our global systems? Salus J. 2021, 9, 2–17. [Google Scholar]
- Kovács, T.Z.; Bittner, B.; Huzsvai, L.; Nábrádi, A. Convergence and the Matthew Effect in the European Union Based on the DESI Index. Mathematics 2022, 10, 613. [Google Scholar] [CrossRef]
- McLennan, M. The Global Risks Report 2021, 16th ed.; World Economic Forum: Cologny, Switzerland, 2021; Available online: https://www3.weforum.org/docs/WEF_The_Global_Risks_Report_2021.pdf (accessed on 18 July 2022).
- Aly, H. Digital transformation, development and productivity in developing countries: Is artificial intelligence a curse or a blessing? Rev. Econ. Political Sci. 2020. [Google Scholar] [CrossRef]
- Li, Z.N.; Drew, M.S.; Liu, J. Augmented Reality and Virtual Reality. In Fundamentals of Multimedia; Springer: Cham, Switzerland, 2021; pp. 737–761. [Google Scholar] [CrossRef]
- Jones, S.; Dawkins, S. The sensorama revisited: Evaluating the application of multi-sensory input on the sense of presence in 360-degree immersive film in virtual reality. In Augmented Reality and Virtual Reality; Springer: Cham, Switzerland, 2018; pp. 183–197. [Google Scholar] [CrossRef]
- Kaivo-oja, J.; Lauraeus, T.; Knudsen, M.S. Picking the ICT technology winners-longitudinal analysis of 21st century technologies based on the Gartner hype cycle 2008–2017: Trends, tendencies, and weak signals. Int. J. Web Eng. Technol. 2020, 15, 216–264. [Google Scholar]
- Rose, M. The immersive turn: Hype and hope in the emergence of virtual reality as a nonfiction platform. Stud. Doc. Film. 2018, 12, 132–149. [Google Scholar] [CrossRef]
- Weiser, M. The computer for the 21st century. ACM SIGMOBILE Mob. Comput. Commun. Rev. 1999, 3, 3–11. [Google Scholar] [CrossRef]
- Bhattacharyya, S.S.; Kumar, S. Study of deployment of “low code no code” applications toward improving digitization of supply chain management. J. Sci. Technol. Policy Manag. 2021. [Google Scholar] [CrossRef]
- Upadhyay, A.; Mukhuty, S.; Kumar, V.; Kazancoglu, Y. Blockchain technology and the circular economy: Implications for sustainability and social responsibility. J. Clean. Production. 2021, 293, 126130. [Google Scholar] [CrossRef]
- Javed, F.; Antevski, K.; Mangues-Bafalluy, J.; Giupponi, L.; Bernardos, C.J. Distributed Ledger Technologies for Network Slicing: A Survey. IEEE Access 2022, 10, 19412–19442. [Google Scholar] [CrossRef]
- Valadares, D.C.G.; Will, N.C.; Caminha, J.; Perkusich, M.B.; Perkusich, A.; Gorgônio, K.C. Systematic literature review on the use of trusted execution environments to protect cloud/fog-based Internet of Things applications. IEEE Access 2021, 9, 80953–80969. [Google Scholar] [CrossRef]
- Pilkington, M. Blockchain technology: Principles and applications. In Research Handbook on Digital Transformations; Edward Elgar Publishing: London, UK, 2016. [Google Scholar] [CrossRef]
- Curran, G.; Gibson, M. WikiLeaks, anarchism and technologies of dissent. Antipode 2013, 45, 294–314. [Google Scholar] [CrossRef]
- Saran, S. Technology: Digital epiphany? COVID-19 and our tech futures. In Challenges and Opportunities in the Post-COVID-19 World; World Economic Forum: Cologny, Switzerland, 2020; pp. 24–27. Available online: https://www3.weforum.org/docs/WEF_Challenges_and_Opportunities_Post_COVID_19.pdf (accessed on 19 July 2022).
- Hanjra, M.A.; Qureshi, M.E. Global water crisis and future food security in an era of climate change. Food Policy 2010, 35, 365–377. [Google Scholar] [CrossRef]
- Webster, K. A circular economy is about the economy. Circ. Econ. Sustain. 2021, 1, 115–126. [Google Scholar] [CrossRef]
- Türkeli, S.; Schophuizen, M. Decomposing the complexity of value: Integration of digital transformation of education with circular economy transition. Soc. Sci. 2019, 8, 243. [Google Scholar] [CrossRef]
- Semanjski, I.; Lopez Aguirre, A.J.; De Mol, J.; Gautama, S. Policy 2.0 platform for mobile sensing and incentivized targeted shifts in mobility behavior. Sensors 2016, 16, 1035. [Google Scholar] [CrossRef]
- Bada, A.O.; Damianou, A.; Angelopoulos, C.M.; Katos, V. Towards a green blockchain: A review of consensus mechanisms and their energy consumption. In Proceedings of the 17th International Conference on Distributed Computing in Sensor Systems (DCOSS), Pafos, Cyprus, 14–16 July 2021; pp. 503–511. [Google Scholar] [CrossRef]
- Dann, D.; Peukert, C.; Martin, C.; Weinhardt, C.; Hawlitschek, F. Blockchain and Trust in the Platform Economy: The Case of Peer-to-Peer Sharing. Wirtschaftsinformatik 2020, 1459–1473. [Google Scholar] [CrossRef]
- Manda, M.I.; Ben Dhaou, S. Responding to the challenges and opportunities in the 4th Industrial revolution in developing countries. In Proceedings of the 12th International Conference on Theory and Practice of Electronic Governance, Melbourne, VIC, Australia, 3–5 April 2019; pp. 244–253. [Google Scholar] [CrossRef]
- Bernal-Cárdenas, C.; Moran, K.; Tufano, M.; Liu, Z.; Nan, L.; Shi, Z.; Poshyvanyk, D. Guigle: A gui search engine for android apps. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), Montreal, QC, Canada, 25–31 May 2019; pp. 71–74. [Google Scholar] [CrossRef] [Green Version]
- Xu, C.; Wang, Y.; Quan, W.; Yang, H. Multi-person collaborative interaction algorithm and application based on HoloLens. In Recent Trends in Intelligent Computing, Communication and Devices; Springer: Singapore, 2020; pp. 303–315. [Google Scholar] [CrossRef]
- Yang, A.M.; Li, S.S.; Ren, C.H.; Liu, H.X.; Han, Y.; Liu, L. Situational awareness system in the smart campus. IEEE Access 2018, 6, 63976–63986. [Google Scholar] [CrossRef]
- Fatima, S.A.; Hussain, N.; Balouch, A.; Rustam, I.; Saleem, M.; Asif, M. IoT enabled smart monitoring of coronavirus empowered with fuzzy inference system. Int. J. Adv. Res. Ideas Innov. Technol. 2020, 6, 188–194. [Google Scholar]
- Butakova, M.A.; Chernov, A.V.; Guda, A.N.; Vereskun, V.D.; Kartashov, O.O. Knowledge representation method for intelligent situation awareness system design. In International Conference on Intelligent Information Technologies for Industry; Springer: Cham, Switzerland, 2018; pp. 225–235. [Google Scholar] [CrossRef]
- Gajdošík, T. Towards a conceptual model of intelligent information system for smart tourism destinations. In Software Engineering and Algorithms in Intelligent Systems; Springer: Cham, Switzerland, 2018; pp. 66–74. [Google Scholar] [CrossRef]
- Wilson, K.B.; Karg, A.; Ghaderi, H. Prospecting non-fungible tokens in the digital economy: Stakeholders and ecosystem, risk and opportunity. Bus. Horiz. 2021, 65, 657–670. [Google Scholar] [CrossRef]
- Mercan, S.; Kurt, A.; Akkaya, K.; Erdin, E. Cryptocurrency solutions to enable micropayments in consumer IoT. IEEE Consum. Electron. Mag. 2021, 11, 97–103. [Google Scholar] [CrossRef]
- Bran, E.; Bautu, E.; Popovici, D.M.; Braga, V.; Cojuhari, I. Cultural Heritage Interactive Dissemination through Natural Interaction. In Proceedings of the 16th RoCHI ACM Conference, Bucharest, Romania, 17–18 October 2019; pp. 156–161. [Google Scholar]
Interaction Participants | Action and Purpose | Logical Sequence |
---|---|---|
User’s device, the app | A user will enable and disable their app, with each scan changing the app’s state. - Good for standalone apps activated in a specific location and personal devices listening to external events | A node on the smartphone holds a Boolean value that enables and disables the starter nodes or the nodes listening for external events. The first time the app is enabled it requests the necessary code from the server. |
User’s device, the app | A user will enable and reenable their app, extending the active state while repeatedly scanning the tag placed in proximity. - Good for standalone apps and personal devices listening and producing events only while present in a specific place | After scanning the tag, a node on the smartphone holds a Boolean value with specific volatility, which automatically disables the node when reaching timeout. |
User’s device, the app | When scanning an NFC tag, a user will trigger an event within the app using multiple tags spread throughout the environment. - Good for tours, XR, and IoT apps that map the natural environment | Through the information present within the scanning tag, the event triggers an action such as visualizing particular content or moving a state machine to the next step (just as any other connected node works) by letting the tag information flow forward. |
First user’s device, one shared device, second user’s device | The first user will connect to the shared device. When the second user connects to the device, they will disconnect the first user. - Good for apps that require a single user to control a device in a shared environment | When scanning the tag, the first user’s device holds the timestamp and sends it to the shared device. All events coming from them have the timestamp information and pass the shared device’s filter until the second user scans the tag, and the device will filter by the updated timestamp. |
First user’s device, one shared device, second user’s device | The first user will connect to the device, then the second user will also connect, and on the device their inputs will be merged. - Good for apps that require the collaboration of multiple users for the same task | When the users scan the tag, they get a timestamp for identification, which is known by their device and the shared device as well. Their inputs are then merged with the help of a formula-based processing node on the shared device. |
First user’s device, one shared device, second user’s device | The first user will connect to the device, then the second user will also connect, and each user will control a specific component. - Good for apps that require the collaboration of multiple users executing distinct tasks | The users will each scan different tags in order to enable distinct components of the distributed application and collaborate using distinct input or output components. |
Name | Input | Output |
---|---|---|
Numerical node | Simple (plain data/object) | Complex (formula statement) |
Nominal node | Complex (IF statement) | Simple (plain data/object) |
General node | Complex (IF statement) | Complex (formula statement) |
Application Description | Code Resources | Processing Nodes | Comments |
---|---|---|---|
Application for notifying you a few hours before the rain will start |
|
| The application was first confused with an IFTTT app with no processing. There should be a decisional node for classifying weather. The application is a good fit for the platform. |
Application for organizing school billboards and wall art according to your school class, current events, and personal preferences |
|
| The application is a good fit for the platform, using the implemented template for no-code programming. |
Application for notifying your partner with vibrations on a smart bracelet/jewelry that you are trying to reach them while their phone is muted |
|
| The application is a simple IFTTT app that may be also designed with current platforms on a smartphone. The smart device needs to be browser compatible with functioning on the current architecture. |
Application for donating money to a cause in an amount directly proportional to screen time on social-media apps |
|
| The application is a good fit for the platform. The screen-time-measuring method was not included in the logic, but it may be done by accessing social media with an NFC scan and scanning another tag to stop monetizing. A future platform should be bound to a digital wallet. |
Application for a smart refrigerator that alerts the user of approaching expiry dates and low quantities of staple food |
|
| The application is a good fit for the platform. It was said to be inspired by the pattern of connecting the mobile device to the screen in the museum application. Here a date is merged with the initial weight and then constantly checked. |
An application for house or private-property safety against burglars or stalkers who stay still for more than 5 min |
|
| The application is a good fit for the platform. Another pattern exposed in the presentation was implemented, which is slightly more complex in requiring a feedback loop. |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bran, E.; Nadoleanu, G.; Popovici, D.-M. Towards an Accessible Platform for Multimodal Extended Reality Smart Environments. Information 2022, 13, 439. https://doi.org/10.3390/info13090439
Bran E, Nadoleanu G, Popovici D-M. Towards an Accessible Platform for Multimodal Extended Reality Smart Environments. Information. 2022; 13(9):439. https://doi.org/10.3390/info13090439
Chicago/Turabian StyleBran, Emanuela, Gheorghe Nadoleanu, and Dorin-Mircea Popovici. 2022. "Towards an Accessible Platform for Multimodal Extended Reality Smart Environments" Information 13, no. 9: 439. https://doi.org/10.3390/info13090439