Smartphone-Enabled Interaction on Large Displays—A Web-Technology-Based Approach

Abstract

This paper proposes the design and implementation of a system for simultaneous communication of several users over a single display, such as large public displays or video walls. Motivated by the lack of software frameworks that are both free and suited for the desired level of complexity, we present a novel, out-of-the-box solution for collaborative and remotely controlled application use. The solution is based exclusively on web technologies, thus significantly facilitating usability and reach, and eliminating the need for additional software installations or the use of additional devices. The architecture of the developed solution is modular and comprises three main components: (1) the environment, (2) the remote controllable application (RCA), and (3) the controller. The environment serves as a central hub that manages the data flow and ensures synchronization between the user devices. The RCA, which runs on the large display, serves as the interface for interacting with the content, while the controller, which is loaded on the users’ smartphones, enables control and interaction with the RCA in real time. By using web sockets and a server–client architecture, the system supports device- and platform-independent remote control and centralized collaboration through an off-the-shelf communication framework entirely flexible regarding client application. In order to demonstrate the possibilities of the proposed system, several different scenarios were covered, and solutions were implemented and tested on a large display in the faculty lobby.

1. Introduction

With the advancement of technology, the ubiquity of computers in everyday life and the more affordable prices of certain components, such as LCD and LED displays, large, often promotional or informational displays have become a common sight in many open spaces. These range from shopping malls, banks, and various government institutions to schools or college lobbies. Most of these devices, especially those of larger formats, are noninteractive or unidirectional, meaning they display specific preprepared content, such as announcements, information, or advertisements, usually in a loop. Smaller devices are sometimes interactive, allowing communication through touch interfaces on the screens. However, for larger screens, this is neither practical, because of their size, nor feasible, as they are generally not touch-enabled. Over the past decade, there has been a considerable amount of research and proposals on this topic, but none of the suggested solutions have gained significant traction. Some of the proposed solutions involved additional communication hardware, which is often impractical and requires regular maintenance, such as repairs, battery changes, etc. The majority of the solutions, therefore, centered around the idea of gesture communication or the use of smartphone devices as a sort of remote controller. The latter approach has proven to be the most straightforward, as smartphones are widely available, connected to the internet, equipped with various sensors, and can be utilized for communication. Due to ease of use and the ability to access all smartphone components and sensors, some proposed systems utilize specially developed mobile applications.

We argue that this is not the optimal approach because it requires users to make an additional effort to install a new application, delaying the start of use. This delay could be a sufficient reason for someone to abandon the effort and usage, particularly in public places where users often spend a short amount of time. Furthermore, each solution typically requires a custom application, which may not work with the next device or, at best, needs reconfiguration to work with a different large screen.

Therefore, we propose a contactless solution (crucial in times of heightened awareness of potential disease transmission through touching shared objects in public spaces), entirely based on web technologies, eliminating the need for installing additional software on smartphone devices. Additionally, configuration, control, and communication are coordinated automatically, without the developers needing to bother beyond pure controlled- and controller-application development. This paper describes the architecture and structure of the proposed system, providing a detailed explanation of all modules, communication between modules, and the structure of exchanged messages. The practical implementation of the system will be demonstrated through three case studies illustrating distinct use cases, i.e., showcasing examples of games and applications where multiple users simultaneously interact with the system, whether playing a game together or independently performing other tasks.

It is important to point out that the system we developed represents a pioneering approach that distinguishes itself as the first open-source alternative in the field of such technologies. Unlike proprietary commercial services, this solution has the unique benefit of being both lightweight and versatile, making it ideal for seamless integration into any existing application framework. The key feature of its adaptability lies in its capacity to be easily integrated ‘off-the-shelf’, which means that it can be used immediately without the need for extensive customization. These distinctive features are further enhanced by the system’s ability to be customized to different interfaces on mobile devices, allowing users to easily operate applications from a distance. This flexibility is especially advantageous because it allows a wide range of applications to be remotely controlled, opening up an extensive range of opportunities for innovation in user interaction with advanced web systems used in e-learning.

The remainder of this paper is structured as follows: Section 2 provides a detailed overview of related work, sets the context, and highlights the gaps in current approaches to smartphone-based interactions with large displays. In Section 3, we address the system architecture and organization, and provide a comprehensive overview of our proposed solution, including its key components and functional mechanisms. In Section 4, we present several case studies that show real-world applications of our system and its effectiveness in improving user interaction. Then, Section 5 provides a critical discussion of the results and evaluates the system’s performance, user experience, and areas for improvement. Finally, Section 6 concludes this paper with a summary of our findings for future research and a brief discussion of the wider implications of our work in the field of large-scale interactive visual technologies.

2. Related Work

With the increasing number of large-screen implementations in various public spaces, many research papers and projects have explored the possibilities and user interactions with these displays. Researchers have recognized the underutilized potential of large screens, commonly used only as passive public information boards, prompting efforts to transform them into more lively and interactive devices. In their paper, Alt et al. presented a public-display classified ads app, and discussed smartphone-based interaction techniques for communication and content exchange, concluding that no single technique is universally optimal for creating and exchanging content for big screens [1].

One direction of big screen research focuses on technical challenges, such as communication speed and network latency [2,3], which is a level below the scope of this research. Some researchers have delved into various communication techniques and forms, ranging from wired devices to custom controllers and the use of gestures and motion tracking through embedded cameras or sensors [4,5,6].

A significant portion of research has been dedicated to leveraging smartphones as remote controls for large screens [7,8,9,10], which is the direction undertaken in this paper. For instance, the authors of the paper titled Designing a Big Screen Interaction Based on Smartphone Touch Gestures [11] conducted various experiments with smartphone touchscreen gestures for big screen interaction, and while it shows promising potential, the current design lags behind classic mouse-based interactions. In a survey titled “When smart devices interact with pervasive screens” [12], the authors state that the convergence of pervasive screens and smart devices has given rise to screen-smart device interaction, facilitating numerous innovative interactive use cases through established data communication links. Some applications used a combination of mobile- or computer-tablet-based applications and applications for big screens, for example, to show live results in class educational games [13]. Vosinakis and Anastassakis [14] developed a system that uses mobile device as a controller in order to create a COVID-safe interaction during the pandemic, concluding that their solution and similar systems have a lot of potential to become safe and practical alternatives to common input devices in public spaces. In other papers, authors researched different forms of public-screen interaction using mobile devices and different possible usage scenarios [15,16]. The latest research focuses on newly trending technologies such as augmented reality (AR), which is also employed in the exact same scenario our solution is concerned with (wall-displayed application including remote device-based collaboration) [17]. Another study combined smartphones, Bluetooth microphones, and Kinect sensors for large screen interaction [18], while Huang et al. [19] developed a huge 60-screen display for data visualization and developed an interaction framework with gestures for it. Applications such as these would greatly benefit from the proposed solution in terms of automatized communication and out-of-the-box device-application adaptation. Solutions such as [20] provide infrastructure for remote device-controlled virtual environment collaboration but are nevertheless more specialized in comparison to ours and therefore limited in use-case scope.

Simultaneously, the use of mobile phones as remote or game controllers extends beyond large screens, with many applications replacing traditional remote controls for home TV devices, game console joypads, and systems like Airconsole (https://www.airconsole.com/, accessed on 26 February 2024). Torok et al. [21] developed a customizable game controller app that dynamically rearranges its layout based on user input, playing style, current level, or game screen. Vargas et al. [22] proposed a smartphone-based virtual control for computer games that dynamically adjusts its interface for improved user comfort and experience while maintaining a friendly and visually consistent design. Moreover, CAVRoom [23] is a framework accommodating a multiplatform virtual shared space, sharing striking similarities with ours (client–server architecture, multiplatform flexibility, and client application agnostic approach), showcasing the relevancy of such solutions. However, the listed solutions are either commercial gaming platforms unsuitable for custom development (Airconsole), aimed at only the remote interface [16,17], or specialized in terms of technology ([23], which is based on Unity and Microsoft services), highlighting the lack of a truly off-the-shelf, lightweight, and platform-independent service.

Since the advent of complex application systems, an important concern has been the organization of distributed services such that they are able to cooperate effectively and overcome potential issues arising from their distributed nature [24]. A prominent architecture facilitating the communication between its segments in a centralized manner is the client–server architecture [24,25]. The core idea—i.e., a singular, specialized server application providing services to external clients, who need not be aware of each other, via a customizable interface—has shown to be highly practical in contexts such as the notoriously heterogenous web services accessible to multiple parties simultaneously [26]. Moreover, the pure client–server architecture, as described in [27], allows for an asymmetrically conceptualized network, where solely clients must handle interaction initialization, while the server provides an as-is service based on request responses. Also, while Bass and Clements emphasize the problem of the server’s potential of being a single point of failure or a bottleneck in the pure architecture version, such threats may be mitigated through slight adjustment of the system design (e.g., employing a super-peer organization [28]). To sum up, the client–server architecture appears to significantly ease the development and subsequent functioning of a distributed application system, being especially appropriate in contexts involving a stand-alone service which must cater to multiple external users.

Other directions of research focused on different user tracking modes, to detect motion, attention, and emotion. By using wireless sensors, it is possible to discreetly perform continuous real-time estimation of the attention and emotions of users in front of a video wall. This provides new opportunities for controlling, but also allows the applications to be more personalized, or to create different user profiles and use this information for better experience or educational purposes. In [29,30], the authors developed a prototype of an intelligent e-learning agent that uses a novel set of 22 features to assess student engagement during video lectures. The system uses the OpenPose API or a similar model for human pose estimation and tracking to classify upper body postures and movements in video data, providing insights into student attention, emotion, and participation. The features, which are divided into key-point-based and posture-based categories, are derived exclusively from visual information, which allows for remote, contactless, and nonintrusive monitoring of students and the development of personalized learning experiences. The study shows potential for improving the quality and effectiveness of online education and in other similar educational settings. Also, in [31], as a continuation of the previously mentioned research, the authors propose an adaptive e-learning intelligent agent aimed at optimizing online education during the COVID-19 pandemic but could also be employed in any other environment that enables remote sensing of learners’ physiological signals. This agent assesses students’ engagement and learning curve by analyzing their affective states, behavior parameters, and psychophysiological features through unobtrusive video surveillance and machine learning. The system’s potential to personalize learning and improve pedagogical outcomes is discussed, highlighting the importance of emotion and attention detection in shaping future e-learning environments.

Finally, a number of research papers addressed the designing and modeling of the system architecture or program frameworks, which enable creation of modular and reusable infrastructure to support interactive uses of large displays. Xue and Zhu analyzed different methods of software design for the socket communication between server and clients in the server–client architecture, concluding that good application would have tasks distributed between the two, and a deep understanding of protocols is needed for creation of an efficient system [32]. Furthermore, Bharamble, Pang, and Seshan propose Colyseus, a distributed architecture which enables hundreds of players to simultaneously participate in interactive multiplayer games [33].

3. System Architecture and Organization

The main goal of this paper is the presentation of a novel system designed for interactive communication between smartphones and large displays. This section presents the system’s design, describing its architectural framework and how its components interact. Our approach is based on a modular approach, which allows for flexibility and scalability while ensuring seamless component integration.

Another important aspect of the developed architecture is the use of web technologies, which allow the system to function without the need for specialized hardware or additional software installations. This decision significantly increases the system’s accessibility and usability, both of which are important factors in encouraging user engagement and interaction.

3.1. Overview

The proposed framework aims to provide code support for an easy adaptation of web applications controllable via a stand-alone web interface for usage on a multiplayer server. In particular, the main emphasis is on assuring minimal additional coding and offering a largely predeveloped interface providing a virtually out-of-the-box service.

The proposed structure (illustrated in Figure 1) consists of the following modules:

• Remote controllable application (RCA);
• Controller;
• Environment.

The listed segments are realized as instances of newly introduced RCA, Controller, and Environment classes, respectively, or more precisely, their extending classes for the abstract RCA and Controller. The only programming required is that regarding implementation of a rather small number of application-specific abstract classes and instantiation of the described objects.

The RCA and Controller are remarkably similar, as opposed to each of them compared to the Environment, respectively. Essentially, they are the same thing: an abstract class which, when extended or instantiated, provides automatization of both client socket set-up chores and communication with an Environment instance (more precisely, its server socket) at a set address. They are instantiated on servers running a pre-existing web application. The RCA module represents the application controlled (i.e., interacted with) via a remote device, while the Controller corresponds to a controller GUI. The pre-existing web application interface can pass and receive data regarding socket communication through predeveloped methods.

The Environment is a class containing preimplemented server-socket setup logic, as well as functionality regarding listening to connections and predefined messages from client sockets corresponding to specific RCA and Controller instances. Furthermore, it automatically determines and sends responses, along with passing data from Controller to RCA or vice versa when demanded by the protocol. In other words, it functions as a mediator between an RCA instance and one or many Controller instances, who do not communicate directly and virtually are ignorant of each other. The Environment must be instantiated on a running server, either sharing one with the RCA, Controller, both services, or a separate one.

To further elaborate on the architecture regarding hosting, the module instances are free to run at all different addresses, share the same server, or a combination of both (e.g., RCA and Environment are served at the same host, while Controller and its corresponding web service are not).

The concrete implementation specifics outside of the required abstract method implementations do not concern the framework, which essentially automatizes communication adhering to a strict, noncustomizable protocol. In other words, a concrete RCA instance could be instantiated on an otherwise empty server, or, conversely, be linked to an exceptionally complex web application with which it interacts through the well-defined RCA interface. Analogously, the GUI and logic of the web application that a Controller is instantiated on can be significantly more complex than merely providing buttons and movement detectors, or it can be an empty webpage, in which case it will not report any events—in effect, the same as though no actions have been undertaken through a more elaborate graphic interface. To conclude, extended and instantiated framework objects without any connection to additional web applications or coding beyond basic implementation of abstract methods with empty ones can function as a stand-alone system without any problems—it merely will not be interactive.

3.2. Component Specifics

In the following subsections, we describe in more detail not only the three modules, but also the context that the framework is intended to be incorporated in (illustrated in Figure 2). The code cards in the figure represent main code chunks, all of which can interact provided they run on the same host. Moreover, the object blobs (ConcreteRCA, Environment, and ConcreteController) correspond to concrete object instances existing in memory. Lastly, it might be beneficial to note that the RCA and Controller as socket wrappers also serve as mediators for client applications (main application or remote controller interface) via their abstract classes, through which the developer defines implementation-specific behavior and client–application interaction. The classes were designed to be abstract in order to force definition of implementation-specific behavior, such as the Controller acceptance condition for the RCA or actions triggered by message reception on the Controller. The Environment’s workflow is completely automated and functions exactly the same regardless of context specifics, so it can be instantiated as it is without any extension.

The RCA, Environment, and Controller instances are instantiated and served at either individual, respective servers, only one shared, or a combination of both. Additionally, they are included in a broader context, which for the RCA and Controller typically includes (but does not require) a web application providing a user interface, with which the socket interacts through behavior specified in the abstract method implementations. Each object instance contains a web socket running without any developer intervention—as explained, implementation-specific behavior is defined in extending classes and triggered automatically at predetermined points. As was also stated before, the Environment module only needs to be instantiated as it is, not being an abstract class. Its functionality does not depend on concrete RCA and Controller specifications, as it interacts only with their sockets, whose behavior is also hidden from the developer.

3.2.1. RCA

The RCA, in the narrowest sense, is an abstract class representing an interface mediating the communication between a client web application and the server socket contained in an Environment instance through its own client socket instance. In a broader sense, it encompasses both the abstract class and its context, which includes the aforementioned client application. A detailed illustration of the RCA context is provided on the leftmost side of Figure 2.

The RCA abstract class already contains implemented code regarding instantiation of a client socket instance provided an already known concrete Environment server-socket host address as well as full implementation of the communication—none of which concerns a developer employing the framework.

All interaction with a client application (therefore also its developer, who needs to code only that part) running on the same host as the RCA is through a smaller number of public methods, through the implementation of which one customizes application-specific behavior. Examples of customizable features include predicates for the acceptance of a controller and public functions called in private ones after being triggered by socket events. Due to the fact that implementation specifics need to be specially defined, the RCA class is designed as abstract—thus forcing class extension and their implementation.

Lastly, an RCA-extending class must contain an authentication code, which will be shared with Controllers in some way: e.g., through QR-code scanning, manual input, being already stored in a shared memory, following a hyperlink, etc. The code will serve as a mapping between the RCA and its web application and the Controller, which the Environment will utilize in the case of multiple RCAs being connected to its server socket.

3.2.2. Controller

Similarly to the RCA, the Controller (whose context is displayed on the rightmost part of Figure 2) is an abstract class whose extensions’ instance bridges the gap between a controller interface web application and the Environment’s mediator socket via its own internal client socket. However, extending the Controller class requires less customization than that of the RCA. In particular, the implementations concern already optional functionalities triggered by nonrequired RCA messages. Overall, the core of the Controller interface lies in the calling of already implemented functions, which merely pass data to the Environment, after an undertaken action. The triggering of these functions is defined in the GUI code.

3.2.3. Environment

As opposed to previously described modules, the Environment (see the middle section of Figure 2) is a regular class which must be instantiated in code running on a server. Its instantiation triggers a server-socket set up, whose listening and message emission logic is also entirely defined inside of private class methods. There exists no public interface for potential interaction with external code outside of the receiving of an HTTP server instance that its socket will run on.

The Environment is an RCA–Controller socket-based mediator which ensures adherence to a communication protocol realized through specific, expected RCA and Controller side events. It serves as a traffic regulator, mapping Controllers and RCAs in socket conversation rooms.

Socket communication and their typical behavior are illustrated in Figure 3.

The Environment is a critical component of the system’s architecture, mediating data flow between the RCA and the Controller via socket-based connections. This mediation is more than just communication; it also ensures that both parties strictly conform to an established communication protocol. This protocol is represented by a set of predefined events that both the RCA and the Controller are expected to recognize and handle.

The diagram in Figure 3 illustrates in detail the interaction between an RCA module and Controller-module client sockets and the Environment server socket using a web-socket protocol. There are two client sockets shown, “rca” (RCA) and “ctrlr” (Controller), which represent different types of clients connecting to the server. Each client socket has an “emit” function, which sends messages to the server, such as “hello_from_rca” from the RCA client and “hello_from_ctrlr” from the controller client. The server socket is depicted in the center, with several “on” functions that listen for specific messages from the clients, like “rca_message” and “ctrlr_action”. The server reacts to these messages by emitting responses back to the clients, ensuring two-way communication. It is also capable of directing messages to specific recipients using the “to” method, which targets individual clients based on identifiers. As indicated by the arrows, the flow of messages between clients and server is bidirectional where clients can both send and receive messages. This setup is typical in real-time applications where immediate feedback between the server and multiple clients is necessary.

3.3. Workflow

In this section, we describe the socket communication protocol as pictured on Figure 4. The protocol is triggered by instantiation of an RCA-extended object with an already running Environment server’s address as a parameter. During instantiation, a client socket is automatically set up and connected to the Environment’s server socket. Also, upon introducing a new RCA to the system, it is registered in the Environment with an auth_code so that the Controller can request connection through it. The latter marks the entrance of a Controller instance into the system.

No socket interaction can take place without a running server supporting an Environment instance and its server socket. Without it, connection requests from any client will be unsuccessful.

Given an Environment server socket runs at a publicly available host, individual client sockets may independently send connection requests. To be clearer, an RCA socket connects automatically as it can exist independently in the system. Controllers, however, have to always be linked to an RCA, so they cannot be received into the communication without prior acceptance from an RCA. Finally, an Environment will not by itself decline a connection attempt from either side. In the case of a physical limitation reach, such as server overload, an error will simply be raised.

Nevertheless, under regular conditions, an RCA client socket connects to an Environment server unconditionally, and later decides on acceptance or rejection of a Controller seeking to couple with it via a shared authentication code. The predicament can depend on many variables, of which some examples are the existence of a limit on the number of controllers, dynamically computed resource availability, or some other constraint. If rejected, a Controller instance remains in the state it was before the connection attempt and is free to request it again. On the other hand, if accepted by the RCA and informed of it through the Environment, the Controller module can begin emitting action data and receiving external messages. Additionally, the RCA can assign a temporary id to a newly accepted controller by sending it to the Environment inside of the acceptance alert message, which is then stored on the Environment and mapped to action alerts before emitting them to the RCA.

Information exchange between the RCA and its Controllers, i.e., action alerts from the Controller side together with RCA-emitted messages, operates through the Environment, which serves as a mediator and RCA–Controller mapper. As a result, high component decoupling is achieved, in turn making the system adaptable, flexible, and scalable.

Finally, a standard disconnection protocol begins with Controllers disconnecting while their corresponding RCA still runs unaffected. They do not require permission to disconnect, but the related RCA is informed of the activity and the updated system state.

Conversely, a disconnection request from the RCA side must be specially evaluated on the Environment. Should a Controller mapping to the RCA still be active and connected to the Environment, the proposition will be permitted only after disconnection had been commanded to all such Controllers as well as fully executed and notified of. Only then does an RCA receive the permission to disconnect. Eventually, the Environment instance remains alone in the system and can be shut off at any moment without negative implications.

4. Case Studies

The findings presented in this paper were supported by the implementation and testing of three representative case studies. Together, they demonstrate how the system enables interactive, web-technology-based communication between smartphones and large displays, illustrating its versatility in a variety of user scenarios.

In the first example, the FERillas game demonstrates the system’s ability to engage users in a competitive but collaborative environment. The game’s turn-based nature, combined with the intuitive smartphone controllers, demonstrates the system’s ability to handle sequential user interactions efficiently.

The second case study, Interactive Pong, tests the system’s responsiveness and real-time interaction capabilities. With a greater emphasis on speed and immediate feedback, this example demonstrates the system’s robustness and scalability in a fast-paced, multiuser environment.

Finally, the third case study, the multiuser message board application, examines a more practical application of the system. This example demonstrates the system’s ability to handle dynamic, text-based user inputs and display them in a coherent and organized manner on a large public display. The message board application is a great example of how the system can be used for information and collaboration purposes other than gaming.

4.1. Case Study 1: Turn-Based Game FERillas

As part of the research presented in this paper, a simple turn-based game inspired by the QBasic Gorillas computer game was created. Along with the game, a corresponding smartphone controller (joypad) was developed. The game is intended for two players who join by scanning a QR code. After that, a playfield is generated, consisting of randomly arranged buildings of different heights, with two gorillas placed on them, each in their half of the screen. The players’ mobile devices display the corresponding controller web application—with controls for the game. The goal of the game is to launch a banana by determining the angle and speed of the launch to hit the other player. Players take turns on their smartphones determining the speed and angle of banana launch, attempting to hit the opponent (Figure 5). After a gorilla is hit, the playfield regenerates, placing players in different, randomly selected locations again. The first player to score three hits is the winner. The controller app communicates with the game by exchanging information about the current game state (which player’s turn it is, the current score, the set angle and launch speed, etc.). Upon completion of the game, the application sends a message to the controllers with the final result and game winner information, after which the controllers disconnect from the server.

4.2. Case Study 2: Interactive Pong Game

To assess how well the proposed system performs in a multiuser environment where speed and response time are crucial, another game was developed—this time inspired by the classic computer game Pong. Users enter the game by scanning a QR code, after which a brief countdown ensues, and the game begins. The ball, represented by a small square, moves from one player, represented by a vertical line (paddle), toward the other player, who must deflect it back and forth until the ball “escapes” behind one of the players.

In this game, controllers are implemented as a simple web application with the interface shown in Figure 6. Users control the movement of their player on the game screen by swiping their finger across the phone’s screen. The controller sends its “move” events and new position to the server, while the game informs users about score changes and, ultimately, the winner. After the game concludes, the application deregisters users, who then disconnect from the server.

The game was tested multiple times by different groups of players on a large screen in the hall of the University of Zagreb, Faculty of Electrical Engineering and Computing, and was also used during events such as Open Day or Museum Night (testing illustrated on Figure 7). Open Day and Museum Night are two annual science outreach events at the Faculty that gather the largest number of visitors, approximately 3500–4000 people each. Throughout the entire event, the Pong game was displayed on a large screen in the hall and could be played by visitors passing by. Next to the screen, there was usually a person assisting and explaining to visitors how to start the game (e.g., telling them that they should use their phones, helping to scan QR codes, and similar tasks). However, even when that person was not present, most visitors quickly discovered how to start the game. Players were diverse, ranging from elementary school children to their grandparents, from ICT and technical experts to visitors with very limited knowledge of technology. According to the game logs, over 1200 rounds were played, and users connected either through the university’s Wi-Fi (or available Eduroam system) or via their own mobile internet providers.

Aside from occasional issues (e.g., an incoming toast message taking focus on the user’s mobile phone) or minor display inconsistencies caused by the limitations of the large screen, no significant problems or notable latency were recorded.

Due to the size of the screen, most visitors stood several meters away from it (typically 4–6 m), standing within less than 3 m made the game difficult due to the challenge of seeing the entire screen. On the other hand, players who were more than 6 m away were more often disrupted by passersby walking in front of them, obstructing a portion of the screen.

4.3. Case Study 3: Multiuser Message Board Application

In the third study, to test the specific framework principles presented in this paper, a message board application was also developed. Specifically, its main functionality is based on a remotely controllable web application which works as a dynamic display of short notifications or memos (illustrated in Figure 8) contained in its local memory.

Moreover, the remote controller is realized through a graphic user interface (GUI) offering the functionality of typing in a title and message body text and sending it to the controlled application (see Figure 9). Both services contain a) a web application with a graphic user interface and b) an instance of a class extending RCA and Controller, respectively. Also, both are running on the same host, but on different paths. Finally, the services communicate exclusively through an Environment instance (i.e., via their and the Environment’s respective sockets) instantiated at an entirely different server address and independent of the message board application.

The current version of the message board application was made for framework demonstration purposes and as a real-life proof-of-concept, so the note-posting functionality is implemented at a basic level—the messages simply appear in the order they were received and on random paper color and shape. Positioning and note style will be made more customizable with further application development.

4.4. Lessons Learned

After conceptualizing and developing the framework and implementing three use cases to gain a better understanding of the system’s functionality in a real-world environment, as well as its user friendliness, some valuable lessons were learned. Firstly, all applications, especially games, received positive user reactions, which were particularly evident with the Pong game. It introduced an interactive novelty and refreshed the previously static one-way screen in the Faculty Hall. Users appreciated the ease of use and the fact that they did not have to install any special software or use additional hardware to play the games. They often inquired about other available games to play and whether the same setup could be deployed on another device or even a home television set.

Some of the users complained about insufficient sensitivity of the mobile screens, the lack of feedback (e.g., vibration), or occasional issues primarily caused by software on the mobile device (e.g., issues with the web browser where page refresh on “swipe down” cannot be disabled, pop-up toast messages that distract focus, etc.).

It is important to emphasize that these case studies were not controlled experiments but rather the testing of proof-of-concept applications in the wild, among real and diverse users. Their feedback was provided verbally, immediately after using the system.

5. Discussion

Three case studies were presented to showcase potential system applications, evaluating design and functionality in practical scenarios. Despite minor technical issues, users perceive the system as simple and intuitive. Users in transient or brief encounters with a video wall engage with applications, such as playing a game of Pong, which might be unlikely if the setup time for Controllers were prolonged or involved additional steps. The system was used by several hundred people across three case studies, yielding exclusively positive informal reactions.

The most common issues reported by users include difficulties with QR code scanning, such as lacking the necessary application on their mobile phones or encountering difficulties with QR code readability (in one of the case studies, the QR code image crossed over two screens of a video wall, so some QR readers could not parse the image). Another challenge involved the diversity of web browsers users had on their devices, with older browsers lacking certain features that, while nonessential, contribute to smoother operation. These include preventing page refresh by dragging your finger down from top of the screen, enabling fullscreen mode, or access to additional sensors or devices on the smartphone, for example, vibration or camera.

The system presented in this paper exhibits several limitations that should be considered in its application. Firstly, while the system was created to be as robust, expandable, and generic as possible, it may lack some functionalities or features needed for a specific use case, e.g., it does not provide an out-of-the-box solution for traditional protocols and functions of a standard TV/AV remote controller. Secondly, since the system uses an internet connection for communication, its network dependency introduces the potential for delays or disruptions in environments with low signal or otherwise poor connectivity. The web-technologies-oriented nature of the framework, which is one of its key advantages, means that the framework heavily relies on the speed and capabilities of the installed web browser on mobile devices and large screens. While it can be assumed that the web browser on a large screen is under the control of system developers, users’ mobile devices are not. Therefore, significant attention needs to be given to the compatibility of the system with various mobile web browsers and their versions, as well as the expected speed of user devices. Resource-intensive web applications could potentially slow down user devices and compromise the quality of the experience and the usability of the system.

Despite the system proving successful and user-friendly in practice, conducting more detailed research to obtain additional information from users, such as desired functionalities or identified shortcomings and bugs, would be beneficial. Additionally, exploring to what extent and why users of specific age groups or genders utilize the system could provide valuable insights. This approach will enable a comprehensive understanding of the system and its facets, allowing for the identification of potential drawbacks and suggesting improvements.

6. Conclusions and Future Research

In this paper we discussed different approaches of communication with large screens. Large screens have become ubiquitous in recent years and have captured the interest of researchers around the world and from various fields, who have explored their technical possibilities and various applications.

Despite this, after more than a decade of their presence, large screens are mostly still used as simple informational or advertising billboards, with very little or no interactivity.

A literature review revealed that most of the research was in the direction of solutions that are not easy to use, whether they require a specific hardware device, or, in case of the software, include the installation of a specific application, which limits the reach to only certain platforms, but also introduces additional steps that users must perform before they can interact with a big screen.

To address these limitations, this paper proposed a solution based entirely on web technologies, requiring users only to possess a mobile phone with internet access and a web browser, which has become a commonplace scenario. This paper introduces a modular system model facilitating application development and interaction across various devices. This model employs proven technologies such as web sockets and a server–client software architecture. The core of the system, referred to as the Environment, serves as the coordination and communication hub, forwarding information between other modules. The Remote Controllable Application (RCA) module runs on the large screen, connecting with one or more Controller modules loaded on user devices, enabling them to manage applications on the large screen.

In regard to future research, there are numerous opportunities to further improve the interaction between smartphones and large screens using web technologies. One important area is research into advanced user interface (UI) and user experience (UX) designs. Future studies could focus on developing more intuitive and user-friendly interfaces, potentially integrating gesture or voice control to make interaction more seamless and natural.

Another promising direction is the integration of augmented reality (AR) and virtual reality (VR) technologies. This could change the way users interact with large screens by providing more immersive and engaging experiences. Research could explore how AR and VR can be combined with large displays to create new forms of public interaction and entertainment.

The scalability and security aspects of the proposed system also represent important areas for future research. As the number of simultaneous users increases, it must be ensured that the system remains stable and responsive. In addition, ensuring the security and privacy of users interacting with public displays is of paramount importance, especially when sensitive personal data are involved.

Furthermore, the application of artificial intelligence (AI) technologies holds great potential in this area. AI algorithms could be used to personalize content on large screens based on user preferences or crowd dynamics, improving the overall user experience. This could also be extended to adaptive UI/UX, where the system automatically adapts based on user interactions and feedback.

Finally, we believe that it would be valuable to explore the potential for educational and collaborative applications. The system could be adapted for interactive learning environments or collaborative workspaces, taking advantage of the accessibility and convenience of smartphones. Research in this area could focus on the development of applications that facilitate learning, teamwork, and creative collaboration.

In the future, the authors plan to conduct an in-depth study among users, utilizing a questionnaire to gather their comments, suggestions, and critiques. Based on the collected feedback, the authors intend to continue with the development of the system, both in terms of the proposed model and its architecture, as well as the specific implementation, i.e., the prototype. This involves exploring various forms of interaction, feedback (e.g., sound and vibration), and communication, while also investigating the characteristics of applications that have been developed or will be developed on this platform in the future. Also, future plans involve utilizing the system as a foundation for developing interactive educational games, aiming to enhance the informative and entertaining aspects of the video wall with an educational component.