The Influence of the Labeling Effect on the Perception of Command Execution Delay in Gaming

Abstract

Gaming is one of the largest industries of digital entertainment. Modern gaming software may be susceptible to command execution delay, which may be caused by various factors, such as insufficient rendering capabilities or limited network resources. At the time of writing this paper, the utilized advances in gaming are often accompanied by brief descriptions when communicated to the users. While such descriptions may be compressed into a couple of words, even a single word may impact user experience. Due to the cognitive bias induced by the labeling effect, the impact of such a word may actually be more significant than what the user genuinely perceives. In this paper, we investigate the influence of the labeling effect on the perception of command execution delay in gaming. We carried out a series of subjective tests to measure how the word “optimized” affects gaming experience. The test variables of our experiment were the added delay between command and execution, the speed of the game, and the label that was assigned to gaming sequences. The test participants were tasked to directly compare gaming sequences with the different labels assigned: “optimized” and “not optimized”. In every comparison, both sequences had the same objective characteristics; only the label differed. The experiment was conducted on single-input and continuous-input computer games that we developed for this research. The obtained results indicate that for both of these input types, the labeling effect has a statistically significant impact on perceived delay. Overall, more than 70% of the subjective ratings were affected by the assigned labels. Moreover, there is a strong correlation between the amount of delay and the effect of cognitive bias. The speed of the game also affected the obtained results, yet statistically significant differences were only measured between the slowest and the fastest gameplay.

1. Introduction

In real-time interactive systems, particularly in video games, the perception of responsiveness is a crucial factor influencing the Quality of Experience (QoE). Any noticeable delay in command execution (i.e., the delay between issuing a specific command and perceiving its execution) can disrupt immersion and negatively impact gameplay satisfaction [1].

Latency in gaming environments—whether due to network conditions or hardware limitations—remains a problem when it comes to delivering an optimal gaming experience. While advancements in related technologies minimize the factual, objective extents of latency, human perception does not only depend on subjective time measurement, but also on psychological and cognitive factors, which, therefore, should be taken into consideration. One of such cognitive factors influencing delay perception is the labeling effect—a phenomenon which may alter a person’s subjective experience based on descriptive labels [2].

Despite the growing number of studies of latency perception in human–computer interaction (HCI), the scientific literature is limited when it comes to addressing the impact of the labeling effect on the perception of command execution delay. Multiple studies focus on the technical optimization, but fewer have examined the potential cognitive mitigation that can enhance the perception of responsiveness. If the labeling effect can be used systematically to alter the perception of delay, developers can design user interfaces that reduce frustration whenever latency is inevitable.

Furthermore, it should be stated that often a single word is sufficient to serve as a label for cognitive bias [3,4,5]. In the context of gaming, optimization is becoming more and more pivotal on multiple fronts. For instance, performance optimization in general is essential for making the most out of the user’s gaming system, which is typically either a console, a handheld device, or a personal computer (PC). As it is common that video games are updated and patched regularly, the communication of the changes frequently emphasize the fact of optimization (e.g., in the patch notes). Optimization is also of key importance when it comes to the gameplay itself. For example, at the time of writing this paper, online communities and video-sharing platforms proliferate the significance of optimizing gameplay strategies (e.g., “build order” in real-time and turn-based strategy games) [6,7], which is not limited to competitive gaming. This form of optimization is also known as metagaming, often referred to by gaming communities simply as “meta”.

On the one hand, the scientific literature on the impact of cognitive bias on the perception of delay—and subjectively perceived delay in general—is lacking, particularly for gaming. On the other hand, the optimization-related terms are more relevant than ever in the context of video games.

In this paper, we address the impact of the labeling effect on the perceived command execution delay in video games. We conducted a subjective study in which test participants directly compared identical game sequences, but in each pair, one was labeled as “optimized”, while the other one was “not optimized”. The hypothesis was that such a single word may significantly affect the perception of command execution delay. A single-input and a continuous-input video game was developed to test the hypothesis. The examined game sequences differed in terms of added delay and game speed. The label order was also taken into consideration, as half of the test participants compared the “optimized” sequence to the “not optimized” one, and the other half compared the “not optimized” sequence to the “optimized” one.

The detailed hypothesis statements (i.e., research questions) of our work are the following: (i) Do the assigned labels significantly impact the obtained results? Is there a statistically significant difference between the theoretical ratings based on the lack of objective differences (i.e., the sequence pairs in all of the comparisons are identical) and the obtained results that are influenced by the labels “optimized” and “not optimized”? (ii) Does the order of the label significantly impact the obtained results? Is there a statistically significant difference between the sequence pairs where “optimized” is the first label and “not optimized” is the second and pairs with the opposite order? (iii) Does the type of the input significantly impact the influence of the labels? Is there a statistically significant difference between the results obtained for single-input and continuous-input games? (iv) Does the game difficulty (i.e., game speed) significantly impact the influence of the labels? Are there statistically significant differences between the results obtained for easy, medium, and hard game instances? (v) Does the added command execution delay significantly impact the influence of the labels? Are there statistically significant differences between the game sequences with different extents of added delay? The null hypothesis for the hypotheses above is that measurable differences are due to random error. According to the best knowledge of the authors, these hypotheses have not yet been addressed by the scientific literature, and thus, they are deemed novel within this field of research.

The remainder of this paper is structured as follows. The related research efforts are reviewed in Section 2. The experimental setup of our work is detailed in Section 3. The obtained results are presented in Section 4. The paper is concluded in Section 5.

2. Related Work

The published scientific achievements relevant to our study are discussed in this section. The research efforts related to command execution delay and to cognitive bias are analyzed separately.

2.1. Command Execution Delay

The classic study of Shneiderman [8] concludes that fast response times (i.e., less than a single second) maintain user engagement and minimize the level of frustration, and that any delay greater than that may disrupt the cognitive flow. Furthermore, delays over 10 s may result in task abandonment. However, note that these results were published over 40 years ago, and since then, humanity’s interaction with computers and devices has changed drastically. As a more recent take on the investigated topic, Raaen and Eg [9] found that certain test participants not only noticed a command execution delay of 66 ms, but also became frustrated by it. On the other hand, greater delays may be tolerated if certain technologies are involved, and the user is aware of the fact. For instance, in the case of space communication (i.e., communication via satellites), Wang [10] found that even 10 s may actually be tolerated.

The work of Metzger et al. [11] proposes a lag model for video games. The model approaches lag as the sum of network delay, processing delay, and playout delay (i.e., command execution delay). Additional factors that contribute to the model include command rate, server tick rate (in the case of online games), and codec delay (in the case of cloud gaming). The survey of Pantel and Wolf [12] addresses the delay in real-time multiplayer games. One of the conclusions of the research is that the tolerance towards delay in such games is not only affected by the extent of network delay, but by the genre of the video game as well. The same conclusion was drawn by Schimdt et al. [13].

The study of Claypool and Claypool [14] points out that during gameplay, not every phase of the game is equally sensitive to latency. The so-called “play phase”—where real-time player interaction occurs—is the most vulnerable to latency in online games. However, a carefully designed game can still provide a tolerable gameplay even if the play phase is affected by high network latency. For example, Fritsch et al. [15] confirmed this in their study on the game Everquest2.

Among the research efforts on cloud gaming, Chen et al. [16] measured the command execution delay of PC gaming on different cloud services. The obtained results were between 135 ms and 500 ms. A more recent study [17] measured the latency in mobile cloud gaming. Depending on the type of network connection—which was either Wi-Fi or long-term evolution (LTE)—the latency varied between 70 ms and 177 ms.

In context of virtual reality (VR), Allison et al. [18] addressed the most relevant type of delay for such a technology, which is the delay between head movement and the corresponding field of visualization, and measured 200 ms to be the threshold of delay tolerance. Huang et al. [19] investigated the perception of weight in VR. The authors conducted an experiment in which test participants had to perform a balancing task, with various extents of delay towards visualization. The results indicate that when the delay exceeds 25 ms, the task becomes virtually impossible. These works signify the differences between general tolerance and task performance.

There are numerous research efforts that examine the impact of influence factors on QoE in gaming scenarios. Multiple studies [20,21,22] suggest that age, gender, personality, previous gaming experience, and user performance are some of the main factors that contribute to one’s perception of QoE. To measure the impact of these aforementioned factors, Moller et al. [23] designed the Gaming Experience Questionnaire (GEQ) for measuring seven dimensions of the players’ experience, while Depping and Mandryk [24] introduced the Game-Specific Attributing Questionnaire (GSAQ) to measure the correlation between the players’ characteristics (e.g., emotion, motivation, and behavior) and QoE. Among the mentioned factors, user performance—which is of particular interest to game designers and developers—is heavily dependent on command execution delay. In networked games, a large number of studies on the impact of delay on a game’s difficulty and the player’s performance [5,25,26,27,28] was conducted. The results of these studies converge and indicate that games with delays of around 100 ms are acceptable; when reaching 200 ms games are playable but annoying; and above 500 ms, they are reported to be extremely difficult and scored poorly in terms of QoE. Chanel et al. [29] concluded that performance is linearly affected by game difficulties. When the game is too easy, players are likely to feel bored. On the other hand, when the game is too challenging, players are inclined to feel frustrated. It is important to highlight that both of these cases result in the degradation of QoE. Numerous studies [30,31,32,33] proposed Dynamic Difficulty Adjustment (DDA) models to reduce the impact of the player’s performance on QoE. Sabet et al. [34,35] introduced an improved DDA model for cloud-based gaming which operates in real-time and does not require the tracking of player performance. By modifying in-game mechanics, the model could mitigate the negative effect of delay on QoE [36]. Lee and Chang [37] published the Advance Lag Compensation (ALC) algorithm to improve the QoE of first-person shooter games. Liu et al. [38] proposed that another approach to improve QoE is the player self-evaluation during gameplay. The results show that when players suffered from delay and poor performance, as long as their self-evaluated performance was fulfilled, the game still received positive QoE ratings. Only when these conditions were not met did the games became laggy and annoying to play. Delay-related disturbances to one’s QoE may also be due to the asynchronization of visualization and audio. Studies [39,40] suggest that visual–audio delays in video games can reach up to 250 ms, and the characteristics of the phenomenon and their perception depend on the content.

Most of the studies on the topic of command execution delay focus only on network delay, and local delay is often neglected. However, local delay is evidently an important component of the delay experienced by the player. Raaen and Petlund [41] measured local system delays and found that they can reach up to 100 ms—making them comparable to network delay. The study of Long and Gutwin [42] points out that the local delay can vary between 50 ms and 500 ms, and proposed a predictive model called “Time to React”. The aim of the model is to help developers in assessing and mitigating the effect of local lag.

Claypool et al. [43] investigated players’ performance with a delayed mouse. The test participants of the study were asked to select a fast-moving target with a delayed mouse at different speeds and delays. The results show that user selection times and error rates increase linearly with local delays, which led to significant decrease (i.e., dropped to approximately 25%) in user performance. In a different work, Claypool et al. [44] measured QoE using the same methodology, and came to a conclusion that user experience depends mainly on delay and not on the speed of the target. Liu et al. [4] addressed the effect of local delay on the players’ performance in the first-person shooter game, Counter Strike: Global Offensive. Test participants were asked to fire different weapons at targets under different conditions of delay. The results indicate that in such first-person shooter games, players are sensitive to local delay. It was shown that players’ performance increased 20% when latency was reduced from 125 ms to 25 ms. This conclusion is also supported by Ivkovic et al. [45]. The authors measured the local delay using a high-speed camera, frame-by-frame, from the moment the input was triggered to screen change. The test participants took part in 3D target tracking and target acquisition tasks. The study reveals that real-world local latency ranges from 23 ms to 243 ms, which causes significant and substantial degradation in the performance of first-person shooter games.

The scientific literature reports that first-person shooter video games are more sensitive to delay in general than other games from different genres. The work of Quax et al. [46] suggests that action games and racing games—which require intense interactions—are more vulnerable to delay than puzzle games and strategy games. The results align with findings in cloud-based gaming [47], traditional online gaming [48], and mobile gaming [49]. Claypool [50] explored the impact of latency in real-time strategy games. The study indicates that such games focus more on long-term strategy rather than on real-time reaction, reducing the impact of delay on the player’s performance. Claypool’s finding was supported by Pedri and Hesketh [51]. The authors concluded that a fast-interaction computer task requires more attention, leaving less cognitive capacity to encode temporal information, resulting in the perception that time passes more quickly during the task. Therefore, occurrences of longer delays are more likely to be tolerated.

Sabet et al. [34] conducted an experiment in which each test participant played different games and a local delay was added periodically. The three games—developed by the authors—included Shooting Range (aim and shoot at a target via a mouse), T-Rex (jump over 2D obstacles on a platform), and Rocket Escape (move left or right to avoid randomly spawned obstacles). The authors came to the conclusion that players can compensate the effect of local delay by predicting the next move of the game. Particularly in the case of Shooting Range and T-Rex, as game moves followed certain rules which could be predicted, adaptation was feasible for the players, leading to positive QoE ratings. On the other hand, Escape Rocket spawned obstacles randomly—rendering adaptation impossible—and thus QoE scores were impacted. Normoyle et al. [1] measured that players adapt well to constant delay even at 300 ms, but a small jitter (i.e., the variation of delay) can be detected by all users and causes annoyance.

These findings are supported by the work of Kohrs et al. [52], which investigates the functional magnetic resonance imaging (fMRI) scan of test participants experiencing delay while completing a locally delayed computer task. The results indicate that frequent delays allow the brain to adapt and return neural activity to a baseline, while an unexpected delay distract users from their task, increasing their cognitive load and making delays more noticeable.

Command execution delay may impact both single-player and multiplayer video games. However, delay in multiplayer games—which typically involve competition and/or cooperation—is more likely to degrade the QoE of the players [53] and, as such, may result in a sense of unfairness, as well as frustration towards other players [54]. Therefore, subjective ratings in the case of multiplayer gaming are more susceptible to degradation [55]. In addition, Siu et al. [56] concluded that single-player games are perceived as less challenging than multiplayer games, even when the task is completely identical. Among online multiplayer games, fighting games (FTG) are particularly sensitive to delay, as a lag of just a few frames may change the outcome of a match. Thus, the netcode of such games is carefully developed to provide compensation for delay [57,58,59].

The impact of local delay from other input devices was also studied. Claypool [60] measured the user performance in a selection task with a delayed thumbstick controller. The results show that the time the user needs to complete the task grows exponentially at higher delays, and higher player skills reduce the impact of delay on QoE. This confirms that the player ability is a key factor in delay tolerance. The same conclusion was drawn by Long and Gutwin [61] for pointing game devices (e.g., drawing tablet).

Table 1 lists research efforts that address the perceptual and/or tolerance thresholds of delay in various gaming genres. Perceptual threshold in this context refers to just noticeable difference (JND) [62]. According to the scientific literature, the perceptual threshold for first-person shooter (FPS) games, FTGs, multiplayer online battle arena (MOBA) games, puzzle platform games, and racing games is typically between 50 ms and 75 ms. The same for real-time strategy (RTS) games is measured to be 100 ms, but it can be up to 170 ms for survival role-playing games (RPG), and even 500 ms for team sports. Note that the latter was tested on an American football game in which gameplay was somewhat semi-automated, and thus, required less reactivity from the player. Tolerance thresholds may vary between 200 ms and 1250 ms. This great value was measured for a massively multiplayer online role-playing game (MMORPG). The game genres covered by Table 1 rely on the reactivity of the player, at least to some extent. Other genres, such as simple puzzle games and certain forms of turn-based games (e.g., turn-based strategy games), may tolerate even greater delays as the reactivity of the player does not directly influence the gameplay. One should also consider that in an online multiplayer scenario, those types of games that have low perceptual and tolerance thresholds are those that are most likely to be sensitive to network latency.

Table 1. Perceptual and tolerance thresholds of delay for each game genre in the scientific literature.

Scientific Work	Game Genre	Perceptual Threshold	Tolerance Threshold
Beigbeder et al. [28]	FPS	75 ms	200 ms
Quax et al. [63]	FPS	60 ms	–
Xu et al. [64]	FTG	67 ms	–
Fritsch et al. [15]	MMORPG	–	1250 ms
Tan et al. [65]	MOBA	50 ms	200 ms
Beznosyk et al. [48]	Puzzle platform	60 ms	200 ms
Pantel et al. [12]	Racing	50 ms	500 ms
Claypool [50]	RTS	100 ms	500–800 ms
Hohlfeld et al. [66]	Survival RPG	170 ms	–
Nichols and Claypool [67]	Team sports	500 ms	–

Table 1. Perceptual and tolerance thresholds of delay for each game genre in the scientific literature.

Scientific Work	Game Genre	Perceptual Threshold	Tolerance Threshold
Beigbeder et al. [28]	FPS	75 ms	200 ms
Quax et al. [63]	FPS	60 ms	–
Xu et al. [64]	FTG	67 ms	–
Fritsch et al. [15]	MMORPG	–	1250 ms
Tan et al. [65]	MOBA	50 ms	200 ms
Beznosyk et al. [48]	Puzzle platform	60 ms	200 ms
Pantel et al. [12]	Racing	50 ms	500 ms
Claypool [50]	RTS	100 ms	500–800 ms
Hohlfeld et al. [66]	Survival RPG	170 ms	–
Nichols and Claypool [67]	Team sports	500 ms	–

Numerous research efforts signify the impact of command execution delay on the experience of the players. The results indicate that constant extents of delay (i.e., without significant variation) can be tolerated up to 200 ms as players may adapt to such gameplay; however, those exceeding 200 ms often degrade the performance of the players and the overall QoE. Additionally, jitter, even at lower levels, is typically perceivable and commonly causes frustration. In our work, we address command execution delay at constant levels, without any extent of variation.

2.2. Cognitive Bias

The study of Wilke and Mata [2] concludes that one of the most common forms of cognitive bias is confirmation bias. According to classic studies of Wason [68,69] and of Klayman [70], confirmation bias happens when people are actively looking for confirmation of their idea, even when disconfirming evidence is available. Related reasoning errors arise from the interplay of cognitive limitations and motivational factors, rather than the lack of intelligence or knowledge.

Darley and Gross [71] studied the correlation between confirmation bias and the labeling effect. The authors concluded that people often gather evidence and information before making their judgment. The issue arises when their preconceptions are based on invalid and biased evidence. Jones and Sugden [72] conducted an experiment studying this behavior. The results show that people are willing to pay for evidence that supports their earlier beliefs, even when that evidence has no information value.

The labeling effect changes how human beings perceive their environment. Sakai et al. [73] conducted an experiment to explore the impact of visual images on the sense of smell. Test participants were asked to evaluate the intensity of odor while being shown a color. The intensity of the odors perceived by the test participants significantly increased when the shown color matched their expectations (e.g., dark brown color and Coca-Cola odor). The results indicate the influence of confirmation bias, based on the preconceptions of test participants. The color was used as a label to strengthen the test participants’ belief that the intensity of the odor was stronger, even though it was not. Bentler et al. [74] examined the impact of the labeling effect on human hearing. The test participants were asked to use a pair of the same hearing aids, which were labeled either as “conventional” or “digital”. The results indicate that most of the test participants were biased toward choosing “digital” hearing aids. Some of the test participants genuinely experienced better performance of the “digital” hearing aids. Iglesias [75] conducted a survey on the preconception of a retail bank service. The results show that customers’ preconception did not directly affect the overall evaluation of the service encounter. However, it did influence the perception of quality dimensions.

The labeling effect influences the way people process information. The study of Gao et al. [76] explores the impact of stance label and the readers’ selection of news articles. The study concludes that human beings are not neutral information processors and their existing beliefs, biases, and emotions affect how they interact with the news. De Graaf et al. [77] studied how experts would search and process data in documents. In the experiment, professionals were asked to answer a designed set of questions based on the provided documents. The results show that professionals tend to use their prior knowledge to find answers to questions that are not related to the document. Even when information is available, in some cases, it is ignored, which leads to an incomplete search.

The labeling effect may also have a significant impact on how people consume products. Chovanova et al. [78] concluded that product brands create biases on customers’ decision-making process, regardless of age and gender. The study by Gao et al. [76] shows that consumers are willing to pay more for beef if more information on its attributes (e.g., origin of the beef) is provided. The attribute information of a product is important and can be the decisive factor of the perceived quality of a product [79]. However, too much attribute information may be rather counter-productive. The study of Fitzgerald et al. [80] concludes that the overuse of labels and contradicting information confuse customers, degrading their ability to make decisions, and thus causing negative consequences for business. The study of Christandl et al. [81] shows that customers expect significant price hikes—after value-added tax (VAT) increases—even for products not affected by the VAT change. The expected price exceeded actual changes, confirming a bias in price perception.

In the context of software and video game development, multiple studies were conducted on the impact of cognitive bias. Studies [82,83,84] conclude that cognitive bias in software development mainly occurs during testing and debugging phases. During the testing phase, developers tend to be influenced by the expected behavior of the software. For example, as shown by the work of Calikli and Bener [85], the performance of the software was actively tested with an incomplete specification, confirming the completeness of the software, even when it was not. This phenomenon is also known as “positive testing” [86,87,88]. Rainer and Beecham [89] conducted a survey using evidence-based software engineering (EBSE) to evaluate different requirement management tool (RMT). The results show that the test participants tended to recommend the RMT they are familiar with over better tools supported by evidence. Cognitive bias also causes errors in the estimation of the effort required to develop software [90]. This is due to confirmation bias in selecting what is perceived as relevant project cost information.

Cognitive bias is also an important factor in the perception of video quality and virtual reality. Kara et al. [91,92] explored the impact of the labeling effect on the perceived quality of high-definition (HD) and ultra-high-definition (UHD) video streaming. The test participants were asked to compare videos with labels that were either true or misleading. The study suggests that the perception of visual quality is significantly influenced by labeling and expectations rather than actual resolution differences. In a more recent study, Kara et al. [93] examined the labeling effect in high dynamic range (HDR) video streaming. The study shows that the positive label (“Premium HDR”) assigned to visual stimuli increased the ratings of quality aspects such as luminance, color, and image quality. However, as a preconception of the “cost” of such an improvement, the frame rate was perceived to be worse. Geyer et al. [94] addressed the perceived quality of so-called “rugged” smartphones (i.e., highly durable smartphones) in a series of studies. One experiment compared a rugged phone to two conventional smartphones, while two other tests exhibited the visual stimuli on a single device—either a smartphone or a computer display. The obtained results were compared to ground truth data, excluding the triggers of cognitive bias, and statistically significant differences were found in every instance. Generally, the rugged smartphone was perceived to have worse capabilities as a price for its ruggedness—similar to the case of “Premium HDR” frame rate. Bouchard et al. [95] conducted a study on the impact of preconceptions on the sense of presence. In the experiment, test participants were misled into believing a virtual environment was real. Additionally, there were indicators to suggest that what they perceived was happening in real time. The results show an increase in the sense of presence of the test participants.

In summary, a great number of studies address the influence of cognitive bias on human perception and on decision-making processes. Confirmation bias generally occurs when individuals seek information that supports existing beliefs and preconceptions, ignoring contradictory evidence. The labeling effect may amplify this bias by altering perception through expectations induced by labels. Studies show that this phenomenon occurs on a daily basis in various domains of everyday life. This is particularly applicable to consumer behavior, which fundamentally builds on the perception of quality. In our work, quality corresponds to the responsiveness of video games (i.e., command execution delay). We investigated the influence of the labeling effect on the perception of this performance indicator. The methodology of the subjective tests—detailed in the next section—builds on the experimental setup of earlier works [92,93,94].

3. Experimental Setup

In this section, the experimental setup of our work is introduced. The chosen methodology is explained in terms of test environment (i.e., where the subjective study took place), apparatus and viewing conditions (i.e., the display used to visualize the test stimuli and the positioning of the test participants with respect to the display), test variables (i.e., the characteristics of the tests that changed between the different stimuli), test conditions (i.e., the selected combinations of the aforementioned characteristics), software (i.e., the games developed to test the hypothesis), and test protocol (i.e., how the subjective study was carried out with regards to data collection and stimulus order).

3.1. Test Environment

The subjective study was carried out in a laboratory environment. The test participants were shielded from external audiovisual distractions. During a test, only a single test participant and the test conductor were present within the test environment.

3.2. Apparatus and Viewing Conditions

The tests were carried out on a laptop with a screen size of 15.6 inches. The resolution of the display was set to $1920\times 1080$ pixels. Based on the relevant recommendations [96], the recommended viewing distance was 3.2 H; as the height of the screen was 23.8 cm, the default viewing distance (i.e., the distance between the screen and the eyes of the test participant) was set to be approximately 75 cm. However, as the study did not address any aspect of visualization quality, it was possible for the test participants to deviate from this value.

3.3. Test Variables

The subjective study was designed to accommodate three test variables. The first one was the label assigned to the stimulus, in order to potentially induce the phenomenon of cognitive bias. The label was either “optimized” or “not optimized”. The second test variable was the extent of the added command execution delay, which refers to the time period between the user input (i.e., the test participant presses a button on the keyboard of the laptop) and its execution within the software (i.e., the corresponding action is carried out). In accordance with the scientific literature, the chosen values were 0 ms (i.e., no delay), 50 ms, 150 ms, and 250 ms. As shown in Table 1, since 50 ms is the JND for the most reactive games, we chose this value as the smallest non-zero added delay. The other two values were derived from the tolerance thresholds of such games (i.e., 200 ms minus and plus the aforementioned JND). Furthermore, the work of Ivkovic et al. [45] indicates that real-world local latency is typically up to approximately 250 ms. The third test variable was the difficulty settings of the game (i.e., the speed of game progression and events). Three difficulties were distinguished: easy, medium, and hard. These difficulties are explained in the introduction of the software.

3.4. Software

The study was carried out on two types of video games in order to address the research questions: a single-input and a continuous-input game. Both of these games were developed by the authors, using the Unity Real-Time Development Platform [97]. The test variables (i.e., the labels, the added delays, and the difficulty settings) were also implemented using Unity.

The single-input video game was a 2D rhythm game. The task of the test participant as the player was to press the correct arrow key when the arrow in the game—moving vertically from top to bottom—reached the box at the bottom of the screen. The more the arrow aligned with the box at the time of the input, the more points were awarded. If a successful input was provided, the arrow disappeared, meaning that it was not possible for the test participant to circumvent the task by repeatedly pressing the arrow key. This also nullified the potential for providing a continuous input. The difficulty settings of the game was implemented through the vertical movement speed of the arrows. This means that in harder difficulties, an arrow spent less time in the area covered by the box at the bottom. For the easy, medium, and hard difficulties, this aforementioned time periods were 1500 ms, 750 ms, and 500 ms, respectively. However, each gaming sequence had a fixed 60-second duration. Therefore, harder difficulties involved more arrows. This decision was based on the fact that in this game, poor gaming performance (i.e., the player missed the timing of the input) did not shorten the sequence, as performance was reflected in points. Additionally, it should be noted that at an added delay of 250 ms, the game at hard difficulty was excessively challenging, since the arrow overlapped the bounding box for a total of 500 ms. Yet, as the extent of the delay was constant, it was actually possible to adapt to it by pressing the arrow key slightly sooner. A screenshot of the single-input video game is shown in Figure 1.

The continuous-input video game was a 3D driving game with two input keys: the left and the right arrow keys for steering a vehicle forward at a fixed pace. In this context, the continuous input means that keeping the arrow key pressed continuously steers the vehicle in the given direction. The road on which the vehicle moved forward consisted of three portions: the left, the middle, and the right portion of the road. At uniform distances, two of the three portions were blocked by objects. The task of the test participant was to steer the vehicle in order to avoid collision. In each sequence, the player had five “lives”, meaning that the game was over at five collisions. Additionally, regardless of “lives”, the game was also over if the player over-steered the vehicle, making the vehicle face the wrong direction (i.e., there was a more than a 90-degree rotation of the vehicle either to the left or to the right). The difficulty settings differed by the speed of the vehicle; harder difficulty meant greater constant vehicle velocity. Since each gaming sequence had the same road length, the three game difficulties—easy, medium, and hard—lasted 20 s, 40 s, and 60 s, respectively. Regarding the input provided, the entire input was temporally shifted via the added delay. For instance, at 250 ms of delay, the steering continued 250 ms after the arrow key was let go. This example is illustrated with a 500 ms continuous input, as shown in Figure 2. Adaptation to this phenomenon (i.e., letting go of the key sooner) provided an extra layer of difficulty in addition to that of the single-input video game. A screenshot of the continuous-input video game is shown in Figure 3.

3.5. Test Conditions

The approach of our work was to test every single combination. As there were two labels, four added delays, three difficulties, and two games, the total number of test stimuli was 48 (i.e., 24 for each game).

3.6. Test Protocol

The test was a series of direct comparisons. In each segment of the test, the test participant played the exact same game sequence twice, and compared the latter to the former. The only difference was the label: either the first one was labeled “optimized” and the second one was labeled “not optimized”, or vice versa. Prior to every single sequence, the label was shown on the screen. Additionally, these labels were integrated into the game sequences as well, as shown in Figure 1 and Figure 3. Labeling was consistent for each test participant, which means that either the “optimized” stimulus was always the first one, or always the second one throughout the entire test. The first order applied to half of the test participants, and the other order to the other half of the test participants. However, no such division was applied to input type, as every single test participant played both the single-input and the continuous-input games. In order to compare the two test stimuli within the scope of this double-stimulus method, a standardized seven-point comparison scale [96] was used: “Much worse” ($-3$), “Worse” ($-2$), “Slightly worse” ($-1$), “The same” (0), “Slightly better” ($+1$), “Better” ($+2$), and “Much better” ($+3$).

The temporal structure of the subjective tests is shown in Figure 4. Every single comparison consisted of five elements. First, the label of the first stimulus (i.e., stimulus A) was shown for 5 s, essentially a dark-gray screen exhibiting either the bold white text “optimized” or “not optimized” in the middle. This was followed by stimulus A itself. As detailed earlier, for single-input game sequences, the duration was always 60 s, and for continuous-input sequences, the game was either 20, 40, or 60 s long, based on the difficulty. These duration values assume that the game was not ended earlier due to gameplay. However, if that happened, then the sequence was evidently shorter. These two elements were then repeated, followed by a 30-s assessment period, during which the test participant had to select one of the seven rating options. If a test participant fully completed every single game sequence (i.e., no sequence was aborted), then the total duration of the test was approximately 1 h and 52 min. Optionally, due to such extended test duration, the possibility of a brief break was offered for the test participants. Prior to the test itself, the test participants were familiarized with the games, on various difficulties.

4. Results

The study was completed by a total of 60 test participants. For half of them, the “optimized” stimulus was to be compared to the “not optimized” stimulus, and vice versa for the other half. Of the 60 test participants, 39 were male and 21 were female. The age of the test participants ranged from 20 to 28, with an average age of 23. According to The Entertainment Software Association [98], there are more than 190 million people in the age range from 5 to 90 playing video games, 75% of which are Gen Z (i.e., 13 to 28 years old at the time of writing this paper). In five major European markets [99], there are an estimated 124 million people playing video games, and 78% of them are in the age range from 15 to 24. Therefore, having test participants in the range from 20 to 28 is a reasonable choice that produces representative results. Regarding the collected results and the demographic data, as age and gender were not investigated factors in the study, they were collected in separate datasets (i.e., the comparison scores were not matched with the demographic information). The experiment (i.e., the series of subjective tests that involved a total of 60 test participants) lasted over 43 days.

A total of 1440 results were collected, as there were four added delays, three difficulties, and two games, all the possible combinations of which were assessed by 60 test participants. Each test participant provided 24 comparison scores. The obtained results are separately investigated for all the test variable values, as well as the label order. In the analysis, the rating options of the comparison scale were represented by their numerical counterparts. These quantitative descriptors were also present during the experiment. Additionally, the numerical values for tests where the “not optimized” stimulus was compared to the “optimized” one are inverted, which means that all the results are approached from the perspective of the “optimized” stimulus (i.e., all the ratings express the comparison of the “optimized” stimulus to the “not optimized” stimulus).

4.1. Overall Results

Figure 5 exhibits the theoretical ratings distribution. As there were no objective differences between the stimuli in any given pair, then theoretically, there should not have been any rating other than “The same” (0).

Figure 6 shows the actual rating distribution of the obtained subjective data. It is based on all the 1440 ratings. First of all, it needs to be stated that less than 30% of the ratings report that the stimuli in the pairs were “the same”—in other words, more than 70% indicate a difference in perceived command execution delay. Of the ratings, 57.36% were positive (i.e., the “optimized” stimulus was deemed to perform better than the “not optimized” stimulus) and 13.61% were negative (i.e., the “optimized” stimulus was deemed to perform worse than the “not optimized” stimulus). The most frequent rating option was “Slightly better” ($+1$), while the least used option was “Much worse” ($-3$), with only three ratings in total. What is also interesting to see is that the “Better” ($+2$) option surpassed the amount of “Slightly worse” ($-1$) ratings. It may expected that when such a rating scale is used, the two slight options are the most frequent among the six differentiating options. However, distributions similar this one are also possible if a strong preference in one direction emerges [92,93,94]. The overall results exhibit an overwhelmingly positive assessment of the “optimized” stimulus in contrast to the “not optimized”—which, again, were completely the same. When compared to the theoretical all-zero ratings, Student’s t-test provides $p<0.01$, meaning that the cognitive bias in the experiment resulted statistically significant differences.

Comparing these obtained results to the results of similar studies in the scientific literature (i.e., experiments in which identical stimuli were compared via a 7-point comparison scale) enables a better understanding of the impact of cognitive bias. In this experiment, 70.97% of the ratings were differentiating, non-zero options. In the work on HDR [93], the corresponding value was 77.75% for quality aspects and 77.43% for stalling duration. For UHD [92], it was 72.4%. In the work on rugged smartphones [94], this value was 56.06% when multiple smartphones were involved, 66.91% when a single smartphone was used, and 38.83% when the content was displayed on a computer. It thus can be stated that the impact of the investigated instance of the labeling effect is among the greater ones.

Figure 7 and Figure 8 exhibit data for each test participant. As the study does not address differences related to demographic information (e.g., age, gender, etc.), the characteristics of the test participants are not shown on the axis, since they are irrelevant to the analysis. Moreover, as stated earlier, demographic data were collected independently from the subjective ratings, and therefore, such analysis would not be possible. The aim of these figures is to visualize the most important general rating behaviors of the test population.

In Figure 7, each marker represents the mean rating of a test participant. It is shown that 2 test participants had negative rating means, another 2 had means between 0 and 0.1, while the other 56 had higher means. Among them, 10 had means of 1 or higher. The average of these individual calculations were 0.73.

Figure 8 shows the number of “The same” (0) ratings that each test participant provided out of the 24 ratings in total. One test participant had only one such rating (“not optimized” compared to “optimized”; continuous input; medium difficulty; no added delay), and the other extreme value was 14. For 48 out of 60 test participants, at least two-thirds of the reported ratings consisted of non-zero options. The average number of zero ratings per test participant was 6.97 out of the 24.

4.2. Impact of Label Order

Figure 9 shows the rating distribution of the tests where the first label was “not optimized” and the second was optimized (i.e., the “optimized” stimulus was compared to the “not optimized” stimulus), and Figure 10 shows the rating distribution for the opposite case. Note that in this analysis, the results of the latter are inverted, so that both express the perceived command execution delay of the “optimized” stimulus compared to the “not optimized” stimulus. The means for the obtained results are 0.77 and 0.69, respectively. For both, the order of rating frequency is the same as in the case of the overall results, yet there are apparent differences when “optimized” was the first label: “The same” (0) and “Slightly better” ($+1$) ratings were fewer, while there were more of the others. In particular, the number of “Slightly worse” ($-1$) and “Much better” ($+3$) ratings increased by 5%. However, these apparent differences are not statistically significant, as $p=0.1$.

4.3. Impact of Input Type

The rating distributions of the single-input and the continuous-input games are shown in Figure 11 and Figure 12, respectively. The means for this classification are 0.72 and 0.74, respectively. The most apparent difference is that there are more “Slightly better” ($+1$) and fewer “The same” (0) ratings in the case of the continuous-input game—roughly 3% difference for each in contrast to the single-input game. However, these are far from being significantly different, as $p=0.41$.

4.4. Impact of Game Difficulty

The rating distributions of games at easy, medium, and hard difficulties are shown in Figure 13, Figure 14, and Figure 15, respectively. The means for these difficulties are 0.79, 0.75, and 0.66, respectively. While there is an approximately 10% difference regarding “Slightly better” ($+1$) ratings between easy and medium difficulty, the means are rather similar, and $p=0.26$. However, both the means and distributions indicate more of a difference when compared to the hard difficulty. Medium and hard difficulties do not differ significantly, as $p=0.15$, yet easy and hard do, as $p=0.04$. It is noteworthy to observe that while the amount of differentiating, non-zero rating options change only 2.5% in total, the distribution of these ratings vary considerably. Previous studies on the impact of difficulty on the player experience [20,30] conclude that video games are only enjoyable when the difficulties are maintained within certain thresholds. Exceeding or dropping below the difficulty thresholds may degrade the player experience. Other studies highlight the cognitive demand of increased gaming difficulty [100,101,102], including education games [103]. For instance, Large et al. [101] studied the players of a MOBA game through cognitive tasks, and found a correlation between various cognitive characteristics (e.g., speed of processing) and gaming performance (i.e., player ranking). Future research should address the correlation between cognitive abilities—and the load on these abilities via increased game difficulty—and rating behavior in the context of the labeling effect.

4.5. Impact of Added Command Execution Delay

The rating distributions at 0 ms, 50 ms, 150 ms, and 250 ms added command execution delay are shown in Figure 16, Figure 17, Figure 18, and Figure 19, respectively. The means for these delay values are 0.37, 0.41, 0.91, and 1.23, respectively. When 0 ms and 50 ms of added delay are compared, $p=0.33$, and for every other comparison, $p<0.01$, indicating statistically significant differences. The distribution of the ratings shows that, as the added delay increases, the number of differentiating, non-zero rating options increase as well: it is 55.83% for 0 ms, 69.44% for 50 ms, 77.78% for 150 ms, and 80.83% for 250 ms. With such a tendency, it is not surprising that there is a $0.92$ strong correlation between non-zero ratings and the amount of added command execution delay. It is also rather notable that at the highest delay, the “Much better” ($+3$) ratings reached 16.11%, nearly rivaling “The same” (0) ratings at 19.17%. A straightforward explanation of this phenomenon is that it is quite likely that the term “optimized” was directly associated with the amount of delay. Therefore, the more command execution delay was present, the more it could support a confirmation bias that assumes that the “optimized” gameplay suffered less in contrast to the “not optimized” one. Still, even when no delay was added, more than half of the ratings indicated perceivable differences.

4.6. Additional Discussion

In live service game environments, in order to meet the demand of players, developers usually make an effort to balance the game, fix issues and bugs, and also provide new contents. These novel additions to the game are usually delivered to the player with a label “patch”, or even “update” or “downloadable content” if the novelty is more substantial. The study of Zhong and Xu [104] concludes that frequent updates keep the player engaged and satisfied. This finding is also supported by the work of Liu and Samiee [105]. In addition, the study of Claypool et al. [106] indicates how new patches are able to adjust the way players play, or may even encourage brand new methods to play the game [107]. However, the work of Del Gallo [108] argues that the developers of the MOBA game League of Legends intentionally provide new “patches” in order to distract players from the unresolved issues of the games.

Patches do not always provide new contents as they may focus on technological improvements or even security enhancement. The study of Arora et al. [109] addresses the impact of software security patches, while the work of Lin et al. [110] proposes a model that automatically generates security patches. Security patches are often used in order to prevent cheating and other types of exploitation [111], and thus such patches are likely to be welcomed by players [112,113]. The study of Mertens [114] concludes that games with long lists of bugs and errors can be considered “broken”; however, such negative reputation may vanish as the developer releases new patches, updates, and downloadable contents. The author also argued that these “broken” games usually receive backlash at launch and player count eventually drops. Yet after receiving updates—the public notes of which usually describe the updated version of the game to have better performance and/or improved gameplay—players may be attracted to the game again.

In games where low network latency is essential to gameplay (e.g., FTGs), the improvement of the utilized netcode is crucial [14,57,58,59]. At the time of writing this paper, rollback netcode is widely used. Its main idea is to compensate delay by predicting player input. Of course, if there is an actual difference between the predicted and the real player input, then the game session is reverted to an earlier state and the correct input is carried out. The relevance of such research is that for certain video games, even a few frames worth of delay may have a significant impact on QoE and player performance. Therefore, any change to netcode communicated in patches or updates may have a significant preliminary influence on players that understand such mechanisms, and thus may have preconceptions about the updated gameplay performance.

The results of the experiment presented in this paper emphasize the impact of the labeling effect in the context of gaming. The methodology did not introduce objective differences within the pairs of the pair comparison tests (i.e., the two game sequences were always identical), only between pairs, in accordance with the test conditions. According to the best knowledge of the authors, no such study has been carried out so far that addresses any aspect of gaming QoE where a video game patch or update is communicated to have improved performance when there is actually zero difference. The results of such a subjective study could be highly relevant to the current practices of video game production and maintenance. One may hypothesize that for a portion of the test participants, the combination of the expectations and the lack of actual improvement may degrade certain aspects of QoE—as has already been measured in a prior experiment [115]—while it is also possible that the labeling effect may result in an overall QoE improvement for other test participants.

As stated earlier in the analysis of the overall results, 13.61% of the obtained results in our study is negative, which means that the “optimized” sequences were deemed to perform worse. Earlier works indicate that rating penalization may occur due to the mismatch of expectations and genuine differences [115], or as a form a compensation [93]. While the results are indeed overwhelmingly positive (57.36%), it is worth considering that certain yet-to-be-investigated labels and contexts may induce a greater extent of rating penalization.

5. Conclusions

In this paper, we presented a study on the impact of the labeling effect on the perception of command execution delay in video games. In the experiment, identical game sequences were compared, but one was labeled “optimized”, and the other one “not optimized”. The obtained results indicate that regardless of label order, input type, game difficulty, and added delay, the subjective ratings clearly exhibit the influence of the assigned labels. Generally, we can confidently reject the null hypothesis, as the measured differences are not due to random error. The ratio of differentiating scores in this study is competitive with prior results in the scientific literature (i.e., match the corresponding values or even surpass them), particularly with works using similar methodologies [92,93,94]. For the two label orders and the two input types, the results do not differ significantly. However, for the slowest and fastest gameplay (i.e., easy and hard difficulty), and for the various extents of added command execution delay, the differences are statistically significant. Furthermore, there is a strong correlation between the achieved results and the added delay. More specifically, the more delay is added, the greater the impact of the labeling effect is (i.e., more ratings differentiate the identical game sequences). However, the range of added delay in our experiment was limited by the tolerance levels measured by prior research efforts in the scientific literature. Therefore, greater extents of added delay may result in different tendencies of rating behavior.

One notable limitation of the study is that objective user performance was not recorded (i.e., sequence scores and when a sequence ends). In future work, the correlation between objective performance and subjective ratings should be investigated. Additionally, such gameplay-related characteristics may provide a better insight regarding user behavior. For instance, at higher difficulties, sequence abandonment may occur (i.e., the player forfeits the session either by intentionally providing the wrong input to end the sequence earlier in the case of the continuous-input game or by not providing further input at all), the detection of which may enable better data analysis. Another potential phenomenon is that a player may project operational errors (e.g., the personal mistake of pressing the wrong button at a given moment or pressing the right button too late) onto the game sequence itself, which may heavily influence subjective assessment. It is also worth highlighting that objective data related to reaction times may measure fatigue over time, and therefore, the inclusion of such data may map the connection between fatigue and the impact of the labeling effect. Fatigue may result more instances of operational error, making the aforementioned phenomenon even more relevant. As for command execution delay in general, while our work completely neglected the variation in delay, investigating the perception of jitter in such a context could also be relevant. Regarding the extent of added delay, as implied earlier, delay values that exceed the level of tolerance could be included in future studies.

Beyond command execution delay, other game characteristics should be addressed as well, such as frame rate. Moreover, multiplayer scenarios should be studied, which may have numerous considerations related to inter-user effects. Future work should also address the correlations between viewing conditions, display characteristics, immersion, and perceptual fatigue. For example, the different combinations of display size, display resolution, and viewing distance may affect the influence of the labeling effect on perceived visual quality. It is already well investigated how various viewing distances impact perceived quality [116,117,118], yet research efforts on its connection to the labeling effect and other forms of cognitive bias are still lacking. On the level of methodology, physiological measurements could provide an enhanced insight into the impact of such a form of cognitive bias. Physiological measurements are also utilized to determine the level of immersion [119,120,121], particularly in the context of novel immersive technologies such as VR [122,123,124]. In future works, the correlation between the level of immersion and the impact of cognitive bias should be investigated as well. Finally, as suggested earlier in this paper, it would be particularly relevant for the gaming industry to measure the impact of a performance patch or update that does not implement any objective difference, and therefore, quantify the potential positive and negative influence on gaming QoE.