Game Awareness: A Questionnaire

Abstract

This paper deals with one of the possible methodological approaches to an empirical examination of game awareness. Such an outline should enable the gathering of a deeper understanding of individuals’ awareness. The questionnaire is based on a scenario technique, and it is focused on game awareness in situations with possible strategic interactions. The goal is to assess the instrument regarding its reliability and validity. Internal reliability is assessed using Cronbach’s alpha coefficient. A k-nn analysis was used to estimate divergent construct validity and criterion validity, and nonparametric factor analysis was used to estimate convergent construct validity. The questionnaire satisfies the requirements of reliability and validity. Moreover, there is an indication that each of the eight scenarios may be used separately, as they also meet the criteria for reliability and validity. Possible practical and theoretical implications involve questionnaire use as a basis for educational intervention, other instruments’ development, as well as for the collection and publication of behavioral insights about game awareness. Behavioral insights should enrich the theory and incite behavioral game theory models of game awareness and its implications.

1. Introduction

Game theory allows for the analysis of behavior in strategic situations. While its origins lie in mathematics, its applications spread over various disciplines with an emphasis on economics. The parallel development of behavioral economics revealed the additional potential of game-theoretical analyses of strategic interactions while questioning the boundaries of rationality, which led to the development of behavioral game theory.

Regardless of the type and form of the game, there are basic elements—players, strategies, and payoffs—considered as presumptions and preconditions for the game’s existence. Only recently, another question arose as a possible game presumption, regarding the players’ awareness of the game. The issue can be separated into two questions, where the answers to both have strategic implications. The first one regards whether an individual is aware of the game’s existence. The unaware individual will most likely miss the opportunity for strategy consideration and implementation, namely, miss the opportunity to improve their own outcome. The lack of game awareness may result in situations such as truncated models, where one player passively accepts the consequences of the other player’s decisions (for example, like in a dictator ‘game’). The second question is whether the player is aware of the game in the same way as the modeler. While the answers to the two questions are connected, there is a distinction. The second question can be answered only if the situation is assessed as probably game-like, namely, if an amount of game awareness exists. The answer to the second question requires the awareness and assessment of the game elements: players, strategies, payoffs, and game rules. The players may not be aware of all available and feasible strategies. Still, they may also believe that they can use strategies that are not feasible or that have not been foreseen by the modeler. Similarly, players may not be aware of the existence of another player or may recognize unrelated actors as players. The payoff and outcome awareness are inevitably related to the awareness about the players and strategies. Moreover, a person who believes that there is a game where there is none may initiate a new game. The two questions require a distinction between the existence of game awareness and game awareness, where the latter may also be observed as the awareness of the game elements.

Even in examinations of bounded rationality, standard game representations often implicitly assume players’ awareness and understanding of the game in the same way the modeler does [1]. That also means that the game elements are treated as common knowledge. Still, the growing body of research in behavioral economics regarding human fallacies, biases, and individual differences in perception and decision-making processes calls for a reconsideration of such an approach. Moreover, modeling awareness, or rationality for that matter, as a deviation from the optimum, and behavior deductions from that deviation, is justified if the goal was to prescribe the behavior, and not if the goal was to describe, explain, or predict an actual individual’s behavior.

Depending on the context, the practical consequences for an unaware player may vary from imperceptible to serious. From the modeler’s perspective, the presumption of a player’s awareness requires the model adjustment, as long as the goal is to describe, explain, or predict the behavior and not to prescribe optimal behavior. Therefore, the relevance of the awareness assumption is revealed in various aspects: it enables a more realistic analysis and understanding of individuals’ behavior in strategic interactions; it creates the basis for an intervention; and it allows for an enhancement of the theory and modeling. Nevertheless, the role of awareness should not be misinterpreted, as awareness alone is not a guarantee for success in a game or in decision making, nor for rational reasoning during the game. Instead, it should be observed as a prerequisite.

Due to the importance of an awareness of the game and its elements, and its frequent omission in the game’s theoretical modeling and the scarcity of research, the issue of awareness requires a deeper understanding of its occurrence and chartered methodological approaches for empirical and theoretical aspects. This paper deals with one of the possible methodological approaches to an empirical examination of game awareness that should enable a deeper understanding of individuals’ awareness by using a questionnaire based on scenario techniques. Game awareness is examined through its ‘building blocks,’ the awareness in a narrow sense, and the assessment. As the purpose is to obtain insights at the individual level, it is not assumed that the game awareness must be complete, nor that it must be infallible. In addition, during the process of becoming aware, one might realize that there are some aspects that one is unaware of, or that one’s judgment may be fallible, and thereby assign beliefs or confidence to the accuracy of one’s own awareness and assessment. The goal of this research is to assess the instrument regarding its validity and reliability. Such an instrument could then be used for an exploratory investigation of awareness at the individual level, and to gain new insights into this overlooked but important area.

The following section introduces the methodological aspects, starting with the theoretical basis, and followed by a description of the assembling of the questionnaire and scenarios, of the studies and samples, a comment on the coding issues, and the validity and reliability conditions. The results section presents the reliability and validity results. The discussion and conclusion section tackles the limitations, implications for further research, and possible practical implications.

2. Methodology

2.1. Theoretical Basis

In defining game awareness, it is helpful to start from previously set frameworks. A part of awareness definitions offers a broad and strict approach, where a person is unaware of an event if s/he does not know about an event or does not know if the event is true, and does not know that s/he does not know it [2,3]. Those approaches are exclusive, assuming an absolute state of knowledge, without incomplete information or uncertainty that occurs in reality or personal characteristics that shape one’s awareness. Nevertheless, such approaches are common in economic modeling and standard game representation, where unawareness causes deviations from the optimal outcomes. Considerations that account for an individual’s characteristics result in the inclusion of psychological processes in definitions of the (un)awareness. In that sense, unawareness can be observed as “a real-life phenomenon associated with an unconscious mental state directed toward or lacking positive knowledge about a definite event” [4].

To explain awareness at the individual level, a part of the definitions emphasizes the role of perception [5,6,7,8]. Moreover, awareness involves various psychological processes [6] and is limited with a volume of available time and space [7]. In addition, the notions of seeking, using [5,6], meaning comprehension, and projecting that meaning to the near future [7] are included in behavioral approaches to awareness. That may seem counterintuitive, as seeking, using, and sharing information implies that there is already an amount of awareness that something is going on.

Most definitions, more or less, paint the same picture: there is a piece of information, but it is not used, and the reasons for that may be errors in perception (failure to see) or mistakes in interpretation (due to motivation, preferences, or knowledge). However, the consequences are much the same, as unaware individuals “cannot conceive of, nor articulate, the decision relevant contingencies they are unaware of” [9]. However, this quote calls for a deeper insight. For example, suppose one hears some racket outside of the office. In that case, one is aware that ‘something is going on,’ but, at that point, is still not aware of what exactly is going on in any way relevant for making a decision or taking an action.

From the modeling perspective, in the broadest sense, a game exists if there are: a strategic interaction; at least two players for which the choice of actions of at least one of them should imply the presence of a strategy; and there exists at least two or more possible outcomes shaped by the players’ choices [10]. Given that the basic game elements and presumptions must be satisfied for a game to exist, then the awareness of those elements should also be a threshold for determining whether a game exists or not. Therefore, an individual is game-aware if s/he is aware of the game elements.

Neuberg [11] examined how social behavior can be influenced by information that is perceived outside of conscious awareness. He observed players’ sensitivity to subliminal information through a strategies adjustment. The experiment reveals the intuition of individuals’ who grasp signals about the strategic interaction from their surroundings. The notion of perception outside of conscious awareness implies no deliberate reasoning and no definitive knowledge involved. This captures an aspect of game existence awareness, which may be observed as an aspect of, and as a minimal amount of, the awareness required for the volatile examination of game elements and of the reasoning about the possibilities that result in the game awareness.

In a narrow sense, a person becomes aware of an element as s/he detects an element. However, at that point, a person has yet to become aware of the properties and meaning of the element (identification and evaluation of the objects as the building blocks of awareness, [12]). In that process, one might become aware that there may be some properties or relevant information that one is unaware of ([13], or as in shadow propositions, [14]) or that one’s judgment may be fallible, and assign beliefs or confidence to the accuracy of one’s own awareness and assessment. The dynamic approach to unawareness assumes changes in beliefs as an individual becomes more aware [14]. The awareness and beliefs may change as individuals obtain more information (properties of the object), but the change may arise due to their reasoning about those properties, combined with memory retrievals of previous knowledge and experience, consequently assigning beliefs that their assessment is true. In addition, an individual’s reasoning may be bounded by the cost of reasoning, which may relate to the perceived outcome of the game and individual characteristics [15].

In a game, a player values different objects, or states of those objects, the same if they have the same properties [12]. Even so, the player can value objects differently if they can be categorized differently in any way that s/he is aware of. The player can base their preferences only on the properties that s/he is aware of [12]. This approach assumes a partial awareness. Moreover, a generalized approach to (un)awareness suggests that a person may be involved in a game, but is playing it according to their own subjective representation of the game, which may include ‘virtual’ moves that do not exist in the underlying game [16]. This provides another aspect of awareness—while individuals may not use available information, they may also use information unrelated to the current game [10], which results in a subjective game representation, and as a consequence, payoff beliefs that include moves unrelated to the actual payoff or the underlying game [16]. A standard game approach to unawareness might benefit from behavioral insights.

Blasch et al. [17] offer another approach by differentiating awareness and assessment. However, the awareness and the assessment as psychological terms seem to be entangled in the (un)awareness approaches [12,13,14,15,16], with the addition of understanding [1], seeking, using, and sharing [5,6] information. When combining inductive and deductive approaches [10,11,12,13,14,15,16,17], a compromising approach may be deduced: awareness, in a narrow sense, is a binary occurrence and a precondition for assessing the game elements. At the same time, assessment is required for developing the player’s awareness. Such an approach enables insights into behavioral aspects from empirical data, but in return, it may be compared to the logical and standard game approaches. This is the perspective that will be examined in this research, separating awareness and assessment, but examining both and their relationship to the game awareness.

Such an approach assumes that, if an awareness of game elements occurs, it entices a game elements assessment, resulting in an amount of game awareness. It can be theorized that a ‘fully’ game-aware person realizes the situation, its elements, and the properties of these elements, using all available information in a given framework. Such an approach is relatively close to Laplace’s demon and to a classical approach to rationality. However, the research in behavioral economics and related fields [5,6,10,18] shows that people do not use all available information, and that they use information outside of a given framework. Nonetheless, they also may be aware of only some elements or properties of the elements while disregarding others, and thus, gain an amount of game awareness. Game awareness and its components can but do not have to be observed as a binary occurrence.

Moreover, if the goal is to gain insights into individuals’ awareness, game awareness should not be regarded as a binary occurrence. As bounded and partial awareness approaches—each from its own perspective—denote, the gained awareness does not have to be complete, nor does it have to be infallible, but it enables the articulation of relevant decision-making elements. In addition, it may differ between individuals, as suggested by systematized results of experiments involving awareness and ethical choices [18].

In addition to the experiments, scenarios are often employed to elicit various elements in the decision-making process. While experiments dominate the research of decision making where narrow and particular goals are specified, scenarios are more appropriate for addressing complex and uncertain contexts [19]. Exploring new research directions is always a challenging ordeal, but scenarios are a helpful methodology for challenging existing assumptions while maintaining scientific rigor and usable results [19]. In that sense, this can be observed as an attempt to challenge the standard approach by introducing an examination of behavioral aspects as the first step in the possible implementation of empirical findings into awareness modeling.

If an individual were to play a game and deviate from the optimal choices, it would be hard or even impossible to establish, only by observation, whether the behavior occurred due to the unawareness of some game elements or due to bounded rationality applied to in-game reasoning. Questionnaires help assess phenomena that are otherwise hard to observe or assess. Such is the situation with the game awareness. At the individual level, with human subjects in mind, the awareness inevitably involves psychological processes. Psychological abilities and traits are often explored and assessed using questionnaires. Combining the scenario technique with a questionnaire enables an exploration of game awareness, uniquely positioned within the game theory and behavioral economies, respectively.

Given the chosen approach to the game awareness, scenarios should involve a game-like or a pre-game situation, with a possibility (and not necessarily a requirement) of strategic interaction. Unstructured negotiation games that resemble real-life situations created some of the most valuable findings in pre-game theory [20]. This served as an additional inspiration for the scenario approach and the use of unstructured, open-ended scenarios.

2.2. The Questionnaire and Scenarios

Game-like situations are here considered as realistic situations that involve a possibility for strategic interaction that may lead to virtual or real payoffs (utility). Given that the questionnaire was to be distributed among university students of economics and business economics, the real-life situations chosen for scenarios refer to simple situations.

Chosen situations relate to possibilities of strategic interaction in a purchase situation, a hangout with a friend, a job-related issue, a rumor about a raise, a task division, a competition, an opportunity-seizing situation, a confession (adapted prisoner dilemma), and a sharing situation (adapted trust game). Situations differ given their scope (possible areas of application of findings), complexity, the amount of provided information, and the gravity of potential consequences. While each situation presented in the scenarios would require a specific approach to modeling in the standard game representation, all these situations may occur for a person. This set of scenarios also enables insights into possible differences in the awareness formation given the different settings. Different situations, especially the ones related to economic aspects that an individual may encounter, offer a possibility for the findings’ comparison to the previous approaches to awareness, as well as to other research in behavioral economics. The findings might enrich, but also confirm or refute, existing choices in modeling individuals’ awareness.

Individuals are, at least, partially bound by the evidence and its possible interpretations, meaning that they cannot believe anything they would want to [18]. That means that the awareness, at least partially, arises from the available information. To examine sensitivity to provided information, scenarios should differ concerning the provided information about the player(s), their actions, payoffs, and interaction type. In addition, it can be assumed that the knowledge (possibly related to the situations described in scenarios) and preferences, as well as perception and interpretation errors, vary among individuals. This leads to an assumption that the responses will vary between individuals and between scenarios.

To assess if there is a game, the individual must assess game elements. This can be done by answering a set of questions [10]:

• Are someone’s actions and strategies influencing my possible outcomes, and which are those actions?
• Are there, and who are, the players who choose those actions and strategies?
• What are the possible outcomes, and how do other players’ actions influence the set of possible outcomes?
• What are the other players’ preferences, given the employed actions and strategies?

The questions that follow each scenario regard the list above and consider game elements, namely, the awareness about them, and the assessment of game elements, in line with Blasch et al. [17]. Given that most scenarios are unstructured game-like situations with incomplete information, it is assumed that the respondents are uncertain of their responses. That means that a level of belief, certainty, or confidence is assigned to each answer. These certainties can be observed as: the person believes, at least to a degree p, that there is a game element e (as in [21,22]). The entire range [0, 100] as offered to respondents to express their belief (certainty of confidence), instead of a shortened scale (such as a Likert scale), as it allows respondents the freedom of expressing themselves, follows the theoretical range of probabilities, and enables responses’ analysis as scale variables.

Additionally, observed elements (or signals about elements) may be assessed as deliberate or accidental [23], and such may be their consequences. It seems reasonable to assume that if an activity is assessed as accidental, one will likely assume that the game does not exist. Following that thought, a question about perceived deliberation is added to the questionnaire.

The game awareness questionnaire (GAQ) comprises nine scenarios, with 20 questions related to each scenario, and nine resulting items were created (

Table A1. GAQ: Scenarios, questions, and items.

Scenarios		Scenarios in Original Language (Croatian)
Scenario 1 1,2,3	You are going to the shop to do your usual grocery shopping. In addition to the items on your usual shopping list, you intend to buy your favorite chocolate. When you arrive to the shelf with the chocolates, you learn that your favorite chocolate’s price increased by HRK 10 (approx. EUR 1.34).	Ulazite u dućan s namjerom da obavite svoju uobičajenu kupovinu. Između ostalog, namjeravate kupiti svoju omiljenu čokoladu. Dolaskom do police s čokoladama, pogledom nalazite svoju omiljenu čokoladu i uočavate da je poskupila za 10 kuna.
Scenario 2 1,2,3,*	You are at a usual cafe hanging out with a friend. You notice that your friend is behaving slightly differently and is somewhat restrained. Moreover, you notice an unusual look that can be interpreted as playful or mischievous. You are trying to figure out what that would mean: maybe it is a secret about preparations for your upcoming birthday, maybe the friend knows something you do not know (e.g., some gossip or some footage that has appeared on YouTube), maybe the friend is about to announce something important, maybe he or she is conducting a scheme, maybe… There may be a lot of options and maybe they have nothing to do with you.	Na uobičajenoj ste kavi s prijateljem/prijateljicom. Primjećujete da se ponaša malo drugačije, pomalo suzdržano. Štoviše, tu i tamo primjećujete neobičan pogled koji može asocirati na zaigranost ili podsmjeh. Pokušavate dokučiti što bi to značilo: možda je to tajanstvenost zbog toga što priprema nešto za Vaš skori rođendan, možda zna nešto što Vi ne znate (npr. trač ili neku snimku koja se pojavila na youtube-u), možda se priprema priopćiti nešto važno, možda Vam priprema spačku, možda… Može biti puno mogućnosti, a možda uopće nema veze s Vama.
Scenario 3 1,2,3,*	You arrive to work. You expect the usual tense working pace, and in addition to all your existing obligations, you get another one. Lately, you have caught a few glances and greetings from other colleagues, and the reasons for these changes are not quite clear to you. You figure that you may be promoted to a better position. During the break, you meet a colleague, and you mention the amount of work you need to do. With understanding and good intent, your colleague admits to you that one of your colleagues spoke badly about you. Moreover, it was the person who assigns you your tasks.	Dolazite na posao. Očekujete uobičajen naporan tempo, kad pored svih već postojećih obaveza dobivate još jednu novu. Osim toga, u zadnje ste vrijeme uhvatili nekoliko pogleda i pozdrava kolega, koji Vam nisu bili potpuno jasni. Pada Vam na pamet kako Vam se možda sprema unaprjeđenje na bolje radno mjesto. Tijekom pauze, srećete kolegu kojem se požalite na količinu posla. S razumijevanjem i u dobroj namjeri, kolega Vam prizna da netko od kolega govori ružno o Vama. Štoviše, to je upravo osoba koja Vam dodjeljuje obaveze.
Scenario 4 1,2,3,*	A month ago, you started a seasonal job. This season is better than the last one for the company’s turnover. There is a rumor amongst your colleagues that all the employees could get a raise.	Prije mjesec dana dobili ste sezonski posao na kojem i sad radite. Ova je sezona bolja od prethodne po prometu poduzeća za koje radite. Među Vašim kolegama se šuška da bi svi mogli dobiti povišicu na plaću.
Scenario 5 1,2,3,*	You have received a team assignment to conduct some research and to report the results in a written form. There are four of you in the team. Your part of the assignment is to create a questionnaire. You are unfamiliar with designing questionnaires, so you contact one of the team members to help you. At that time, you learn that one of the team members already assembled the questionnaire.	Dobili ste zadatak u timu provesti manje istraživanje i o tome sastaviti izvještaj u obliku seminarskog rada. U timu vas je četvero. Vaš je dio posla sastaviti anketu. Ne znate ništa o tome i kontaktirate jednog člana tima da Vam u tome pomogne. U tom kontaktu saznajete da je netko već sastavio upitnik.
Scenario 6 1,2,3,*	You are a manager of a department for consulting clients in a leading consulting company. The company is well known for its substantial number of clients and for the low prices of its services. You meet a client and, in an informal conversation at the beginning of the meeting, s/he reveals that s/he has been contacted by a newly opened consultant company that offered their services at a lower price.	Voditelj ste odjela za savjetovanje klijenata u vodećem poduzeću za savjetovanje. Poduzeće je poznato po velikom broju klijenata, ali i najjeftinijoj cijeni usluge. Na sastanak s Vama dolazi klijent. U neformalnom razgovoru na početku savjetovanja, od klijenta saznajete da su ga kontaktirali iz novootvorenog poduzeća koje se odlučilo baviti istom djelatnošću, a savjetovanja nude po nižoj cijeni.
Scenario 7 1,2,3,*	You have come up with an idea to start your own business. Even though all the people who hear about the idea agree that it is an excellent idea, you do not have the capital to start the company. You consider giving up the idea, given that you cannot finance it. At that time, you receive an invitation to pitch your idea in front of five investors. If you impress the investors, they could decide to invest.	Osmislili ste ideju za pokretanje svog poduzeća. Iako se svi koji čuju slažu s Vama da se radi o izvrsnoj ideji, Vi nemate dovoljno kapitala za pokretanje poduzeća. Namjeravate odustati od ideje, jer je ne možete financirati. Tad primate poziv na sastanak na kojem ćete održati prezentaciju svoje ideje pred pet investitora. Ako ih oduševite svojom idejom, investitori bi mogli odlučiti investirati u Vas.
Scenario 8 1,2,3,*	You and your colleague have been accused of cheating in an exam using unallowed electronic devices and are taken in front of the faculty’s ethical committee for interrogation. You and your colleague are placed in separate rooms and cannot know what the other one is saying during interrogation. The ethical committee member who conducts the interrogation makes the following offer to each of you: ”You may choose to confess or remain silent. If you confess and your accomplice remains silent, you will not be punished, and your testimony will be used to ensure that your accomplice gets a year of suspension. Likewise, if your accomplice confesses while you remain silent, s/he will go free while you get a year of the suspension. If you both confess there will be two punishments, but I’ll see to it that you both get only 6 months of suspension. If you both remain silent, I’ll have to settle for a token punishment for having unallowed devices during the exam and you will get 1 month of suspension”.	Vaš kolega/ica i Vi optuženi ste za varanje na ispitu korištenjem nedopuštenih elektronskih uređaja i odvedeni ste pred Etički odbor fakulteta na ispitivanje. Vi i Vaš/a kolega/ica smješteni ste u odvojene prostorije i ne možete znati što druga osoba govori ispitivaču. Član Etičkog odbora koji provodi ispitivanja daje jednaku ponudu Vama i kolegi/ci u drugoj prostoriji: “Možete odabrati priznati ili šutjeti. Ako priznate, a Vaš/a kolega/ica ostane šutjeti, Vi nećete biti kažnjeni, a Vaša će izjava poslužiti kao dokaz i Vaš/a kolega/ica dobiva godinu dana suspenzije na studiju. Također, ako kolega/ica prizna, a Vi šutite, on/a neće biti kažnjen/a i Vi dobivate godinu dana suspenzije na studiju. Ako oboje/obje/obojica priznate, svaki dobiva po 6 mjeseci suspenzije. Ako oboje/obje/obojica odaberete šutjeti, bit ćete kažnjeni samo za posjedovanje nedopuštenih elektroničkih uređaja tijekom ispita i to s jednim mjesecom suspenzije”.
Scenario 9 1,2,3,*	You and your colleague participated in a presentation for your faculty. At lunch, you were given the last sandwich, so your colleague is left with none. Your colleague promises to take you for a coffee later, hoping that you will give him a piece of your sandwich now. You can decide to trust your colleague and give him a piece of the sandwich or keep it all for yourself.	Sudjelovali ste u prezentaciji na fakultetu s kolegom. Na marendi ste dobili posljednji sendvič, tako da je kolega ostao bez. Kolega obećava da će Vas kasnije odvesti na kavu, u nadi da ćete mu sad dati dio sendviča. Možete odabrati vjerovati kolegi i dati mu dio sendviča ili cijelog zadržati za sebe.
Questions per scenario
Q1 1,2,3,*	Is it possible to notice any activity that influences your outcomes (economic or otherwise)? 1,2,3,* Yes 1,2,3,* Don’t know 1 No 1,2,3,*	Je li moguće uočiti ikakvu aktivnost drugih osoba koje utječu na Vaše ishode (ekonomske, ili kakve druge)?
Q2 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q3 1,2,3,*	Can you identify (describe) that activity?	Možete li identificirati tu aktivnost? Opišite je:
Q4 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q5 1,2,3,*	Is there a person (or a group/association/company, etc.) who initiated the activity? Yes 1,2,3,* Don’t know 1 No 1,2,3,*	Postoji li osoba (ili grupa/udruženje/poduzeće) koja je poduzela tu aktivnost?
Q6 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q7 1,2,3,*	Can you identify a person (or group/association/company, etc.) that initiated the noticed activity? Describe.	Možete li identificirati tu osobu (ili grupu/udruženje/poduzeće)—opišite:
Q8 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q9 1,2,3,*	Can you determine desirable outcomes for that person (or group/association/company, etc.)? Yes 1,2,3,* Don’t know 1 No 1,2,3,*	Možete li utvrditi koji su poželjni ishodi za tu osobu (ili grupu/udruženje/poduzeće) s obzirom na poduzetu aktivnost?
Q10 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q11 1,2,3,*	The outcomes for yourself regarding the initiated activity are Positive Don’t know 1 Negative	Koji su ishodi za Vas s obzirom na poduzetu aktivnost te osobe (ili grupe/udruženja/poduzeća)?
Q12 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q13 1,2,3,*	Describe your own outcomes given the initiated activity by a person (or group/association/company, etc.).	Opišite koji su ishodi za Vas s obzirom na poduzetu aktivnost te osobe (ili grupe/udruženja/poduzeća):
Q14 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q15 1,2,3,*	Do you believe that something is going on that affects you and that there exists a person (or group/association/company, etc.) whose activities influence your outcomes? Yes 1,2,3,* Don’t know 1 No 1,2,3,*	Vjerujete li da se događa nešto što utječe na Vas i da pritom postoji neka osoba (ili grupa/udruženje/poduzeće) koja svojim aktivnostima utječe na Vas, te da s obzirom na to za Vas nastaju drukčiji ishodi?
Q16 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q17 1,2,3*	Do you believe that activity was deliberate or accidental? Deliberate 1,2,3 Don’t know 1 Accidental 1,2,3	Vjerujete li da je to djelovanje namjerno ili slučajno?
Q18 1,2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Q19 2,3,*	Is there an action that you can take to influence outcomes? Yes 2,3 No 2,3	Možete li Vi donijeti odluku ili poduzeti aktivnost na način da utječete na ishode?
Q20 2,3,*	How certain are you in the previous answer? (Please, answer in the form of probability that represents your assessment that the offered answer is true, 0–100%.)	Koliko ste uvjereni u prethodni odgovor? (odgovoriti u obliku vjerojatnosti koja predstavlja Vašu procjenu da je ponuđen odgovor točan u obliku broja 0–100%):
Items
	Without complementary probability	With complementary probability (CP)
I1 1,2,3,*	Activity awareness = Q1 × Q2/100	if Q1 = 1, then Q1 × Q2, else (100−Q2)/100
I2 1,2,3,*	Activity assessment = Q3 × Q4/100	if Q3 = 1, then Q3 × Q4, else (100−Q4)/100
I3 1,2,3,*	Player awareness = Q5 × Q6/100	if Q5 = 1, then Q5 × Q6, else (100−Q6)/100
I4 1,2,3,*	Player assessment = Q7 × Q8/100	if Q7 = 1, then Q7 × Q8, else (100−Q8)/100
I5 1,2,3,*	Other player’s outcome awareness = Q9 × Q10/100	if Q9 = 1, then Q9 × Q10, else (100−Q10)/100
I6 1,2,3,*	Own outcome assessment = Q13 × Q14/100	if Q13 = 1, then Q13 × Q14, else (100−Q14)/100
I7 1,2,3,*	Game existence awareness = Q15 × Q16/100	if Q15 = 1, then Q15 × Q16, else (100−Q16)/100
I8 1,2,3	Deliberation = Q17 × Q18/100	if Q17 = 1, then Q17 × Q18, else (100−Q18)/100
I9 2,3	Own activity awareness = Q19 × Q20/100	if Q19 = 1, then Q19 × Q20, else (100−Q20)/100

Notes: The term certainty is used in psychological but not epistemic interpretation. A level of certainty in the provided answer is the respondent’s belief that his or her assessment is true/correct. Questions and items denoted with ¹, ², and ³ were used in the first (2019), second (2020), and third (2021) study, respectively. The findings from the second study were published previously, based on a part of the data (respondents without prior game theory knowledge) [10]. In the first two studies, instead of “How certain are you in the previous answer,” a different formulation (“Please provide your determined certainty in the answer”) was implored. In the third study, along with each scenario, respondents were instructed to “Imagine you are a part of the following scenario” and “Take a couple of minutes to engage with the scenario. You may want to close your eyes and imagine the situation. Have you done that? Given the scenario and your part in it, please, answer the following set of questions”. Questions and items denoted with * have been retained after the reliability and validity analyses.

). The first seven scenarios regard every-day and simple business/economic situations that may (or may not) be games, while the eighth one presents an adapted prisoner dilemma, and the ninth one presents an adapted trust dilemma.

Simple language should ensure the respondents’ cooperation and engagement [24,25], so common terms were used. The terms used for examining awareness (in a narrow sense) were ‘is there some…’ and ‘detect’, and the terms used for examining assessment were ‘determine’, ‘identify’, ‘describe’. Instead of the word ‘action’, the term ‘activity’ is used, due to its close meaning and to avoid confusing respondents who are not familiar with game theory terms. Instead of ‘players’, a descriptive phrase such as ‘a person (or a group/association/company, etc.)’ was used. To avoid using the term ‘payoff,’ in the first study, the term ‘welfare’ was used in the first question, but was later changed to ‘outcome’.

There are only three open-ended questions per scenario, as they tend to be burdensome to respondents [24]. However, open-ended questions are useful when the presented choices could influence responses, when the answers are not entirely predictable, and when there are potentially unanticipated responses that require flexibility [26]. For example, reasoning, computation, and memory retrieval can, but do not have to be done at the conscious level, as Salehnejad [27] finds that learning through continuous belief-updates makes the homo economicus an intuitive Bayesian statistician—where some of the reasoning does not occur at the conscious level, and neither does the update strictly follow conditional probabilities. The individual’s assessment can be achieved by reasoning or intuition [28], where the intuition, along with the individual’s base of experience and knowledge, might lead to unexpected responses. In addition, an individual’s reasoning may be improved, as it involves skills that can be learned, so revealed fallacies in answers to open-ended questions in a situation assessment may be used (for example, in the classroom) as a basis for intervention [29].

Each scenario is a complex issue, and there are 20 questions per scenario. Examination of a complicated issue with a single question could induce context effects [30]. The sequence of the questions is also relevant for the successful measurement by a questionnaire [24,31,32], so questions are ordered to allow a step-by-step assessment of the situation, which is in line with a specific research goal [33].

The ambiguity of the first seven scenarios is achieved, to an extent, by not introducing the possible action for the respondent. The motivation for such a choice lies in the theoretical distinction between game awareness (a person being ‘fully’ aware) and game existence awareness (a person being aware in a narrow sense). One should first become aware that something is going on, assess what is going on, and only then approach an assessment of what one can do to influence one’s own outcome (introduced as the last two questions, Q19 and Q20). In other words, based on the game’s existence awareness, the situation can be suspected to be a game or not. Only if one believes that there is a possibility for a strategic interaction, it makes sense to put any effort into the examination of the set of possible and feasible strategies/actions as a reaction to other players’ actions. The first seven scenarios (S1-S7) mimic every-day situations, which usually do not occur as games with predefined rules but require individuals’ awareness and assessment. On the contrary, scenarios 8 (S8) and 9 (S9) are structured and, as such, can be used as reference points.

The questionnaire was pre-tested during the winter semester of the 2017/2018 academic year with a small sample of 89 first-year students. Each of them responded to questions from one of the first seven scenarios. The insights revealed that the respondents were able to grasp the received information, assess (at least some) elements and/or their properties (with a few unexpected conjectures), and were aware of (at least some) game elements. The questionnaire was completed in the pen-and-paper form. During its completion, a researcher was present to answer any students’ questions and kept notes to remove unnecessary ambiguity in scenarios or to paraphrase questions for the subsequent studies. While the pre-test was helpful, only a few necessary changes were required, which is in line with Hunt et al. [34].

It is also necessary to note that there is an amount of logical dependency between the questions: awareness and assessment are intertwined, even in theory; the action must be taken by a player; and if a person can be detected as a player, then there should also be a related action. In addition, if there is an action, there should exist different outcomes. At least, this is true from a logical and classical game theory perspective. However, humans are prone to biases and frequently escape the bounds of rationality. The pre-test showed that students could differentiate elements and could determine that there was an activity, even if they could not assess who initiated it, or that they could notice another player and not be able to pinpoint the activity. This was expected, as individuals may evaluate each object or, in this case, game elements, separately [12,14]. Moreover, this confirmed that the questionnaire captured patterns and (in)consistencies, due to the information provided in the scenarios.

Table 1. Information provided in scenarios.

Information	Scenario
Scenario 1
action	The price increases
player(s)	Not stated explicitly (it can be recognized as shop/shop manager, manufacturer, distributor, etc.)
outcome	Loss of 10 HRK/the lack of satisfaction from favorite chocolate
Scenario 2
action	Non-verbal signals
player(s)	Not stated explicitly (a friend with whom an individual is drinking coffee, or a third party)
outcome	Not stated explicitly (the payoff could be both positive and negative)
Scenario 3
action	Two separate actions are stated: workload and a rumor
player(s)	Two players are stated: colleague and task assignor
outcome	Not stated explicitly, but both positive and negative payoffs are indicated
Scenario 4
action	A rumor/company’s turnover
player(s)	Not stated explicitly (the rumor’s source is the colleagues, but the player is the company)
outcome	Not stated explicitly (a raise is expressed as a possible payoff, but the information about the higher workload can be deduced from more tourist arrivals, which could also be related to higher salaries)
Scenario 5
action	Activities within a team assignment
player(s)	Team members
outcome	A grade/workload
Scenario 6
action	New competitor offers service at a lower price and/or a client’s attempt to negotiate a lower price
player(s)	Newly opened consultant company and/or a client
outcome	Not stated explicitly, but indicates possible loss (implicitly, in the long run, it can be the loss of a market share to a competitor and/or, in the short-run, a diminished price charged to the client or the loss of the client)
Scenario 7
action	Invitation to pitch the idea
player(s)	Five investors
outcome	Not stated explicitly (immediate—presentation of the idea (or not), and consequently a possibility of investment—positive payoff—of an unknown amount is indicated)
Scenario 8
action	Confess or remain silent, for both players
player(s)	A colleague
outcome	0, 1, 6, or 12 months of suspension
Scenario 9
action	A promise of the colleague and one’s own action about the sandwich allocation
player(s)	A colleague
outcome	A possibility of a coffee and a piece of/whole sandwich

offers an overview of the provided information. Scenarios S2 and S4 offer the least amount of information, followed by scenarios S1, S3, S6, and S7. Scenarios S5, S8, and S9 offer the most information, whereas scenarios S8 and S9 also offer alternatives for their own action choice and respective outcomes. Scenarios S1, S6, and S8 suggest adverse outcomes, scenarios S4, S7, and S9 suggest positive outcomes, and scenarios S2, S3, and S5 imply possibilities for positive and negative outcomes. Scenarios S6 and S7 allow for an additional examination of short- or long-term respondents’ orientation. Scenarios S1, S4, and S6 offer possibilities for parallel grasping of the situations: at the personal and the market levels (where the latter requires more deduction and abstraction). Scenarios S1, S5, S6, and S8 offer a simple, observable type of activity, while scenarios S2, S3, S4, and S6 introduce a possible activity as a rumor or non-verbal signal. It could be argued that scenario S9 offers a simple, observable activity, but that is true only for the respondent’s actions. In contrast, the colleague’s action is a promise, which may be observed as cheap talk. Scenario S7 is distinct regarding the activity, as it suggests only an invitation, where a further development of the situation does not occur unless the respondent chooses to pitch the idea. For any other scenario, passive acceptance leads to an outcome—as in a different state of virtual or real utility, including affective states (such as satisfaction)—compared to the state before the initiated activity. On the contrary, scenario S7 requires an action by respondents to change the outcome. Noted variations in scenarios should enable the capture of how respondents form game awareness (or a lack thereof), given different amounts and qualities of the provided information. In addition, while the questionnaire may be used as a whole, each scenario may be used separately to examine the desired aspect. Other aspects of the questionnaire’s development are explained in the following sections.

2.3. Studies and Samples

The research ethics assessment committee at Juraj Dobrila University of Pula gave their positive opinion on the questionnaire and research design. Informed consent was obtained from each participant. Three studies were conducted via an online questionnaire during the summer semesters of the 2018/2019, 2019/2020, and 2020/2021 academic years in Pula, Croatia. The first study included only the first seven scenarios, without the questions Q19 and Q20, and with the ‘don’t know’ option. The second study was enriched for the eighth and the ninth scenarios, with minor paraphrasing of the questions and the removal of the ‘don’t know/not sure’ option. That option was removed due to coding and item-interpretation difficulties (which will be discussed in the Coding Issues section). The third study involved nine scenarios with 20 related questions.

In the third study, the students were not present in the classroom while filling in the questionnaire, so an additional written instruction was provided at the beginning of the questionnaire: “Please, stay focused while answering the questions, but also try to answer as spontaneously and honestly as possible. An answer to every question is required. It takes about 30 min to complete the questionnaire, and answering questions about the assessment of the situation requires a higher level of attention. There are nine scenarios in the questionnaire. For each of them, you are asked to notice and assess whether there is an element and to attribute your belief to it, i.e., the probability of how confident you are in your answer”.

In each study, completion was voluntary, but the points incentivized the students if they chose the questionnaire instead of their regular practice assignment. The students received the same amount of points as they would for a regular assignment. They were informed that the responses would be evaluated before the points assignment and that the points would be assigned only if they provided thoughtful answers. For the 2019 study, in the assignment involving the questionnaire, students functioned as pollsters and collected the responses from other students at the university after their own completion of the questionnaire. As a result, the 2019 sample is more heterogeneous than the other two. The samples from the 2020 and 2021 studies involve only first- and second-year undergraduate economics and business economics students. The samples’ characteristics are provided in

Table 2. Sample characteristics.

Study	Age (Mean, Standard Deviation, Min, Max)	Gender (Frequency Male, Female)	Year of Study (Mean, Standard Deviation, Min, Max)	Previously Learned Game Theory (Frequency)	Used Knowledge of Game Theory (Frequency)	Number of Responses
2019	22.03	48.54%	1.934	54%	21.8%	822
	4.788		1.1556
	18	51.45%	1
	46		5
2020	19.91	27%	1.048	43.5%	18.9%	756
	3.648		0.2131
	18	73%	1
	52		2
2021	19.86	33.5%	1.06	41.8% *	12% *	452 *
	2.318		0.2373
	18	66.5% *	1
	34 *		2 *
Total	20.758	37.2%	1.409	47.37%	18.53%	2030
	4.065		0.8703
	18	62.8%	1
	52		5

* A total of 46 responses for the socio-demographic data are missing in the 2021 study; the total number of responses in the 2021 study is 498; the total number of responses for all studies is 2076.

(Here is reported a summary of the data coded by LR without complementary probability (more on this topic in the Coding Issues section). The differences in the number of responses arose because of the removal of unusual cases during the data examination. The summaries for the loosely coded data with complementary probability and the strictly coded data with complementary probability are presented in

Table A2. Sample characteristics for loosely coded data with complementary probability.

Study	Age (Mean, Standard Deviation, Min, Max)	Gender (Frequency Male, Female)	Year of Study (Mean, Standard Deviation, Min, Max)	Previously Learned Game Theory (Frequency)	Used Knowledge of Game Theory (Frequency)	Number of Responses
2019	22.03 4.788 18 46	48.54% 51.45%	1.934 1.1556 1 5	54%	21.8%	822
2020	19.931 3.687 18 52	26.9% 73.1%	1.049 0.2154 1 2	43.8%	18.8%	739
2021	19.83 2.31 18 34 *	33.5% 66.5% *	1.06 0.2382 1 2 *	45.74% *	13.37%	448
Total	20.768 4.0822 18 52	37.23% 62.77%	1.414 0.8738 1 5	48.41%	18.82%	2009

* A total of 47 responses for socio-demographic data are missing in the 2021 study; the total number of responses in the 2021 study is 495; the total number of responses for all studies is 2056.

and

Table A3. Sample characteristics for strictly coded data with complementary probability.

Study	Age (Mean, Standard Deviation, Min, Max)	Gender (Frequency Male, Female)	Year of Study (Mean, Standard Deviation, Min, Max)	Previously Learned Game Theory (Frequency)	Used Knowledge of Game Theory (Frequency)	Number of Responses
2019	22.03 4.788 18 46	48.54% 51.45%	1.934 1.1556 1 5	54%	21.8%	822
2020	19.87 3.469 18 52	27.1% 72.9%	1.048 0.2139 1 2	43.6%	18.7%	750
2021	19.855 2.315 18 34 *	33.92% 66.08% *	1.06 0.2367 1 2 *	46.04% *	13.88%	454
Total	20.711 3.9348 18 52	37.33% 62.67%	1.38 0.8763 1 5	48.37%	18.88%	2026

* A total of 47 responses for socio-demographic data are missing in the 2021 study; the total number of responses in 2021 study is 501; the total number of responses for all studies is 2073.

, respectively).

The applied incentives may be considered partial or weak when examining economic actions and decisions, which take effort and time to carry out. In this case, only the awareness of possible strategic interaction is examined. In other words, concerning situations precede situations that require extensive in-game deliberation and action. In addition, to ensure that the responses (and deviations from the described situations) were an indicator of awareness and not possible mistakes due to lack of motivation, the responses to the open-ended questions were considered as a proxy for the engagement, and uniform or unusual responses were removed from the further analysis. This is also one of the reasons for the lower percentages of response retention.

In addition, a rate of completion must be discussed. In the first study, 198 respondents approached the survey and completed the questionnaire, and 117 (59.09%) responses were retained after data insights. In the second study, 113 respondents approached the survey, 84 completed the questionnaire, and 77 (68.14%) responses were included for further analysis. In the third study, 94 respondents approached the survey, 65 completed the questionnaire, and 47 (50%) responses were kept. However, each of the retained respondents replied to every scenario, which leads to a much higher number of responses per study. The reason for such a small retention lies in duplicate and uniform responses. Some duplicate responses occurred if a person clicked on ‘Finish’ more than once before receiving feedback from the survey platform. Additionally, the difference between surveys and completion approaches occurs, for most cases, when respondents provide informed consent, but after that, leave the survey (the next approach to filling in the survey registered them as different respondents). Incomplete and duplicate responses to scenarios were excluded, as well as unusual cases detected in the data examination. Uniform responses to open-ended questions (as in ‘I don’t know’ or ‘I cannot assess’) between scenarios are used to exclude all answers by identified respondents from the data. Another reason for the lower shares of usable data may be the questionnaire length, so the possibility of using the scenarios separately must be examined.

While the first sample is more heterogeneous, the second and the third samples regard respondents with similar backgrounds. Despite homogeneity by some characteristics, some of them could have experience or knowledge about one or more situations examined in the scenarios, while others will face the scenarios without prior ‘intuitive probabilities’ assigned to the events. If previous experiences or knowledge vary too much between the respondents, it could affect responses and consequently the questionnaire’s internal reliability.

2.4. Coding Issues

The questionnaire involves qualitative responses, which should be recoded for further analysis. A specified amount of certainty is assigned to each answer as a percentage (0–100%), which does not require recoding. All ‘yes’ and ‘no’ answers are coded 1 and 0, respectively. The ‘don’t know’ options were supposed to highlight uncertainty and were initially coded as 0.5, but the problem occurs in the interpretation: a person does not know and is, for example, 50% confident that s/he does not know. If a complementary probability approach is applied, then it follows that there is 50% certainty that the person does know. However, it remains unclear whether the response would be ‘yes’ or ‘no,’ although it is plausible that the response would lean towards the ‘no’ option. While the interpretation does not make much sense from a mathematical standpoint, the reason for such choices may be the lack of information or the lack of trust in one’s own assessment. While the ‘don’t know’ option offers insight for descriptive, qualitative analysis, it represents a difficulty for quantitative analysis. For further analysis, ‘don’t know’ options were not excluded from the 2019 study data to prevent data loss (all responses for a scenario per respondent should be removed if one ‘don’t know’ option appeared) and were coded here as 0. This issue does not occur for the second and third studies, as there were no ‘don’t know’ options.

The next issue regards open-ended questions (Q3, Q7, and Q13, Table A1) and the coding of explanatory answers. The content of the answers was coded by only one human coder—the researcher. There are two possible approaches, coding by the loose rules (LR) and coding by the strict rules (SR). The first step in both approaches was an a priori approach to the coding [35,36], where the replies containing keywords strictly in line with the scenario were coded 1 (with the help of the search option). The second step included another overview to check for responses with different word choices that reflect the meaning in line with the scenario content. The first seven scenarios do not offer all elements, so the next step required evaluating answers about not offered elements by the criteria of reasoning that directly reflects the information provided in the scenarios. In SR coding, the rest of the replies were coded 0, while these steps created a base for the next step by the LR coding. Within the next step in LR coding, the rest of the responses were additionally examined by the criteria of any possible reasoning of responses in line with the scenarios. Moreover, one of the advantages of the scenario technique—multiple interpretations that can be considered in terms of their plausibility [19]—could not be utilized without such an approach. While this step might be observed through the lenses of content analysis, it must be noted that this part of coding included only an evaluation of whether the offered responses could be regarded as a possible element assessment (coded as 0 or 1), without further description, a subcategories derivation, an additional thematic clustering, or a narration of the answer to the research question, which are the usual elements of content analysis [37,38].

Nevertheless, that step may be observed in line with the interpretivist paradigm, which must be addressed. When a researcher performs coding in the content analysis alone, there is a risk of subjectivism, leading to potential replicability problems [36,37]. The usual approach to the issue is using coders unrelated to the research (or unaware of the research hypothesis) and checking inter-rater reliability. However, that approach is not always recommended and can also be criticized [39,40]. In addition, it can be “beneficial that the analyst is able to use their own unique skills, talents and expertise” in choosing the method and coding qualitative responses [36], but with constant internal scrutiny. The rest of the coding was conducted with a continuous reflective audit to acknowledge and minimize subjective framing [36,41]. While performing the third step in LR coding, due attention was paid to coding similar concepts of responses in the same way [42]. The revealed possible interpretations that can be reasoned from the scenarios were compared and their meaning in relation to scenarios rechecked. The following explanations and examples will describe LR coding issues and display the comparison to SR coding.

The idea behind the LR approach is to allow the respondents to subjectively perceive the available information about a situation. The responses are coded as 1 if respondents provided any assessment that can be reasoned from the scenario. This approach is considered more appropriate for examining how individuals actually assess situations and form their awareness. The SR approach is a ‘modeler’s perspective’ approach, where only the responses deduced directly from scenarios are coded as 1. For the first seven scenarios, the difference between LR and SR data is smaller. An example from scenario 8 paints a picture of the differences in the assessment and game understanding: a part of the respondents perceived the situation as a cheap-talk game with the interrogator (examples of responses for the player identification in such cases: “ethical committee,” “a member of the ethical committee,” “the professor who accused me”), which was coded as 1 by LR, and 0 by SR. In the SR approach, only the responses about player identification that indicated a colleague were coded 1, else 0. In scenario 9, a part of the respondents observed the situation as the problem of commons, where someone took an extra sandwich or the organizers did not prepare enough for everyone (thus, a kind of a protest is in order) with activity descriptions such as “they did not have enough sandwiches to share” in combination with the player described as “the person in charge for the brunch” (coded 1 by LR, 0 by SR).

Another example is from scenario 7, where a few respondents introduced their family or friends as players, explaining that they must have contacted investors who offered to pitch the idea. In the same scenario, a few respondents described the activity as an “economic activity,” coded 1 by LR and 0 by SR (as no explicit description related to the scenario was offered). In scenario 1, a part of the respondents introduced the other buyers or product presenters as players and used their reactions to the price increase to justify their positions (mostly decisions to buy) because of persuasion (“influencers impact my thinking about what I like and love”, “…in the shop, there are employees that perform so-called activities that may influence my purchase”, “other buyers must be protesting to such things”—all coded LR: 1, SR: 0). In the same scenario, if an outcome description was “I will be sad because of a loss of a chocolate,” it was coded 1 by both LR and SR. However, if the outcome was described only as “I will be sad” (or any other affective state, without the explanation or the relationship with the scenario), it was coded 0. The same approach was applied to similar situations in different scenarios. While a relatively small percentage deviates as much from the information provided in the scenario, it is relevant to note the differences and address them. All descriptive responses of “I don’t know”, “I’m not sure”, or “I am uncertain” were coded 0. It might be interesting that in each of the studies, a couple of responses about the player and activity description that sporadically occurred were, for example, “no one can influence me”, “no one can influence the outcomes in my life but me”, or “there may not exist such activity that would influence my outcomes” (as in general), and were coded 0 (although, interestingly, the outcomes’ descriptions of the same respondents indicated that they do recognize the differences in the outcomes). While coding, descriptive responses about the activity, player(s), and the outcome have been considered together to get a comprehensive insight about a person’s grasp of the situation, as it can be seen from some of the quoted descriptions that the activity, players, and the outcome may be interlinked. The frequencies for binary-coded questions are presented in

Table A4. Frequencies for binary-coded answers.

	Study	2019		2020		2021
Scenario	Questions	% (q = 1) LR	% (q = 1) SR	% (q = 1) LR	% (q = 1) SR	% (q = 1) LR	% (q = 1) SR
S1	Q1	83.62	/	87.18	/	77.08	/
	Q3	73.28	75.00	83.33	56.96	56.25	32.65
	Q5	76.72	/	87.18	/	77.08	/
	Q7	57.76	59.48	79.49	58.23	62.50	55.10
	Q9	73.28	/	79.49	/	70.83	/
	Q13	76.72	72.41	92.31	87.34	85.42	61.22
	Q15	96.55	/	85.90	/	89.58	/
	Q17	74.14	/	67.95	/	60.42	/
	Q19	/	/	79.49	/	77.08	/
	n	116	114	78	79	48	49
S2	Q1	75.86	/	91.76	/	68.97	/
	Q3	49.14	47.41	81.18	79.01	55.17	52.63
	Q5	70.69	/	78.82	/	67.24	/
	Q7	47.41	51.72	65.88	70.37	56.90	54.39
	Q9	53.45	/	43.53	/	53.45	/
	Q13	49.14	45.69	84.71	76.54	65.52	40.35
	Q15	95.69	/	87.06	/	86.21	/
	Q17	47.41	/	61.18	/	55.17	/
	Q19	/	/	85.88	/	91.38	/
	n	116	113	85	81	58	57
S3	Q1	86.44	/	90.59	/	80.70	/
	Q3	69.49	65.25	82.35	76.47	64.91	61.40
	Q5	75.42	/	92.94	/	78.95	/
	Q7	59.32	62.71	85.88	77.65	68.42	66.67
	Q9	61.86	/	64.71	/	75.44	/
	Q13	70.34	67.80	90.59	76.47	78.95	59.65
	Q15	96.61	/	90.59	/	82.46	/
	Q17	74.58	/	91.76	/	78.95	/
	Q19	/	/	83.53	/	87.72	/
	n	118	118	85	85	57	57
S4	Q1	84.75	/	81.18	/	85.00	/
	Q3	79.66	73.73	69.41	71.43	73.33	73.77
	Q5	73.73	/	82.35	/	85.00	/
	Q7	65.25	64.41	75.29	73.81	71.67	72.13
	Q9	68.64	/	78.82	/	88.33	/
	Q13	76.27	0.00	96.47	95.24	83.33	72.13
	Q15	96.61	/	81.18	/	91.67	/
	Q17	59.32	/	62.35	/	70.00	/
	Q19	/	/	58.82	/	71.67	/
	n	118	118	85	84	60	61
S5	Q1	74.79	/	92.86	/	77.78	/
	Q3	73.11	65.55	79.76	82.35	74.07	74.55
	Q5	73.95	/	89.29	/	87.04	/
	Q7	60.50	60.50	77.38	77.65	79.63	80.00
	Q9	63.03	/	60.71	/	75.93	/
	Q13	68.07	65.55	96.43	85.88	85.19	72.73
	Q15	96.64	/	84.52	/	90.74	/
	Q17	57.14	/	63.10	/	66.67	/
	Q19	/	/	90.48	/	87.04	/
	n	119	119	84	85	54	55
S6	Q1	79.13	/	88.24	/	90.91	/
	Q3	72.17	63.48	85.88	79.76	83.64	80.36
	Q5	75.65	/	88.24	/	87.27	/
	Q7	62.61	66.09	76.47	73.81	74.55	76.79
	Q9	68.70	/	82.35	/	89.09	/
	Q13	75.65	66.09	96.47	90.48	90.91	82.14
	Q15	97.39	/	84.71	/	92.73	/
	Q17	64.35	/	80.00	/	63.64	/
	Q19	/	/	90.59	/	96.36	/
	n	115	115	85	84	55	56
S7	Q1	83.33	/	96.47	/	91.07	/
	Q3	73.33	30.83	82.35	78.82	75.00	75.00
	Q5	72.50	/	90.59	/	80.36	/
	Q7	51.67	50.00	65.88	61.76	78.57	75.00
	Q9	65.83	/	92.94	/	85.71	/
	Q13	71.67	59.17	95.29	95.29	87.50	83.93
	Q15	97.50	/	84.71	/	92.86	/
	Q17	68.33	/	77.65	/	73.21	/
	Q19	/	/	81.18	/	92.86	/
	n	120	120	85	85	56	56
S8	Q1	/	/	82.14	/	83.64	/
	Q3	/	/	84.52	63.86	74.55	74.55
	Q5	/	/	82.14	/	85.45	/
	Q7	/	/	78.57	36.14	63.64	25.45
	Q9	/	/	73.81	/	87.27	/
	Q13	/	/	96.43	93.98	90.91	78.18
	Q15	/	/	89.29	/	87.27	/
	Q17	/	/	67.86	/	65.45	/
	Q19	/	/	92.86	/	92.73	/
	n	0	0	84	83	55	55
S9	Q1	/	/	89.41	/	83.64	/
	Q3	/	/	82.35	80.95	78.18	76.36
	Q5	/	/	85.88	/	78.18	/
	Q7	/	/	78.82	77.38	74.55	70.91
	Q9	/	/	90.59	/	83.64	/
	Q13	/	/	90.59	88.10	83.64	78.18
	Q15	/	/	78.82	/	81.82	/
	Q17	/	/	70.59	/	49.09	/
	Q19	/	/	97.65	/	90.91	/
		0	0	85	84	55	55

and enable a comparison of the data per scenario and study, given the LR and SR approach.

The open-ended questions are relevant, as they provide a deeper insight into awareness and assessment that cannot be revealed by yes/no questions. In this case, they were coded in the simplest way to be utilized for questionnaire reliability and validity examination. However, these questions offer much more by enabling a descriptive, qualitative analysis. In addition, the responses may be used for future questionnaire development by forming statements concerning nine scenarios.

The set of questions was later used to derive nine items: activity awareness (I1), activity assessment (I2), player awareness (I3), player assessment (I4), other players’ outcome awareness (I5), own outcome assessment (I6), game existence awareness (I7), deliberation (I8), and own activity awareness (I9). Two approaches were used for that: with and without assuming that the belief is governed by complementary probability (see Items in Table A1). Following the first approach, the binary answers were multiplied by their respective beliefs and divided by 100 to create a respondent’s value for an item. Following the second approach, the calculation was applied: if the binary coded answers took the value of 1, then the item was created the same way as for the first approach; if the binary coded answers took the value of 0, then the complementary probability (CP) was used to derive a value. The framing effect suggests that individuals would respond differently if asked ‘is there no activity,’ so this represents a modeler’s standpoint, ignoring possible biases. Likewise, Piermont [9] examines decision makers’ unawareness and offers two explanatory sentences: (1) “It is raining or it is not raining”; and (2) “The axiom of choice implies that every vector space has a basis”; and concludes that although they are tautologies, the decision-maker will treat these statements differently if s/he is unaware of set theory. Following the same thought, in this case, respondents should understand complementary probabilities and apply the rule $p\left(A\right)+p\left(A^C\right)=1$ when assigning probabilities. Although there are theoretical considerations that argue against CP coding, that approach will also be explored here, and the results will govern the decision about its applicability.

While the first approach (LR) may seem stricter, it reflects provided choices and responses directly. The second approach (SR CP) can be observed as a modeler’s perspective, with an underlying assumption of logic and complementary probability. It can be argued that the second approach is favored if the goal is to fit the empirical data into the game theory framework and measure deviations. In contrast, the first one is favored if the goal is to gain insights from empirical behavioral data and use them to improve modeling [10], as is the case in the behavioral economy using psychological approaches.

Given the different coding possibilities, three datasets were created from the original data, coded by: loose rules without complementary probability (LR), loose rules with complementary probability (LR CP), and strict rules with complementary probability (SR CP). Given the coding issues, validation and reliability must be examined not only from the perspective of the questionnaire, but the coding variations as well.

2.5. Reliability and Validity

The next step in the questionnaire’s development is an examination of its validity and reliability. Reliability can be observed from three aspects: equivalence, stability, and internal consistency [30]. The first one denotes the equivalence of measures by applying two or more instruments that measure the same phenomenon. This aspect cannot be examined, as no other questionnaire of game awareness is known to the author, so none was applied. Stability denotes how stable the measurement is, in the sense that the same or similar measures are obtained with repeated measurements. This involves repeated instrument administration to the same group of respondents over time. In this case, a proxy for the stability is considered—the results of instrument application to distinct groups of respondents with similar characteristics and backgrounds. Internal reliability, or homogeneity, denotes how consistent the instrument is over all its parts [31,43]. For this purpose, Cronbach’s alpha coefficient is usually used [24,31,43,44] The decision-making criteria for using Cronbach’s alpha are: 0.9–1 is excellent reliability, 0.7–0.9 is high reliability, and 0.5–0.7 is moderate reliability [34]. Kline [45] suggests the value of 0.8 as a threshold for ‘very good’ reliability, and that approach will be used here. While it is a useful indicator, reliability must be combined with the validity, and vice versa, as instruments should be both valid and reliable [46,47].

Validity measures how successfully the questionnaire captures what was intended to be researched. In other words, it answers the questions of whether it measured what was intended to be measured. Several aspects of validity can be explored, such as content validity, construct validity, and criterion validity [43]. Moreover, construct validity should always be examined to justify the meaning of the measure and its subsequent use [48]. Convergent construct validity refers to the degree to which an examined measure correlates with measures that assess the same phenomenon.

In contrast, divergent (discriminant) construct validity indicates variables that are not supposed to be correlated to an examined construct and, that in fact, are not correlated [49]. Most often, it is performed using factor analysis [43]. Exploratory factor analysis can be conducted using different approaches, but principal component analysis (PCA) is often applied. However, whether exploratory or confirmatory, factor analysis requires data that satisfy the assumptions of a linear relationship, multivariate normality, a correlation between variables and factors, and no multicollinearity [50]. While in some cases of minor skewness in the distribution of individual variables, normality may be considered a soft requirement, a detailed examination reveals that the data are not normal, and transformation does not levigate the issue. While there are weak-to-moderate correlations between the variables and no multicollinearity, pair-wise observation reveals that underlying relationships are not linear. These data require nonparametric approaches, such as logistic, logit, and probit models, Spearman correlation coefficient, clustering, decision trees, or k-nn (kth nearest neighbor) analysis.

Each of these methods have pros and cons. Most of the proposed nonparametric approaches allow conclusions about discriminant construct validity, but logit and probit models also allow for decisions on the predictive power of the instrument (criterion validity). However, logit and probit models require dependent variables, meaning that one of the variables should be treated as a dependent; as a logical and theoretically supported choice, that would be I7 (game elements should be detected and assessed to form the game awareness). However, logit models belong to a group of models with limited dependent variables, meaning that I7 should be recoded into a binary or ordinal variable, which implies information loss.

The most frequently used approach for nonparametric discriminant analysis is k-nn analysis, which is a classification method. The method classifies observations, given their distance to the nearest neighbors. The choice of the number of neighbors to use for observation classification is governed by error minimization. The analysis can be conducted with or without a target variable and allows for additional validation by partitioning the sample into training and testing subgroups. There are more approaches to measuring distances between the neighbors, but the Euclidian metric is often applied. Usually, the mean of nearest neighbor values is used to predict scale target, but median values can be used too. The k-nn analysis will be conducted by setting I7 as a target variable (allowing the conclusion about criterion and discriminant construct validity), with partitioning the sample to the training (60%) and testing (40%) part, with medians of the nearest neighbor values used for the prediction of a scale target and, Euclidian metric used for the distances between the neighbors.

There is also a nonparametric alternative for exploratory factor analysis that can be used for convergent construct validity assessment. Such an approach is based on a scale-free, nonparametric, Spearman’s correlation coefficient matrix, instead of a Pearson’s correlation coefficient matrix [51]. As Pearson’s and Spearman’s correlation coefficients lead to the same values if scale variable values are replaced with their ranks, a rank transformation is a link that connects parametric and nonparametric statistics [51,52,53]. In general, Spearman’s rho is a much better choice for heavy tails and asymmetric distributions [52]. If used in a PCA, “factor structure is more stable if researchers simply base their multivariate analysis on the r_s matrix” [52] (p. 287). The rest of the exploratory factor analysis procedure follows the standard approach, using matrix algebra to derive the component matrix, where the criteria for the number of components are eigenvalues greater than or equal to one and relevant factor loadings take values greater than or equal to 0.4. The analysis will be conducted using SPSS (IBM, New York, NY, USA) and exploratory factor analysis based on Spearman correlation coefficients matrix with principal component analysis as extraction method will be used following [54]’s approach.

However, validity somewhat relies on the researcher’s interpretation, so reliability is required to assure validity [55,56]. In this case, reliability and validity refer to the questionnaire written in the original language (Croatian), and not the translated English version, both of which are presented in Table A1. If one wanted to use the questionnaire in another language, its reliability and validity examination should be repeated [30].

3. Results

3.1. Reliability

Reliability was examined per coding type, study, and scenario, and for different combinations of items. It can be noticed, in

Table 3. Reliability statistics per study with 7, 8, and 9 items included, respective to the coding.

Coding	LR			LR CP			SR CP
Study/No. of Items	9	8	7	9	8	7	9	8	7
2019	/	0.67 (0.751)	0.659 (0.755)	/	0.761 (0.763)	0.771 (0.771)	/	0.74 (0.744)	0.757 (0.761)
2020	0.744 (0.751)	0.758 (0.767)	0.759 (0.764)	0.724 (0.73)	0.742 (0.747)	0.75 (0.753)	0.677 (0.684)	0.689 (0.699)	0.695 (0.703)
2021	0.819 (0.818)	0.829 (0.823)	0.841 (0.84)	0.805 (0.802)	0.817 (0.819)	0.826 (0.826)	0.7812 (0.782)	0.795 (0.801)	0.796 (0.802)

Notes: For 9 examined items: I1–I9; 8 examined items: I1–I8; 7 examined items: I1–I7. Cronbach’s alpha is reported in the brackets, based on standardized items. LR stands for loosely coded data, SR stands for strictly coded data and CP stands for complementary probability applied to certainties.

, that the highest reliability measures appear for the 2021 study and LR coding, pointing out a high (though not excellent) reliability. It can be noticed that only LR and LR CP data from 2021 meet the threshold of 0.8 [45].

At this point, it can be concluded that the last version of the questionnaire, when coded by LR, measures game awareness best. The lowest reliability statistics occurred for the 2019 study. This was expected, due to revealed necessities for changes during the questionnaire’s formation and “don’t know” responses recoded into 0. The lower reliability related to SR CP coding indicates that such coding could be a somewhat worse measure of game awareness between scenarios, while still reaching an acceptable level of reliability for the 2021 study [34].

However, these are statistics for the entire questionnaire, while variations among scenarios might differ. The best results are shown in

Table 4. Reliability statistics for the 2021 study data with I1–I9 and I1–I7 items, coded LR.

Reliability Statistics	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items
Scenario 1	0.750	0.743	9	0.780	0.767	7
Scenario 2	0.861	0.853	9	0.870	0.870	7
Scenario 3	0.852	0.847	9	0.875	0.875	7
Scenario 4	0.783	0.797	9	0.811	0.815	7
Scenario 5	0.727	0.735	9	0.782	0.779	7
Scenario 6	0.846	0.862	9	0.884	0.890	7
Scenario 7	0.842	0.845	9	0.832	0.833	7
Scenario 8	0.824	0.822	9	0.833	0.833	7
Scenario 9	0.778	0.779	9	0.794	0.794	7

, while results for the complete analysis can be found in

Table A5. Reliability statistics for items I1–I9 per scenario, study, and type of coding.

Coding	LR			LR CP			SR CP
Reliability Statistics	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items
2020 study S1	0.777	0.788	9	0.765	0.780	9	0.668	0.683	9
2020 study S2	0.766	0.775	9	0.742	0.753	9	0.677	0.695	9
2020 study S3	0.696	0.708	9	0.691	0.695	9	0.599	0.618	9
2020 study S4	0.713	0.730	9	0.663	0.683	9	0.621	0.644	9
2020 study S5	0.595	0.640	9	0.551	0.589	9	0.573	0.604	9
2020 study S6	0.793	0.802	9	0.788	0.795	9	0.785	0.792	9
2020 study S7	0.707	0.726	9	0.704	0.720	9	0.714	0.738	9
2020 study S8	0.758	0.754	9	0.747	0.743	9	0.544	0.551	9
2020 study S9	0.814	0.820	9	0.773	0.785	9	0.785	0.796	9
2021 study S1	0.750	0.743	9	0.712	0.706	9	0.680	0.667	9
2021 study S2	0.861	0.853	9	0.852	0.841	9	0.834	0.822	9
2021 study S3	0.852	0.847	9	0.848	0.845	9	0.803	0.803	9
2021 study S4	0.783	0.797	9	0.777	0.784	9	0.773	0.781	9
2021 study S5	0.727	0.735	9	0.718	0.726	9	0.706	0.720	9
2021 study S6	0.846	0.862	9	0.799	0.815	9	0.792	0.808	9
2021 study S7	0.842	0.845	9	0.817	0.825	9	0.816	0.825	9
2021 study S8	0.824	0.822	9	0.770	0.763	9	0.711	0.715	9
2021 study S9	0.778	0.779	9	0.778	0.779	9	0.779	0.781	9

and

Table A6. Reliability statistics for items I1–I7 per scenario, study, and type of coding.

	LR			LR CP			SR CP
	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items	Cronbach’s Alpha	Cronbach’s Alpha Based on Standardized Items	No. of Items
2019 study S1	0.793	0.793	7	0.778	0.779	7	0.786	0.788	7
2019 study S2	0.660	0.679	7	0.664	0.675	7	0.660	0.659	7
2019 study S3	0.756	0.781	7	0.779	0.778	7	0.753	0.757	7
2019 study S4	0.350	0.760	7	0.801	0.801	7	0.807	0.811	7
2019 study S5	0.784	0.789	7	0.747	0.749	7	0.714	0.727	7
2019 study S6	0.825	0.836	7	0.795	0.795	7	0.804	0.808	7
2019 study S7	0.823	0.826	7	0.803	0.799	7	0.709	0.725	7
2020 study S1	0.827	0.827	7	0.827	0.827	7	0.699	0.714	7
2020 study S2	0.793	0.800	7	0.793	0.800	7	0.709	0.727	7
2020 study S3	0.674	0.680	7	0.674	0.680	7	0.617	0.632	7
2020 study S4	0.750	0.754	7	0.750	0.754	7	0.686	0.690	7
2020 study S5	0.612	0.660	7	0.612	0.660	7	0.580	0.611	7
2020 study S6	0.792	0.804	7	0.792	0.804	7	0.774	0.783	7
2020 study S7	0.755	0.761	7	0.755	0.761	7	0.754	0.769	7
2020 study S8	0.784	0.782	7	0.784	0.782	7	0.578	0.581	7
2020 study S9	0.798	0.802	7	0.798	0.802	7	0.795	0.802	7
2021 study S1	0.780	0.767	7	0.747	0.744	7	0.708	0.704	7
2021 study S2	0.870	0.870	7	0.868	0.866	7	0.849	0.849	7
2021 study S3	0.875	0.875	7	0.864	0.865	7	0.809	0.816	7
2021 study S4	0.811	0.815	7	0.790	0.791	7	0.786	0.789	7
2021 study S5	0.782	0.779	7	0.793	0.796	7	0.766	0.781	7
2021 study S6	0.884	0.890	7	0.842	0.848	7	0.826	0.836	7
2021 study S7	0.832	0.833	7	0.782	0.785	7	0.776	0.782	7
2021 study S8	0.833	0.833	7	0.772	0.775	7	0.688	0.707	7
2021 study S9	0.794	0.794	7	0.805	0.805	7	0.798	0.799	7

.

The results point out that there is high reliability, meaning that there is consistency between the items. While still indicating acceptable reliability according to [34], the lowest measures occurred for scenarios 1, 5, and 9. For scenario 9, the reliability statistics improve (0.803) if examined for six items (removing I6), but the same approach shows no improvement for scenarios 1 and 5, indicating their potential exclusion from the instrument.

As for the proxy for stability, Table A4 and

Table A7. Descriptive statistics for certainties per scenario and study.

Descriptive Statistics 2019 Study						Descriptive Statistics 2020 Study						Descriptive Statistics 2021 Study
S1 n = 116	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S1 n = 78	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S1 n = 48	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
Q2	20.00	100.00	85.457	1.714	18.458	Q2	50.00	100.000	81.218	1.879	16.597	Q2	0.000	100.000	67.542	4.091	28.344
Q4	0.00	100.00	89.647	1.873	20.171	Q4	0.00	100.000	78.949	2.553	22.545	Q4	0.000	100.000	80.792	3.783	26.212
Q6	0.00	100.00	89.224	1.886	20.309	Q6	0.00	100.000	84.103	2.551	22.528	Q6	50.000	100.000	88.729	2.437	16.887
Q8	0.00	100.00	86.716	1.902	20.488	Q8	0.00	100.000	81.090	2.208	19.497	Q8	45.000	100.000	89.229	2.474	17.141
Q10	0.00	100.00	83.483	2.161	23.280	Q10	30.00	100.000	81.231	2.180	19.249	Q10	20.000	100.000	82.292	3.169	21.954
Q14	0.00	100.00	88.284	2.127	22.913	Q14	30.00	100.000	86.500	1.970	17.395	Q14	50.000	100.000	90.229	2.150	14.896
Q16	2.00	100.00	85.664	1.967	21.191	Q16	45.00	100.000	87.718	1.685	14.879	Q16	30.000	100.000	88.375	2.650	18.359
Q18	8.00	100.00	85.534	2.121	22.841	Q18	40.00	100.000	82.462	2.013	17.780	Q18	50.000	100.000	85.125	2.844	19.704
Q20						Q20	50.00	100.000	86.256	1.855	16.385	Q20	20.000	100.000	87.646	2.825	19.569
S2 n = 116	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S2 n = 85	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S2 n = 58	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
Q2	3.00	100.00	82.862	2.045	22.030	Q2	20.00	100.000	85.612	1.831	16.885	Q2	0.000	100.000	82.845	3.100	23.606
Q4	0.00	100.00	83.621	2.200	23.695	Q4	0.00	100.000	81.529	2.445	22.546	Q4	20.000	100.000	88.845	2.553	19.442
Q6	0.00	100.00	84.741	2.251	24.243	Q6	20.00	100.000	85.212	2.162	19.933	Q6	40.000	100.000	88.569	2.585	19.689
Q8	0.00	100.00	85.724	2.196	23.652	Q8	0.00	100.000	83.271	2.480	22.865	Q8	0.000	100.000	89.431	2.653	20.202
Q10	0.00	100.00	82.526	2.133	22.972	Q10	20.00	100.000	78.976	2.152	19.836	Q10	20.000	100.000	81.638	3.000	22.846
Q14	0.00	100.00	85.129	2.069	22.279	Q14	0.00	100.000	83.400	2.331	21.492	Q14	30.000	100.000	85.776	2.776	21.140
Q16	0.00	100.00	86.560	1.899	20.449	Q16	0.00	100.000	81.894	2.191	20.198	Q16	25.000	100.000	86.190	2.623	19.975
Q18	0.00	100.00	78.060	2.465	26.545	Q18	20.00	100.000	79.871	2.237	20.620	Q18	20.000	100.000	80.621	3.125	23.801
Q20						Q20	20.00	100.000	87.388	1.864	17.182	Q20	50.000	100.000	86.000	2.205	16.792
S3 n = 118	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S3 n = 85	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S3 n = 57	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
Q2	20.00	100.00	90.042	1.625	17.652	Q2	50.00	100.000	89.906	1.640	15.121	Q2	20.000	100.000	87.895	2.517	19.006
Q4	30.00	100.00	88.441	1.578	17.140	Q4	0.00	100.000	86.082	2.186	20.153	Q4	0.000	100.000	84.649	3.114	23.511
Q6	0.00	100.00	88.958	1.821	19.778	Q6	10.00	100.000	89.365	1.930	17.794	Q6	20.000	100.000	87.789	2.616	19.748
Q8	0.00	100.00	87.051	2.051	22.284	Q8	10.00	100.000	88.106	1.994	18.387	Q8	20.000	100.000	88.175	2.798	21.128
Q10	0.00	100.00	81.653	2.040	22.165	Q10	10.00	100.000	80.600	2.200	20.281	Q10	20.000	100.000	85.175	2.684	20.263
Q14	0.00	100.00	84.373	2.089	22.697	Q14	0.00	100.000	85.059	2.308	21.282	Q14	0.000	100.000	83.649	3.383	25.540
Q16	0.00	100.00	84.068	2.126	23.089	Q16	50.00	100.000	88.506	1.562	14.403	Q16	20.000	100.000	88.018	2.453	18.523
Q18	0.00	100.00	85.186	2.196	23.856	Q18	50.00	100.000	89.200	1.603	14.783	Q18	50.000	100.000	88.509	2.279	17.207
Q20						Q20	46.00	100.000	86.035	2.011	18.537	Q20	30.000	100.000	88.474	2.472	18.660
S4 n = 118	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S4 n = 85	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S4 n = 60	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
Q2	0.00	100.00	87.975	1.863	20.239	Q2	30.00	100.000	87.353	1.788	16.486	Q2	39.000	100.000	88.217	2.455	19.016
Q4	7.00	100.00	83.644	1.923	20.894	Q4	20.00	100.000	84.247	2.021	18.635	Q4	20.000	100.000	83.900	2.717	21.044
Q6	0.00	100.00	85.398	2.032	22.076	Q6	20.00	100.000	85.294	2.103	19.392	Q6	20.000	100.000	88.150	2.609	20.211
Q8	0.00	100.00	85.186	2.080	22.594	Q8	0.00	100.000	84.494	2.172	20.022	Q8	10.000	100.000	87.700	2.745	21.266
Q10	0.00	100.00	82.280	2.287	24.838	Q10	20.00	100.000	84.012	2.029	18.706	Q10	40.000	100.000	86.917	2.460	19.055
Q14	0.00	100.00	86.568	1.984	21.555	Q14	10.00	100.000	86.659	1.968	18.143	Q14	0.000	100.000	90.267	2.535	19.636
Q16	20.00	100.00	87.161	1.856	20.165	Q16	0.00	100.000	84.765	2.125	19.591	Q16	35.000	100.000	88.583	2.323	17.996
Q18	0.00	100.00	82.017	2.313	25.121	Q18	20.00	100.000	82.082	2.053	18.930	Q18	34.000	100.000	86.150	2.451	18.985
Q20						Q20	50.00	100.000	84.129	1.898	17.503	Q20	28.000	100.000	84.717	2.629	20.365
S5 n = 119	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S5 n = 84	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S5 n = 54	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
Q2	0.00	100.00	85.193	2.051	22.377	Q2	50.00	100.000	89.643	1.649	15.111	Q2	50.000	100.000	86.852	2.539	18.661
Q4	0.00	100.00	85.538	2.015	21.985	Q4	40.00	100.000	84.369	1.874	17.172	Q4	30.000	100.000	88.056	2.599	19.097
Q6	0.00	100.00	88.345	1.917	20.908	Q6	40.00	100.000	88.679	1.973	18.086	Q6	25.000	100.000	90.778	2.512	18.458
Q8	0.00	100.00	86.731	2.045	22.303	Q8	0.00	100.000	89.405	2.128	19.500	Q8	1.000	100.000	89.444	2.666	19.591
Q10	0.00	100.00	85.076	2.150	23.452	Q10	50.00	100.000	82.917	1.962	17.983	Q10	20.000	100.000	84.130	2.764	20.310
Q14	0.00	100.00	84.361	2.112	23.037	Q14	10.00	100.000	85.643	2.119	19.417	Q14	40.000	100.000	86.907	2.485	18.261
Q16	0.00	100.00	85.571	2.198	23.979	Q16	30.00	100.000	85.940	1.842	16.886	Q16	40.000	100.000	90.667	2.081	15.293
Q18	0.00	100.00	82.336	2.285	24.925	Q18	0.00	100.000	80.036	2.254	20.660	Q18	20.000	100.000	80.963	3.002	22.056
Q20						Q20	50.00	100.000	87.381	1.705	15.627	Q20	40.000	100.000	87.944	2.320	17.045
S6 n = 115	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S6 n = 85	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S6 n = 55	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
Q2	0.00	100.00	89.061	2.116	22.691	Q2	30.00	100.000	90.212	1.858	17.133	Q2	25.000	100.000	90.345	2.409	17.868
Q4	0.00	100.00	84.730	2.344	25.132	Q4	0.00	100.000	85.929	2.431	22.413	Q4	10.000	100.000	86.473	3.021	22.408
Q6	0.00	100.00	86.061	2.104	22.559	Q6	30.00	100.000	89.318	2.004	18.474	Q6	1.000	100.000	89.545	2.847	21.112
Q8	0.00	100.00	86.643	2.123	22.772	Q8	50.00	100.000	88.988	1.754	16.172	Q8	10.000	100.000	88.473	2.950	21.881
Q10	0.00	100.00	84.009	2.192	23.505	Q10	50.00	100.000	89.788	1.712	15.787	Q10	20.000	100.000	84.909	3.027	22.447
Q14	0.00	100.00	87.861	1.908	20.460	Q14	0.00	100.000	85.906	2.247	20.715	Q14	1.000	100.000	86.127	2.931	21.740
Q16	0.00	100.00	86.843	1.989	21.335	Q16	20.00	100.000	85.447	1.940	17.887	Q16	50.000	100.000	89.545	2.269	16.830
Q18	0.00	100.00	83.078	2.091	22.428	Q18	0.00	100.000	83.988	2.254	20.780	Q18	34.000	100.000	82.709	2.828	20.972
Q20						Q20	10.00	100.000	86.341	2.084	19.210	Q20	50.000	100.000	90.327	2.141	15.882
S7 n = 120	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S7 n = 85	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S7 n = 56	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
Q2	0.00	100.00	88.842	1.901	20.830	Q2	50.00	100.000	92.388	1.497	13.804	Q2	40.000	100.000	90.357	2.385	17.850
Q4	0.00	100.00	85.458	2.187	23.962	Q4	50.00	100.000	89.506	1.732	15.966	Q4	40.000	100.000	85.357	2.713	20.304
Q6	0.00	100.00	85.367	2.047	22.426	Q6	46.00	100.000	87.800	1.867	17.210	Q6	38.000	100.000	88.464	2.578	19.293
Q8	0.00	100.00	85.208	2.181	23.891	Q8	0.00	100.000	87.671	2.107	19.422	Q8	30.000	100.000	88.625	2.649	19.820
Q10	0.00	100.00	81.583	2.344	25.677	Q10	40.00	100.000	87.400	1.882	17.354	Q10	20.000	100.000	85.625	2.916	21.818
Q14	0.00	100.00	85.275	2.161	23.675	Q14	50.00	100.000	90.576	1.526	14.069	Q14	20.000	100.000	89.304	2.364	17.687
Q16	0.00	100.00	86.233	2.037	22.312	Q16	6.00	100.000	87.447	1.930	17.795	Q16	40.000	100.000	85.661	2.663	19.930
Q18	0.00	100.00	84.258	2.174	23.814	Q18	43.00	100.000	86.576	1.881	17.343	Q18	44.000	100.000	84.089	2.594	19.415
Q20						Q20	20.00	100.000	88.200	1.978	18.236	Q20	8.000	100.000	88.589	2.628	19.664
						S8 n = 84	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S8 n = 55	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
						Q2	30.00	100.000	89.762	1.883	17.257	Q2	10.000	100.000	88.145	2.920	21.652
						Q4	30.00	100.000	84.464	2.272	20.823	Q4	0.000	100.000	86.727	3.177	23.564
						Q6	10.00	100.000	87.536	2.319	21.255	Q6	10.000	100.000	88.345	2.914	21.613
						Q8	1.00	100.000	85.583	2.449	22.449	Q8	10.000	100.000	87.691	3.139	23.276
						Q10	35.00	100.000	87.548	1.954	17.907	Q10	10.000	100.000	85.964	3.000	22.250
						Q14	30.00	100.000	84.702	2.095	19.203	Q14	10.000	100.000	88.473	2.613	19.381
						Q16	50.00	100.000	88.857	1.734	15.889	Q16	45.000	100.000	89.127	2.341	17.361
						Q18	0.00	100.000	85.524	2.141	19.623	Q18	20.000	100.000	86.800	2.704	20.052
						Q20	46.00	100.000	90.976	1.777	16.283	Q20	50.000	100.000	91.855	1.918	14.222
						S9 n = 85	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s	S9 n = 55	Min	Max	$\overline{x}$	$\mathrm{SE} (\overline{x})$	s
						Q2	43.00	100.000	89.600	1.832	16.891	Q2	50.000	100.000	89.800	2.238	16.597
						Q4	0.00	100.000	86.306	2.222	20.487	Q4	20.000	100.000	90.327	2.576	19.107
						Q6	6.00	100.000	85.894	2.365	21.805	Q6	45.000	100.000	90.055	2.360	17.501
						Q8	17.00	100.000	88.000	2.185	20.148	Q8	50.000	100.000	92.182	1.991	14.765
						Q10	20.00	100.000	91.082	1.806	16.647	Q10	0.000	100.000	86.055	3.265	24.217
						Q14	10.00	100.000	86.624	2.357	21.731	Q14	10.000	100.000	89.782	2.418	17.935
						Q16	0.00	100.000	87.718	2.184	20.134	Q16	40.000	100.000	89.055	2.200	16.312
						Q18	0.00	100.000	83.494	2.567	23.667	Q18	10.000	100.000	87.600	2.730	20.244
						Q20	45.00	100.000	93.647	1.444	13.315	Q20	45.000	100.000	95.364	1.565	11.607

show comparable results over the three groups of respondents in terms of frequencies and averages. Only minor differences can be noticed for most frequencies presented in Table A4, with the distinction of frequencies in scenarios 1, 5, and 9, where more variability in frequencies between studies exists. While that variability may be the reason for the somewhat lower reliability statistics for these scenarios, the average certainties do not follow the same pattern. Moreover, an insight into Table A7 reveals similar averages, and the differences between the averages are much smaller than their respective standard deviations. Observing all the indicators combined, it can be concluded that the last version of the questionnaire sufficiently satisfies the reliability requirement.

3.2. Validity

The discriminant construct validity is examined with k-nn analysis. Given the reliability insights, only the results for the 2021 study will be presented.

The k-nn analysis is conducted by setting I7 as a target variable, partitioning the sample to the training (60%) and testing (40%) parts, with medians of the nearest neighbor values used for the prediction of the scale target and Euclidian metric for the distances between the neighbors. By setting the I7 as a target, and other items as predictors, this analysis can also serve as a criterion validity estimation.

The sums of squares errors vary between scenarios, but can still be considered low (Figure 1). The k selection error logs for the 2021 LR, LR CP, and SR CP coded data are presented in Figure A1, Figure A2 and Figure A3, respectively. The k selection error logs for scenarios S1–S9 with LR CP and SR CP coded data are shown in Figure A4 and Figure A5, respectively. The sums of squares errors are higher for entire datasets, but are still acceptably low considering the number of observations.

The number of nearest neighbors ranges from 2 (S6) to 10 (S4). It can be interpreted that in scenarios with a lower number of nearest neighbors, more individual differences between the respondents come to the fore. While lower k values may lead to model over-fitting, the selection criteria used in this case is a minimal error, that is, the minimum of the sum of squares error.

The k-nn analysis also allows for the examination of predictor importance. It can be noticed (

Table 5. Predictor importance for target I7, for scenarios S1–S9, and for the 2021 LR coded data.

	S1	S2	S3	S4	S5	S6	S7	S8	S9	2021 LR
I1	0.15	0.12	0.13	0.13	0.12	0.1	0.13	0.15	0.13	0.12
I2	0.12	0.13	0.12	0.12	0.11	0.11	0.14	0.15	0.12	0.12
I3	0.11	0.12	0.13	0.12	0.13	0.1	0.11	0.15	0.13	0.13
I4	0.12	0.11	0.13	0.13	0.12	0.12	0.14	0.1	0.12	0.12
I5	0.13	0.14	0.13	0.13	0.12	0.15	0.11	0.1	0.13	0.13
I6	0.12	0.14	0.13	0.12	0.14	0.15	0.12	0.1	0.13	0.12
I8	0.12	0.13	0.12	0.13	0.12	0.16	0.13	0.12	0.11	0.13
I9	0.12	0.12	0.12	0.12	0.14	0.1	0.12	0.14	0.12	0.13

) that predictor importance is almost uniformly distributed for each scenario. That means that all items I1–I6 and I8–I9 are relevant predictors for the target I7. The results for the scenarios in the 2021 LR CP and SR CP coded data are presented in

Table A8. Predictor importance for target I7, for scenarios S1–S9 and the 2021 LR CP coded data.

	S1	S2	S3	S4	S5	S6	S7	S8	S9	2021 LR CP
I1	0.12	0.13	0.12	0.13	0.12	0.14	0.12	0.15	0.13	0.12
I2	0.12	0.12	0.12	0.12	0.13	0.1	0.13	0.14	0.12	0.13
I3	0.13	0.14	0.13	0.12	0.11	0.14	0.12	0.14	0.13	0.12
I4	0.13	0.12	0.11	0.13	0.11	0.15	0.13	0.09	0.13	0.13
I5	0.13	0.12	0.13	0.13	0.13	0.16	0.12	0.14	0.12	0.12
I6	0.13	0.12	0.13	0.13	0.13	0.08	0.12	0.11	0.12	0.13
I8	0.13	0.12	0.12	0.13	0.13	0.09	0.13	0.12	0.13	0.13
I9	0.12	0.13	0.13	0.12	0.13	0.14	0.12	0.13	0.13	0.13

and

Table A9. Predictor importance for target I7, for scenarios S1–S9 and the 2021 SR CP coded data.

	S1	S2	S3	S4	S5	S6	S7	S8	S9	2021 SR CP
I1	0.13	0.14	0.15	0.12	0.12	0.13	0.12	0.12	0.13	0.12
I2	0.13	0.12	0.14	0.12	0.12	0.12	0.12	0.14	0.13	0.13
I3	0.12	0.13	0.11	0.13	0.12	0.12	0.11	0.13	0.13	0.12
I4	0.12	0.12	0.12	0.12	0.13	0.14	0.12	0.13	0.14	0.12
I5	0.12	0.13	0.12	0.13	0.14	0.11	0.12	0.13	0.13	0.12
I6	0.13	0.11	0.13	0.12	0.14	0.12	0.13	0.12	0.1	0.13
I8	0.13	0.13	0.12	0.13	0.13	0.12	0.14	0.11	0.11	0.13
I9	0.13	0.13	0.12	0.13	0.13	0.15	0.14	0.12	0.12	0.12

, respectively. While some minor differences may be observed when comparing Table 5, Table A8 and Table A9, the general conclusion about items’ importance is sustained.

While the additional analysis was conducted after removing the least important predictors, it led to higher sum of squares errors. It can be concluded that none of the predictor variables (I1–I9) are discriminated from further analysis. Furthermore, given the small sum of squares errors, it can be deduced that the scenarios S1–S9 satisfy the criterion validity.

Further analysis requires an examination of convergent validity. For the entire 2021 LR coded dataset, Kaiser–Meyer–Olkin’s measure of sampling adequacy equals 0.887, and Bartlett’s test of sphericity is significant (${\chi }^2=1634.35$). Similarly, for the 2021 LR CP coded data, the KMO measure is 0.879, and Bartlett’s test of sphericity indicates that the data is suitable for further analysis (${\chi }^2=1507.22$). For 2021 SR CP coded data, the KMO measure is 0.866, and Bartlett’s test of sphericity is significant (${\chi }^2=1283.72$). There is only one component with an eigenvalue greater than or equal to one for the data from the 2021 study coded each way. The resulting component matrices are presented in

Table 6. Component matrices for the 2021 data.

	Component (LR)	Component (LR CP)	Component (SR CP)
I1	0.801	0.795	0.793
I3	0.773	0.78	0.777
I4	0.758	0.736	0.654
I2	0.736	0.705	0.66
I5	0.725	0.719	0.72
I7	0.714	0.688	0.684
I6	0.634	0.612	0.544
I9	0.496	0.466	0.478
I8	0.480	0.459	0.443

. The singular components for data coded each way have almost the same order of loadings (for LR CP, the ranks of the loadings for I2 and I5 switch places, and for SR CP the ranking is: I1, I3, I5, I7, I2, I4, I6, I8, I9).

The same approach is taken for each of the scenarios in LR, LR CP, and SR CP coded data from the 2021 study, and the results are presented in

Table 7. Component matrices for scenarios S1–S9 from the 2021 LR coded dataset.

	S1			S2	S3	S4	S5	S6	S7	S8	S9
	1	2	3	1	1	1	1	1	1	1	1
I1	0.684			0.865	0.850	0.761	0.744	0.811	0.826	0.887	0.809
I2	0.712			0.763	0.759	0.681	0.799	0.867	0.686	0.747	0.590
I3	0.761			0.829	0.858	0.766	0.843	0.804	0.756	0.743	0.835
I4	0.706			0.827	0.837	0.776	0.774	0.724	0.811	0.678	0.766
I5	0.736			0.662	0.693	0.751	0.640	0.849	0.673	0.757	0.614
I6	0.612	0.57		0.539	0.709	0.667	0.450	0.722	0.651	omitted	omitted
I7		0.45	0.61	0.704	0.668	0.748	0.671	0.733	0.706	0.786	0.838
I8	0.505		0.65	0.624	0.588	0.490	omitted	omitted	0.587	0.721	omitted
I9		0.72		omitted	omitted	omitted	omitted	0.722	0.686	0.575	omitted
variance explained	66.48%			53.94%	56.41%	50.52%	50.93%	61.00%	50.82%	54.98%	56.07%
KMO	0.696			0.835	0.848	0.801	0.807	0.845	0.846	0.831	0.802
Bartlett’s test	132.32 (p < 0.001)			212.3 (p < 0.001)	226.12 (p < 0.001)	193.36 (p < 0.001)	135.32 (p < 0.001)	244.3 (p < 0.001)	212.75 (p < 0.001)	215.81 (p < 0.001)	135.64 (p < 0.001)

,

Table A10. Component matrices for scenarios S1–S9 from the 2021 LR CP coded dataset.

	S1		S2	S3	S4	S5	S6	S7	S8	S9
	1	2	1	1	1	1	1	1	1	1
I1	0.758		0.816	0.843	0.776	0.713	0.797	0.825	0.878	0.808
I2	0.738		0.772	0.664	0.664	0.808	0.843	0.626	0.716	0.578
I3	0.785		0.831	0.866	0.789	0.866	0.787	0.755	0.763	0.860
I4	0.7		0.835	0.801	0.786	0.792	0.705	0.768	0.600	0.768
I5	0.693		0.706	0.670	0.732	0.679	0.811	0.667	0.747	0.604
I6	omitted		0.461	0.709	0.651	omitted	0.611	0.645	omitted	omitted
I7		0.752	0.657	0.659	0.718	0.682	0.691	0.699	0.728	0.840
I8	omitted		0.597	0.626	omitted	omitted	omitted	0.574	0.746	omitted
I9		0.562	omitted	omitted	omitted	omitted	0.703	0.701	0.519	omitted
variance explained	57.68%		51.87%	54%	53.73%	57.76%	55.81%	48.93%	51.74%	56.43%
KMO	0.629		0.815	0.85	0.783	0.844	0.804	0.819	0.836	0.792
Bartlett’s test	93.38 (p < 0.001)		200.37 (p < 0.001)	206.2 (p < 0.001)	176.4 (p < 0.001)	129.66 (p < 0.001)	219.86 (p < 0.001)	201.69 (p < 0.001)	187.97 (p < 0.001)	146.26 (p < 0.001)

and

Table A11. Component matrices for scenarios S1–S9 from the 2021 SR CP coded dataset.

	S1			S2	S3	S4	S5	S6	S7	S8	S9
	1	2	3	1	1	1	1	1	1	1	1
I1	0.691			0.823	0.818	0.787	0.713	0.794	0.823	0.85	0.819
I2		0.813		0.777	0.57	0.665	0.808	0.793	0.615	0.667	0.584
I3	0.771			0.853	0.881	0.783	0.866	0.791	0.772	0.7	0.827
I4	0.666			0.802	0.806	0.796	0.792	0.724	0.722	omitted *	0.692
I5	0.728	−0.43		0.731	0.656	0.731	0.679	0.8	0.664	0.77	omitted *
I6	0.775			omitted *	0.537	0.612	omitted *	0.531	0.655	0.46 *	0.592 *
I7			0.87	0.65	0.66	0.705	0.682	0.685	0.692	0.749	0.864
I8	omitted			0.628	0.606	omitted *	omitted	omitted	0.591	0.769	omitted
I9	omitted			omitted	omitted	omitted	omitted	0.688	0.697	0.536	omitted
variance explained	71.77%			57.157%	49.25%	53.06%	57.76%	53.45%	48.38%	48.76%	54.49%
KMO	0.716			0.832	0.835	0.779	0.844	0.832	0.75	0.808	0.773
Bartlett’s test	74.823 (p < 0.001)			183.84 (p < 0.001)	178.41 (p < 0.001)	172.42 (p < 0.001)	129.66 (p < 0.001)	189.91 (p < 0.001)	209.03 (p < 0.001)	162.62 (p < 0.001)	132.13 (p < 0.001)

Note: * denotes differences in comparison to LR coded dataset.

, respectively.

The evidence presented in Table 7 shows further support for a decision about S1 exclusion. First, the KMO measure of sampling adequacy is relatively lower in comparison to other scenarios, and this is the only scenario that contains multiple components and cross-loadings, even after an item’s omission (as presented in Table A10 and Table A11), meaning that it measures other concepts in addition to game awareness, and it does not measure game awareness as it was intended. Moreover, I7 appears in the second and third components, showing that the scenario does not allow conclusions about game awareness in relation to game elements.

All other scenarios, except for S7, revealed two components when all items were included. Considering that the goal is to determine whether the scenarios measure game awareness, this exploratory analysis also examines which items do not measure game awareness, aiming for the singular component where the loadings construct game awareness. In most cases, I9 appeared to load the second component, and its exclusion led to unidimensionality. Moreover, I9 refers to the awareness about one’s own possible actions, which requires more intense logical deliberation about possibilities and even in-game reasoning. Hence, it makes sense that it appears in a separate component. The removal of I9 resulted in one component for scenarios 2, 3, and 4. Another problematic item, I8 (deliberation awareness), led to the second component, and its removal prompted a one-component output for scenarios S5 and S6. In comparison, when omitted, these two variables improved reliability statistics (Table 4), too. However, they seem to contribute to the understanding of game awareness in scenarios S7 and S8.

Unusual and somewhat unexpected was the role of I6 (own outcome assessment) in scenarios S8 and S9. However, those are the only two scenarios that offered complete information that enables not guessing but the reading-out of possible outcomes. It is possible that the complete information and the lack of uncertainty led to such variable values and inter-variable correlations that made I6 appear in the second component. Given that interpretation, it may be that the questionnaire better measures game awareness under uncertainty and incomplete information or in a pre-game context, that is, in situations similar to the ones that occur in the real world.

4. Discussion and Conclusions

The goal of this paper was to offer a methodological contribution to the empirical examination of individuals’ game awareness. The proposed questionnaire has been examined regarding its internal reliability, criterion validity, and construct validity. The questionnaire comprises different scenarios that vary regarding provided information, activity type, outcomes, and diverse perspectives. Those variations should enable the capture of how respondents form a game awareness (or a lack thereof), given different amounts and qualities of the provided information. Such an approach allows for empirical insights into the individual’s game awareness formation, which may be used in the future to enrich theoretical comprehension. In addition, while the questionnaire may be used as a whole, there are indications that scenarios may also be used separately to examine the desired aspect.

4.1. Reliability and Validity

The examined reliability per coding type, study, scenarios, and for different combinations of items revealed that the 2021 LR coded data offer the highest reliability measured with Cronbach’s alpha. While this method is not designed to assess the coding approach, infringed reliability sheds light on the consequences for data and the possibility of its use. Undoubtedly, modifications in questionnaire versions contribute most to the higher reliability revealed in the 2021 study data. Given the reliability findings, the validity examination was conducted only for the 2021 study. A k-nn analysis was used to estimate divergent construct validity and criterion validity (by setting I7 as a target), and nonparametric principal component analysis based on the Spearman correlation coefficients matrix was used to estimate convergent construct validity. Reliability and validity analyses point out that the last version of the questionnaire is both reliable and valid. However, the entire questionnaire is lengthy and may be boresome [24] for respondents, and this was the motivation to examine reliability and validity for each scenario. The more profound insights into the scenarios revealed that S1 should be excluded from the questionnaire. The other scenarios show indications of internal reliability, as well as criterion and construct validity requirements. Questions Q19 and Q20 can be omitted from scenarios 2, 3, 4, 5 and 9, or interpreted with caution. Questions Q17 and Q18 can be omitted or interpreted at one’s discretion in scenarios 5, 6, and 9. The same approach is viable for Q13 and Q14 in scenarios 8 and 9 (and 5, for CP coding). While there are differences in results given the different approaches to the coding, overall results’ similarities indicate the questionnaire’s relative robustness. Scenarios S2–S9 reach over the reliability and validity threshold, which may serve as an indication for their separate use.

Moreover, the principal component analysis results in one component for scenarios S2–S9. One component indicates that the included questions measure the same concept and that there are no other underlying latent variables. As a distinction has been made between awareness and assessment, and since separate questions examined them, these results indicate no fundamentally different underlying process that distinguishes the two. This also confirms the approaches that include elements of the assessment in the awareness’ definitions and approaches [5,6,12,14,16], that is, the awareness in a narrow sense seems to be a precondition for the assessment, while assessment is required for the player to obtain an amount of game awareness, but they co-occur and act as building blocks for shaping one’s awareness of a situation.

4.2. Limitations

In this paper, the effort was focused on shedding light on various aspects of the questionnaire, from its topic to its applicability. Nevertheless, there are still questions that remain unanswered. The questionnaire is examined from the perspective of internal consistency reliability, leaving equivalence and stability unchecked. The equivalence was not examined, as no other questionnaire of game awareness is known to the author, so none was applied. Perhaps this questionnaire can serve as a motivation for the further development of research instruments for the behavioral perspective on game awareness, which would enable equivalence assessments in the future.

While descriptive statistics from the three studies were used as a proxy for stability, that cannot be understood as evidence of stability. It is only an indication and is derived based on similarities of corresponding data (not all questions and not all scenarios are present in each study). Furthermore, content validity is suggested only by theoretical basis for questionnaire and scenarios development, as well as by subjective assessment of the researcher (face validity), while no scrupulous appraisal by different experts has been done.

Although a decent amount of data has been gathered overall and validity and reliability have been confirmed at the questionnaire level, there were only 498 observations for the 2021 study spanning nine scenarios, so additional reliability and validity confirmations for each scenario are advised. Moreover, one must keep in mind that reliability and validity refer to the questionnaire written in the original language (Croatian) and not the translated English version. In addition to the stated limitations, the limitations regarding the scenarios’ contents and further enrichment by introducing different scenarios remain for future research.

Considering the coding approaches, the LR approach may seem somewhat subjective, which might influence the results’ replicability. However, a comparison with the results obtained by the SR approach demonstrates a high similarity and indicates the questionnaire’s robustness regarding the coding approaches. Furthermore, the comparison of the convergent validity results suggests that both approaches capture the essence of the phenomenon of interest.

In addition, one might favor the standard game approach and might want to approach the data by measuring deviation. The last two scenarios would be the best choice for proceeding in such an endeavor. This would also require a recoding, such that ‘correct’ responses are denoted from the information offered in scenarios as a baseline (for example, coded 0, which would stand for no deviation) while denoting the other responses as deviations. While that was not the intended purpose of the questionnaire, that option remains a possibility for future instrument use and as a line of development.

Lastly, the provided incentive may be considered weak or partial by some standards, which may influence motivation for providing responses and mistakes in responses. However, the answers to the open-ended questions were considered a proxy for the engagement, and uniform or unusual responses were removed from the further analysis. That is one of the reasons for the lower responses’ retention percentages. Nevertheless, the examination can be repeated in the future with a stronger incentive.

4.3. Implication for Further Research

The questionnaire allows for a quantitative and qualitative (open-ended questions) examination of responses. Open-ended questions provide a deeper insight into awareness and assessment that cannot be revealed by yes/no questions [25]. In addition, the responses may be used for future questionnaire development by forming the statements about scenarios, and thus ease the completion for respondents. However, a possible problem with such an approach is that the respondents may become aware of possible interpretations because they are offered in the statements [25], whereas the current questionnaire form does not enable such potential deviation. Nevertheless, that remains as an option that can be further examined.

Another instrument development direction may be converting the probability assessment into a slider bar or a Likert-type scale based on the possible groupings or by centering percentages around specific values, thus making the instrument more attractive to fill in for the respondents who have a hard time dealing with percentages.

A specific area of possible future in-depth instrument development regards the examination of game element awareness separately. For example, a separate questionnaire could examine what, exactly, is relevant for a person to detect and assess an activity, what properties, what amount of information, and of what quality should be present, etc.

The (un)awareness in game theory is relatively new and mainly dealt with using a logical approach or a standard game representation. The most notable contributions to behavioral aspects of awareness (in general, but also in games) stem from the research of business and ethical questions [5,6,18,29]. Although the contributions are worthwhile, potential consequences of unawareness spread over various disciplines where individuals play a role as a decision maker and engage in strategic interactions. This questionnaire is focused on game awareness in different situations of possible strategic interactions, and hopefully, it will encourage other instruments’ development, as well as the collection and publication of behavioral insights into game awareness. One should notice the difference and the connection between the psychological insights and behavioral game theory: the first one serves as an input for broadening knowledge about human thinking and behavior, and the latter utilizes that knowledge in model creation. Models—including behavioral game theory models—inevitably contain a level of abstraction, idealization, and caricaturing, but with the growing body of theoretical and empirical knowledge, accompanied by the increasing variety and power of quantitative approaches, should strive for more realistic modeling, thus becoming more useful for theoretical and practical advancements. In other words, empirical data enable data-driven conclusions, and in this case, an inductive approach to the role of (un)awareness in games, therefore, complements the existing deductive approach. One possible approach is to use the findings from the data as building blocks for modeling game behavior (instead of modeling behavior as a deviation from the implicitly assumed full awareness and rationality), as has been done for other individuals’ traits, for example, by using personality traits and general attitudes in different situations of bargaining [57,58,59]. The underlying idea is that the behavioral insights should endow the theory and prompt behavioral game theory models of game awareness and its implications. This could also lead to the requirement of revising the current approach to the in-game bounded rationality—a part of the deviations currently ascribed to the bounded rationality might actually stem from unawareness.

4.4. Practical Implications

A certain amount of (un)awareness is present in most situations for everyone. In some cases, consequences may refer to an individual’s life; for example, unawareness of an alternative or its attributes may skew the choices of individuals toward less favorable outcomes. In individuals’ strategic interactions, such occurrences may influence personal and business relationships, careers, personal finance management, health strategies, savings strategies, voting strategies, etc. However, the consequences could multiply if unawareness affects an individual in a position of power, whether it is a managerial position in a company or a political position. Becoming aware of unawareness is not the only requirement, but is a critical one for better decision making and better modeling of actual human behavior. This notion tackles the area of reflective awareness [29], where it was suggested that it can be trained and improved. Accordingly, this questionnaire can be used in the classroom or for professional skill training to assess the individuals’ game awareness and use it as a basis for intervention in the case of revealed fallacies.