Nowadays, managers face challenging conditions, dynamic environments, and complex processes [1
]. Their decision skills affect the success of the companies [3
] and determines their economic results [5
]. Although human decision-making is described by a number of theories, it still remains a complex process that is difficult to research [6
]. Moreover, to perform such research in the real world, where a significant sample of participants face identical conditions and situations, is almost impossible. Modern technologies, however, bring the possibility to use computer simulations and games for the decision-making research [1
]. Games provide an adequate and safe space for experimentation [13
]. Their use for the research of decision-making is based on the idea, that respondents reflect their knowledge, experience, and skills during play [11
]. Nevertheless, the research of decision-making using games is still a little researched area. In addition, it requires adequate methodological procedures as different methods reveal different aspects of decision-making. Thus a description of their pros and cons will enable their better utilization [6
]. In the field of decision-making research, a number of methods have been generally applied as questionnaires, observation, interviews, eye-tracking, think-aloud, decision analysis, etc. [6
A few studies, which included the use of games, agree in particular on the use of eye-tracking [10
]. It is the method of recording eye movements by a special apparatus (eye-tracker) [21
]. This technique is commonly used in a human-computer interaction studies [21
] and finds application in many areas of research [23
]. The use of eye-tracking for research purposes is based on the assumption that humans perceive and cognitively work with what they see [24
]. Vision is the most important sense in terms of acquiring information [25
]. People perceive information by the sensory system [27
] and then they integrate and compare it with expectations and knowledge, which results in a behavioral response [28
]. Therefore eye-tracking is considered a valuable tool to study the cognitive processes that accompany various human mental activities, ranging from less demanding ones such as reading [26
], writing [31
], and perception of images and objects [34
], up to the more complex ones such as learning [39
], and decision-making [10
]. In the case of decision-making it was proved that the method significantly contributes to the accumulation of evidence about this process [28
] and improves its understanding [49
]. However, there are also studies mentioning relevant shortcomings of the use of eye-tracking in decision-making research [21
]. Eye-tracking cannot reveal by far all aspects of the decision-making [50
], because it is not entirely clear if humans comprehend the information they watch [51
], and whether watched information is incorporated into the decision-making process [52
Therefore, the combination of eye-tracking with other methods is necessary [21
]. Especially a combined use of thinking-aloud and eye-tracking data can provide deeper insights into cognitive processes [53
] and help to solve the limitations associated with eye-tracking [54
]. Think-aloud (often also referred to as verbal protocols) is a methodology for studying behavioral and cognitive processes while people solve problems [55
]. However, there are two main types (methods) of think-aloud: (1) Concurrent think-aloud (CTA), where subjects are asked to do the tasks and verbalize thoughts simultaneously; and (2) Retrospective think-aloud (RTA) where subjects are asked to do the task silently first and then retrospectively report on the solving process [56
]. Over time, both methods have penetrated many fields of science, which gives rise to constant discussions of their advantages and disadvantages, as well as their applicability and validity for various research fields [53
]. The related problem of CTA is the dual cognitive load when the thinking process and the verbalization process compete with each other [63
]. Thus, the cognitive workload of respondents can be too high [62
] and it can impact on their standard working process [56
]. Therefore, some ideas may be lost as they cannot be expressed in real-time [63
]. The data omissions occur especially when the presented information is difficult to verbalize or the processes are automatic for participants [55
]. RTA use does not disturb participants during the task [56
]. Nevertheless, it has also limitations like forgetting information or omitting its interpretation [55
] and post-rationalization or fabrication of thoughts [68
]. Therefore, many scientists overcome the RTA shortcomings by re-playing recordings to participants [53
Interestingly, many authors, who deal with think-aloud, also recommend combining them with eye-tracking [47
]. The involvement of eye-tracking can help solve the shortcomings of think-aloud because it allows one to confirm and extend the gathered data [47
] or to complete omitted and forgotten points [54
On the other hand, these studies take little or no account of the possible impact of thinking aloud on eye-tracking data. This impact is expected specifically in the case of CTA [67
], as this method may bring the unnatural behavior of humans during task solving [72
]. Although these statements can be considered as logical, it has not been empirically verified to what extent this may be the case.
Therefore, the main aim of this study, which is focused on decision-making research in the game environment, is to compare and assess if and how CTA and RTA methods differ in terms of their impact on data provided by eye-tracking. In addition, with the respect that the game is a process, where a player approaches can evolve, the research is not only concerned with the overall final assessment and comparison of both methods but also examines whether the potential differences gradually change during the game progress or not. Based on the findings it will be possible to discuss which of the think-aloud methods in combination with eye-tracking is more suitable for decision-making process research in the game environment.
Existing studies have provided much evidence of differences between CTA and RTA. One of the frequently discussed topics is the quality and quantity of data provided by these methods [55
]. Scientists, who have described deficiencies of verbal protocols, recommend their combination with eye-tracking, which helps to complete missing data and insights [47
]. However, they often do not consider the possible impact of think-aloud on eye movements. In this study, it was empirically verified that there is a significant difference in data provided by eye-tracking when using CTA or RTA.
4.1. Overall Comparison of the Impact of CTA and RTA on Eye-Tracking
The total fixation duration is significantly higher when using CTA than when the task is carried out in silence (RTA). Longer fixation duration can indicate two types of cognitive processes: (1) an object is interesting to a participant or (2) the cognitive processing of data is more difficult [21
]. In the case of this study, the 1st assumption can be rejected. Both groups of respondents faced the same experimental environment, the same game’s design with the same layout and data structure on the screen. There is no indication that the CTA players considered some parts of the game to be more attractive.
The 2nd assumption is much more likely. The higher total fixation duration of CTA is mainly due to their higher number because the average length of fixation is conversely a little higher for the RTA than for CTA. A higher number of fixations then also indicate a higher cognitive effort [79
]. It seems that revealed higher cognitive processing is evidence of the presence of the CTA’s dual cognitive load, described by Ericsson and Simon [56
]. The cognitive effort is divided among the process of task solving and the process of verbalization. Therefore, it can be considered that the number of fixations can also be divided between these processes. In other words, the part of the CTA’s fixations arises during the decision-making process itself when the player concentrates on the task, and part of the fixations arises during the verbalization when the player concentrates on speaking. The reason for this assertion is based on fact, that CTA and RTA players achieved the same game results (Table 2
and Table 3
), which suggests their similar decision-making skills. Thus, it cannot be assumed that the task itself was more demanding for one or another group. Therefore, we assume, that the decision-making process itself needed the same cognitive effort, and thus also a similar number of fixations of CTA’s and RTA’s players.
The task solving process during the selling phase can be divided into several sub-processes with different demands on cognitive processing. The extent to which the fixations are divided between two cognitive processes can then be related to the complexity of these sub-processes and may vary among them. Verbalizing simple tasks carries a lower risk of the dual load than more demanding tasks [56
]. Studies that investigate CTA validation by eye-tracking [65
] suggest that for simple tasks such as reading or describing procedures, fixations take place simultaneously with verbalizations. Thus, it can be assumed that when a player reads data from the screen or performs simple tasks (mouse-clicking, decision writing), fixations may correlate to verbalizations. Contrarily, Elling et al. [64
] argue that verbalization may not correlate with fixation in many cases of more demanding tasks when many verbalized thoughts cannot be associated with fixations at all. Therefore, the distribution of fixations between two cognitive processes may occur especially in cases of more complex tasks (e.g., decision-making), when participants verbalize some cognitively more demanding thoughts. In such cases, the participants performing complex tasks can also pause their speech [64
], but this does not necessarily mean that task circumstances would not be verbalized at all. We noticed many short moments where CTA players themselves redressed the pauses. They added comments immediately after the actions during which they were silent for a few seconds. This usually happened when they needed more cognitive capacity as they thought deeply about something. This suggests that in these short periods of increased cognitive stress, players can naturally switch from CTA to RTA, by commenting on the actions, they have just completed. As a result, the number of fixations increases during these “delayed verbalizations”. The participants either observed again cues needed for the already done decision—“repeated fixations” happen, or watched some part of the screen, without a higher cognitive perception of what is observed, because they were focused only on the verbalization of previous steps—“purposeless fixations” happen. The evidence of the presence of purposeless fixations while CTA use is also provided by the Dwell time ratio, where CTA participants did more fixations outside the AOIs important for the decision.
Nevertheless, the presence and more detailed analysis of repeated and purposeless fixations should be subject to future research, as the abovementioned evidence in this study is not quite direct and unambiguous. In fact, such situations can occur in very short time periods and they may quickly alternate or partially overlap. Therefore, it may not be easy to separate the eye-tracking data accompanying individual cognitive processes when CTA is used. In addition, the negative impact on eye-tracking may not be caused by only dual cognitive load. The unnatural physical behavior of CTA participants can also contribute to the increased number of fixations as speaking may influence the head movements and thus also the eyes [67
]. CTA players in the present study achieved a lower average fixation duration, but the number of fixations was much higher. This could indicate that keeping a stable gaze at an exact point of the screen was more difficult for them and a higher number of shorter fixations can occur in a certain area of this point.
4.2. Comparison of the Impact of CTA and RTA on Eye-Tracking over Time (in the Game Progress)
Eye-tracking data subdivided into individual game rounds offer another view on the task-solving process as well as on the verbalization process. Eye-tracking is widely used also in the education field, where several studies evidenced that there are significant differences in eye movements between experienced and less experienced participants [10
]. The development of number of fixations and total fixation duration in the present study has a decreasing trend over time, which applies to the use of both think-aloud methods. As fixations indicates cognitive load [24
], it can be assumed, that their decreasing number and total duration relates to increasing experience. Players’ performance during the game increases with experience gathered in each round. During the first rounds, players think about decisions more and therefore they paid more attention to various objects on the screen. When players’ orientation in the game and decision knowledge are getting better, then they need less time to grasp data from the screen and they are more focused only on cues necessary for the decision. Therefore, a decreasing number and total duration of fixations are evidence of the learning process within the game. The original complex and cognitively demanding processes are becoming more and more simple depending on the number of repetitions of the situation they are associated with. This is in line with the purpose of the used simulation FactOrEasy®
, which is not only reflects the knowledge and experience of the participants [11
] but also serves as a learning tool [73
Nevertheless, the comparison of RTA and CTA curves of number of fixations (Figure 2
) and total fixation duration (Figure 4
) suggest that differences among the methods’ impact on eye-tracking metrics are also developed during the game progress. Higher values on the CTA curves are caused by the fact that CTA fixations are affected by both the task-solving process and the verbalization process, while the RTA curves contain the only fixations connected with task-solving. It means that eye-tracking metrics recorded in silence (when RTA is used) provide independent evidence about player’s behavior in the simulation game.
RTA’s logarithmic trends of the mentioned metrics confirm this conclusion as they correspond with the usual course of FactOrEasy®. In the first rounds, players try to better understand the circumstances of each decision and they set an overall strategy. This needs more cognitive effort, which is accompanied by higher values of monitored metrics. In the following rounds, players try to follow their strategies and they are more focused on individual decisions that are becoming more and more routine. This is accompanied by the lower values of monitored metrics.
In the case of CTA, it is too difficult to separate fixations associated with a task-solving process from fixations associated with verbalizations. Therefore, eye-tracking in combination with the CTA cannot be considered as accurate, for the purpose of monitoring player’s progress in the game. On the other hand, the results suggest that both cognitive processes are also undergoing certain development during the game. CTA brings higher values of the number of fixations and total fixation duration during the opening rounds than RTA. Nevertheless, the CTA values quickly decrease in the linear trend and they are getting closer to those of RTA in the latter rounds. This development allows to confirm or further extend several assumptions about CTA mentioned in the paper:
The CTA can make participants feel unnatural and confused. It may take some time while they get used to the task, especially at the beginning of the experiment.
Both cognitive processes become less and less demanding depending on the experience gained. As well as players repeat the process of task-solving in each round, they also repeat verbal comments on what they do. At the beginning of the game, participants must create verbal expressions, name objects and thoughts, and interpret logical connections among them. This process is optimized during the game, comments are shortened, and the verbalization becomes more natural.
When the task-solving process is becoming easier (the original complex and cognitively demanding processes of decision-making are getting simple), participants can be better focused on verbalizations. The verbalizations of simpler tasks are more natural and therefore, they start to correlate more frequently simultaneously with eye movements.
The abovementioned findings are also supported by the results of dwell time ratio. The CTA’s trend has been increasing over the rounds and has been reaching the same values as in the RTA case in recent rounds. This indicates that CTA players are increasingly focusing only on substantial AOIs and the number of purposeless fixations is reduced over the game.
4.3. Comparison of the Data Gathered by CTA and RTA
The latest analysis of our study aimed to compare the data gathered by CTA and RTA. It was examined whether participants really verbalized data of the game screen, which they considered during the decision-making. In this regard, we found that RTA players omitted significantly more information (27.2%) than CTA players (10.4%). Thus, we can confirm claims of a number of scholars [55
], that RTA is associated with a higher risk of forgetting information. Therefore, if researchers considering the use of RTA want to have sufficient evidence of which screen information is considered by participants during the decision-making process, then they always should combine this method with eye-tracking. In the case of CTA, the risk of losing such evidence is lower because it is more natural for participants to verbalize the information, which they can read from the screen while solving the task [64
However, a choice of the appropriate verbal protocol may depend on the specific aspects of the decision-making which are an object of the intended research. We are aware, that there are other points of view such as references about knowledge, inner conviction, justifications of thoughts, used strategies, etc. These aspects are not empirically assessed in the present study, which is one of the main paper’s limitations. Nevertheless, several previous studies proved that explanations of these aspects are better provided by the RTA method while CTA provides mainly comments on basic actions (reading, writing) and their outcomes [61
]. Even though an accurate qualitative analysis of CTA and RTA transcripts by text coding has not been performed, the verbalizations collected during our study appear to suggest similar findings. CTA players verbalized more screen information because they read them loudly. However, they provided subsequently fewer explanations of how they cognitively worked with them. Given that both groups have achieved similar game results, this does not necessarily mean that CTA players thought about these cues less than RTA players. A much more likely explanation is that it was difficult for them to express all thoughts when they were concurrently focused on playing the game. This is consistent with previous findings that claim that some ideas may be lost during CTA use because they cannot be expressed in real-time [82
]; and that cognitive processes are quicker than verbal processes, which means that people can think more about something then they are able to concurrently express [83
In view of the abovementioned findings, we can afford to make the recommendation that a combination of RTA with eye-tracking is a more appropriate way to study the decision-making processes in a game environment. Despite the fact RTA results in more omissions of what screen information was considered during decision-making, eye-tracking can easily add this evidence. On the other hand, CTA, unlike RTA, provides less evidence of how this information is cognitively processed, which is a shortcoming that cannot be solved by adding eye-tracking.
4.4. Limitations of the Study
The results of this study and their validity are limited by several factors, which indicate some other possibilities for further research. First, the experiment was conducted in the specific simulation game FactOrEasy®. Applying think-aloud methods in different games can produce different results of eye-tracking metrics. These results may depend on game designs and on the level of the difficulty of the tasks, which participants face. Secondly, the present study deals mainly with differences in eye-tracking metrics when combining this method with verbal protocols. Therefore, it provides only a basic assessment of data obtained by verbal protocols. Accurate examination of CTA and RTA players’ transcripts by coding of the text was not performed. However, we assume that the scope of such an analysis is beyond the aims of the present study. For instance, it is very likely that such an analysis will require the development of a specific encoding method, which would be suitable for the game environment and research purposes. Therefore, we consider this topic as an appropriate topic to create a further separate study. Thirdly, only the general use of the mentioned methods for the given purpose is described. We do not provide any conclusions about the specific decision-making processes of participants, any assessment of their decision-making skills, etc. The choice of the appropriate verbal protocol, its combination with eye-tracking, and the choice of right metrics may depend on the specific aims, which will be objects of the intended research of further studies.
Many authors recommend using a combination of verbal protocols and eye-tracking to investigate cognitive processes [47
] However, so far there have been few references to a possible difference in the use of CTA and RTA in terms of their impact on data gathered by eye-tracking. In this study, we empirically verified that there is a significant difference in data provided by eye-tracking when using CTA or RTA.
Gerjets et al. [71
] state that eye-tracking brings an extension of think-aloud findings by pieces of evidence of fine-grained or implicit cognitive processes. However, the findings in this study suggest that such an extension makes sense only in the case of RTA use. All examined indicators empirically confirm the claims, that RTA has no impact on eye-tracking metrics, while CTA distorts them significantly [67
]. Metrics recorded in silence provide independent evidence about player progress in the game. Thus, our results suggest that RTA is more suitable for combined use with eye-tracking for the purpose of decision-making research in this environment. When using CTA, eye-tracking metrics are affected by dual cognitive load and unnatural physical behavior of participants. A combination of the task-solving process with the verbalization process brings repeated and purposeless fixations, which are redundant and distorting.
However, the development of monitored metrics over time suggests that the problem of the dual cognitive load decrease in the game progress. In later game rounds, the CTA players achieved the same number of fixations as RTA players. In addition, they were also able to concentrate equally well only on important cues. It means that if participants repeat the same or similar task multiple times, the negative effect of the dual load on CTA may decrease. This is in line with studies [56
], which recommend performing a suitable training task before the experimental one and repeating it until participants prove sufficient ability of concurrent verbalization. However, the question remains whether this recommendation is also valid for decision-making research, especially if the participants are dealing with a series of several different decisions (like in game). Frequent repetition of the training task, which is similar to the experimental one, can distort the results of the experiment itself.
Nonetheless, even the comparison of the data gathered by both think-aloud methods suggested that a combination of RTA with eye-tracking should be a more appropriate method for the research of decision-making processes in the game environment. The reason is that CTA provides more evidence about what screen information is considered during decision-making, while RTA verbalizations contain more evidence about how this information is cognitively processed during decision-making. In the case of RTA, the omissions of “what” can be resolved by adding eye-tracking data, whereas, in the case of CTA, eye-tracking cannot help to explain missing “how”.
However, the findings of the present study do not either mean a definitive rejection of the CTA for research purposes in the field of simulations and games. The final choice of method always depends on the aims of the intended research [6
]. The conducted research was focused only on the examination of aspects of the decision-making process in the game environment. There are other fields, like usability testing [60
] or education [68
], where CTA has added value. The gathered results suggest the potential of the application, in particular, in the second-mentioned field. The CTA is often associated with reactivity [56
] that may have a positive impact on the learning process [85
]. Eye-tracking metrics indicate that the learning process in a game environment may have different development depending on whether the RTA or CTA is being performed. Thus, future research could focus on whether thinking-aloud can support the learning process in the environment of simulations and games, and on what benefits CTA and RTA brings for this purpose.