Observation of Human–Robot Interactions at a Science Museum: A Dual-Level Analytical Approach

Abstract

This study proposes a dual-level analytical approach to observing human–robot interactions in a real-world public setting, specifically a science museum. Observation plays a crucial role in human–robot interaction research by enabling the capture of nuanced and context-sensitive behaviors that are often missed by post-interaction surveys or controlled laboratory experiments. Public environments such as museums pose particular challenges due to their dynamic and open-ended nature, requiring methodological approaches that balance ecological validity with analytical rigor. To address these challenges, we introduce a dual-level approach for behavioral observation, integrating statistical analysis across demographic groups with time-series modeling of individual engagement dynamics. At the group level, we analyzed engagement patterns based on age and gender, revealing significantly higher interaction levels among children and adolescents compared to adults. At the individual level, we employed temporal behavioral analysis using a Hidden Markov Model to identify sequential engagement states—low, moderate, and high—derived from time-series behavioral patterns. This approach offers both broad and detailed insights into visitor engagement, providing actionable implications for designing adaptive and socially engaging robot behaviors in complex public environments. Furthermore, it can facilitate the analysis of social robot interactions in everyday contexts and contribute to building a practical foundation for their implementation in real-world settings.

1. Introduction

Service robots are increasingly being deployed in everyday environments such as museums, restaurants, and public facilities, moving beyond the confines of laboratory testing [1,2,3,4]. As these robots become integrated into various aspects of daily life, it has become critically important to understand how users interact with them from a human–robot interaction (HRI) perspective. This is particularly relevant in the service sector, where optimizing user–robot interactions is essential for delivering meaningful and satisfying user experiences [5].

Traditionally, HRI research has relied heavily on survey-based methods to assess user perceptions and preferences [6]. However, surveys often fall short in capturing the subtle, dynamic, and context-dependent behaviors that occur in real-world settings [7]. Designing robots that can effectively respond to the diverse needs of users in complex public environments requires moving beyond self-reported data and adopting more naturalistic investigative methods [8].

Observational methods offer a distinct advantage in HRI research by capturing user behavior. Among various observational techniques, naturalistic observation is particularly valuable for identifying authentic behavioral patterns as they occur in real-life contexts [7,9]. By studying behavior in everyday environments without experimental interference, researchers can uncover subtle interaction dynamics that are often difficult to detect through surveys, controlled experiments, or post hoc interviews [8,10]. Despite these advantages, prior studies in HRI over the past decade have frequently lacked structured analytical frameworks to systematically interpret behavioral data, limiting the generalizability and practical application of their findings [11,12,13].

To address this limitation, the present study introduces a dual-level analytical approach to behavioral observation, which integrates group-level statistical analysis based on demographic factors with individual-level temporal modeling of user engagement. This approach is intended to support a more comprehensive understanding of HRI in real-world public settings and to inform the practical deployment of socially engaging and adaptive service robots.

To introduce the dual-level analytical approach, the remainder of this paper is organized as follows: Section 2 reviews related works on observational methods in HRI, highlighting the need for structured analytical frameworks. Section 3 details the development of our behavior coding scheme, including the initial identification and subsequent refinement of visitor behaviors. Section 4 presents the observation results, encompassing both group-level statistical analyses based on demographic factors (age and gender) and individual-level temporal modeling of engagement using a Hidden Markov Model (HMM). Section 5 discusses the implications of our findings for behavior-driven, user-centered design, adaptive interaction strategies, and the utility of time-based engagement modeling, along with study limitations. Finally, Section 6 concludes this paper and suggests future research directions.

2. Related Works

Service robots are increasingly being integrated into daily life, accompanied by a rise in large-scale field studies. This trend has led to growing interest in observational research methods suited for real-world settings. As a result, recent efforts in the HRI field have focused on systematizing observational research methodologies, and recent studies have been introduced that examine human behavior through observational techniques.

In HRI research, field studies have frequently employed observational methods to explore diverse user behaviors and perceptions, especially in deployment contexts outside the laboratory. One of the key strengths of observational research is its ability to capture behaviors in natural environments where direct intervention may be impractical or disruptive [8]. This is especially valuable in everyday social settings and public-facing environments, where artificial manipulation could unintentionally influence the behaviors being observed. By capturing user behavior in its natural context, observational methods allow researchers to identify genuine interaction patterns and uncover insights that are often inaccessible through surveys or controlled experiments [14].

In this way, observational data complement self-reported measures by providing a more accurate reflection of actual user behavior and mitigating biases inherent in retrospective reporting [15]. These strengths are particularly evident in analytical observational studies, which allow researchers to investigate behavioral outcomes in real-world settings without experimental manipulation, while still enabling causal inference through advanced techniques such as matching and stratification [16].

Recent studies have demonstrated the value of observational approaches in public and semi-public environments. For instance, Babel et al. analyzed pedestrian interactions with cleaning robots in train stations, identifying points of conflict and proposing design strategies to enhance robot acceptance [17]. Laura-Dora et al. examined how greeting behaviors used by a museum guide robot influenced visitor engagement, highlighting the role of spatial and cultural context in shaping HRI [18,19]. Lettinger et al. applied the Technology Acceptance Model [20] to study elderly users’ interactions with a social assistive robot in nursing homes, reporting generally positive emotional responses and improved social communication [21].

Matsumoto et al. proposed a conceptual approach to proximate human–robot teaming, which considers various task contexts, platforms, and sensors. Their approach supports the exploration of complex HRI components through iterative and time-series data collection and analysis [22]. However, the framework lacks specific methodological guidance, instead offering a high-level overview of potential components such as research questions, task environments, and analysis methods ranging from causal inference to quantitative assessment.

Diehl and Ramirez-Amaro proposed a causal inference framework that allows robots to understand, explain, predict, and prevent task execution failures by employing Causal Bayesian Networks [23,24]. Their method models cause–effect relationships using conditional probabilities, enabling contrastive explanations and failure prediction with preventative strategies. While offering a valuable methodological foundation for analyzing robotic task execution, their work primarily addresses controlled manipulation scenarios, limiting its applicability to broader social HRI or unstructured environments.

Boos et al. proposed the Compliance–Reactance Framework, which uses conditional probabilities to evaluate human responses to robot cues in both experimental and field studies [25]. This framework supports the analysis of compliance, cooperation, and resistance behaviors and offers recommendations for improving robot interaction design. However, it primarily focuses on the binary presence or absence of compliance, overlooking the nuanced progression of user engagement.

Kim et al. conducted an observational study using Bayesian networks to model subtle engagement behaviors in interactions between individual children with autism spectrum disorders and a social robot [26]. Their research showed the feasibility of probabilistic inference in understanding complex interaction dynamics in therapeutic contexts. However, the study focused on a highly specific setting—prosocial skill development in autism therapy—and involved a small sample size, limiting the generalizability of its findings.

While prior studies demonstrated the value of observational approaches in HRI, many have lacked structured and reproducible analytical frameworks, often relying on qualitative categorization or descriptive reporting without standardized coding schemes or formal behavior modeling [7]. To address these limitations, the present study introduces a structured dual-level observational approach that combines group-level statistical analysis based on demographic trends with individual-level temporal modeling of engagement using an HMM [27]. This method is supported by a flexible behavior coding scheme grounded in formalized behavioral grammar. The primary contributions of this study include the following: (a) formalizing grammar to define human behavioral patterns; (b) developing a low-specificity, adaptable coding scheme applicable across diverse real-world contexts; (c) implementing an iterative refinement process for improving coding reliability; (d) providing guidelines for consistent video tagging; (e) applying a probabilistic sequential model to analyze the link between time-series behavioral data and user engagement; and (f) validating the proposed approach in a real-world public environment, specifically a science museum.

3. Behavior Coding Scheme

The development of the behavior coding scheme was divided into two stages: an initial identification stage, where behavior types were classified and predefined, and a refinement stage, where the classification and labeling of these predefined behaviors were revised. This approach was adopted to ensure that behaviors observed during video tagging were interpreted objectively and consistently. Since open-ended or ambiguous definitions could lead to subjective interpretations and compromise coding reliability, separate groups of coders were assigned to each stage to maintain objectivity and consistency across both phases.

3.1. Initial Identification of Visitor Behaviors

In this stage, two coders reviewed the collected videos and formalized the specific situations and conditions in which behavior types occurred to define the coding scheme. The coding scheme was primarily defined as physical proximity [28] and interaction attempts [29,30].

Andrés et al. classified the characteristics of interactions based on the types of interactions between humans and robots [31]. They mentioned that the characteristics of interactions were categorized into interaction level, role in interaction, physical proximity, spatiotemporal context, and level of intervention. The most clearly demonstrated interaction characteristic of the visitors at the RoboLife Museum was the physical proximity. Hence, physical proximity was applied as one of the behavior factors in the coding scheme. Physical proximity indicates the distance at which humans feel a sense of intimacy with others [28]. The act of humans reducing the distance to a robot can be interpreted as feeling a sense of intimacy. In the collected videos, the visitors exhibited behavior types related to physical proximity, such as approaching, avoiding, and passing by the museum guide robot.

The second behavior factor, interaction attempts, includes the various kinds of behaviors. The visitors who approached the robot closely attempted multiple interactions with the museum guide robot. Their behaviors were categorized into three main types: attempting social interactions such as greeting and gesturing, attempting to obtain information from the screen by touching the robot, and visually observing the robot without attempting interaction.

Table 1. Types of visitor behavior toward robots observed during the initial identification phase.

Types of Visitor Behavior	Snapshot from CCTV
A person avoiding robot after recognizing where the robot is.
A person passing by the robot without knowing where the robot is.
A person greeting the robot by waving hands.
A person touching the screen after following the robot while it moves.
A person touching the screen after approaching the robot.
Two persons touching the screen after approaching the robot.
A person pointing to the screen in order for another person to touch it together.

shows representative behavior cases and selected video clips used during the initial identification phase. These examples include both individual and multi-person interactions. While some of the behaviors involve multiple people (e.g., two persons touching the screen together), each individual’s actions were later coded separately for consistency in analysis. The selected behavior types were confirmed to be classifiable based on the attributes of physical proximity and interaction attempts, which served as the foundational categories for the behavior coding scheme.

To provide a descriptive definition of visitor behaviors based on selected behavioral types, behavioral grammar was developed as shown in

Table 2. Behavioral grammar.

Item	Descriptions
Grammar 1	When the robot performs (action), the visitor (gazes/directs head) while (maintaining distance).
Grammar 2	(Interaction attempt) is made.

. The grammar was designed to express behaviors starting from the point at which a visitor intends to initiate an interaction with a robot. In addition, behavioral definitions were structured to be described based on stimuli and responses, where stimuli refer to specific robot actions (stationary or moving) and responses refer to characteristic behaviors, such as physical proximity and interaction attempts, categorized as primary behavioral factors. Grammar 1 applies to behaviors before an interaction attempt while Grammar 2 applies to behaviors after an interaction attempt. For instance, using Grammar 1, we can descriptively define behavior as ‘approaches while looking (a response related to recognizing the robot’s presence) and passes (a response related to distance) when the robot is stationary (stimulus)’. The action of recognizing the robot’s presence, such as gazing or head orientation, was considered synchronous behavior with the distance-related response. Grammar 2 allows us to define behaviors that occur after approaching the robot, that is, after reaching a distance where interaction with the robot is possible. For example, descriptive definitions such as ‘greets (a response related to social interaction)’ and ‘touches the screen (a response related to information seeking)’ are possible.

The initial draft of the coding scheme included four behaviors related to physical proximity (approach, pass, avoid, and follow) and three behaviors related to interaction attempts (touch, gesture, and none) for a total of seven behaviors. In describing the seven behaviors according to Grammars 1 and 2, behaviors that were subject to various subjective interpretations were expressed differently. For example, ‘approaches’ in approach was changed to ‘stops and stands’, and ‘avoids’ in avoid was changed to ‘steps aside’. ‘Approaches’ implies closing the distance with the robot, connecting to an interaction attempt. However, it is difficult to express with ‘approach’ the meaning that the behavior related to distance ends to attempt the next interaction, and also difficult to distinguish the visitor’s behavior shown in the video data from ‘avoid’, ‘pass’, and ‘follow’ with ‘approach’. Therefore, it was modified to a more specific expression, such as ‘stops and stands’, which conveys the meaning of the end of the behavior.

Although the behaviors of visitors avoiding or passing the robot can be distinguished by the letters ‘avoid’ and ‘pass’, they appeared as similar behaviors in the video data. Hence, expressions that clearly distinguish ‘avoid’ and ‘pass’ were needed. To this end, we distinguished between the cases where the robot is stationary or moving as stimuli, added temporal concepts such as ‘passes immediately’, and also added directional concepts such as ‘steps aside from the path the robot is moving’.

3.2. Refinement of Behavior Coding Scheme

To ensure intercoder reliability, a refinement process was conducted for the coding scheme. Two additional coders were involved in this phase. These coders were tasked with tagging videos using the established coding scheme. Before initiating the video-tagging process, they reviewed and revised the descriptive definitions for behaviors. This revision addressed instances that the coders either overlooked during the initial identification of visitor behavior, subject to varied interpretations, or were incongruent with the natural context in the representative behavior selection and initial analysis.

Modifications were primarily made to the definitions of interaction initiation. For instance, there were instances where visitors often touched not only the screen but also the robot’s body. Given the robot’s mobility within the museum lobby, determining whether the screen or the body was touched could be challenging due to the video’s angle. Consequently, the definition of touch was expanded to encompass interactions with the robot’s screen, its body, or both.

Furthermore, observations revealed that children exhibited a wide range of gestures toward the robot. There were various social interaction attempts, such as waving their hands in greeting, shaking their heads as they approached, and raising their arms and bringing them toward the robot’s face. As a result, gesture was refined to include all forms of interaction attempts, excluding touch.

Lastly, it was observed that some visitors would approach the robot, halt, and remain stationary but then engage in no further action. Since the interaction attempts are considered to be a series of continuous behaviors with physical proximity, the behavior of none, where no action is taken, was added in the refinement process as a pause. The revised coding scheme is detailed in

Table 3. Behavior coding scheme.

Group	Code	Descriptions
Physical proximity	AP (approach)	Look at the robot’s location, approach it, and stop in front of it.
	P (pass)	When the robot is stationary, look at it and immediately walk past it.
	AV (avoid)	When the robot is moving, step aside in the direction it is heading.
	F (follow)	Follow the robot as it moves in the same direction.
Interaction attempts	T (touch)	Touch the robot’s screen or body.
	G (gesture)	Make gestures toward the robot (e.g., waving, nodding, raising your arms, etc.).
	N (none)	Remain still and do nothing to interact with the robot.

. This includes the seven behaviors of physical proximity and interaction attempts, along with their respective descriptive definitions.

All behaviors were coded per individual, even when multiple people appeared in the same video frame.

3.3. Video Tagging

The two coders involved in the video tagging completed the refinement process and proceeded to tag the videos based on the refined coding scheme. The tagging focused on the behaviors exhibited by individuals toward the robot, based on the assumption that behaviors are meaningful only when the visitor has clearly recognized the robot’s presence. All behaviors were coded on an individual basis, even in scenes where multiple people appeared in the same video frame. Consequently, five key guidelines were established for the tagging process:

• The tagging process is based on the subject’s behavior. All actions become significant once the subject acknowledges the presence of the robot. Therefore, observations begin when the subject’s face is oriented toward the robot.
• The behavior code “Pass” was used when a visitor noticed the robot but continued to move past it without halting, determined by observing the direction of the visitor’s head.
• If the visitor followed the robot and eventually stopped while looking at the robot, “F-AP” was tagged sequentially. Conversely, if the visitor started to follow the robot but then diverged onto a different path, “F” was tagged.
• The behavior code “None” was specifically tagged only for the behavior after either an approach or follow action. It was used when no gesture or touch occurred after the visitor approached the robot. “None” was also used to denote the absence of interaction or the interval between different interactions.
• Continuous occurrences of the same interaction, even if separated by intervals, were considered a single action and tagged as such.

Before starting the video tagging, to accurately identify the behavior of specific individuals across different footage, we documented the external characteristics (age, gender, and clothing) of all visitors shown in the videos, along with the filename of the video in which they appeared. Subsequently, we trimmed the original footage to only include the segments featuring the targeted individual for easier reference in Dartfish Software (Pro S), a video-tagging program.

Through this data preprocessing stage, a total of 290 samples were extracted. To maintain the consistency of the tagging data, the two coders collaboratively tagged the same set of 290 people, rather than dividing the workload between them. The videos were analyzed to display both the physical proximity and the interaction attempts of all visitors, with the timeline unit set to seconds. Each value was presented in Interlinear Text format, separated by tabs for clarity. The inter-rater reliability analysis yielded a Cohen’s kappa of 0.8, indicating a substantial level of agreement between the two coders at 84.2%, as shown in

Table 4. Intercoder reliability test for two coders.

Factors	Test Results
Percent Agreement	84.19244
Scott’s Pi	0.7998
Cohen’s Kappa	0.7998
Krippendorff’s Alpha (Nominal)	0.7998
Number of Agreements	980
Number of Disagreements	184
Number of Cases	1164
Number of Decisions	2328

.

4. Observation Results

4.1. Environment

This observation study was conducted at the RoboLife Museum in Pohang, South Korea. The video data about the interactions between visitors and the museum guide robot were collected under Institutional Review Board approval (approval number: KIRO-2023-IRB-01). Upon entering the museum, visitors were to confirm their reservation status at the information desk, where they were informed about the purpose of data collection, its use, and security measures. Written consent was obtained from all participants, and especially for children, written consent was provided by their guardians, ensuring full ethical compliance.

The museum guide robot is equipped with a service that allows it to patrol and provide guidance at both the entrance and central lobby. To facilitate this, CCTV cameras capable of monitoring the entire entrance and lobby from all angles were installed. Video data were recorded for approximately 4 h each day, from 10 A.M. to 6 P.M., aligned with the four scheduled daily exhibition tours. This resulted in a total of 24 h of collected footage. Data was collected through video recordings of 290 visitors. The coding of the video data followed the behavior coding scheme described in Table 3.

Given that the observations were conducted on an unspecified population of visitors to the RoboLife Museum, demographic analysis was not feasible. Nonetheless, comparisons were made between visually discernible gender groups, and between the adult group and the children and adolescent group.

4.2. Group-Level Behavioral Observation

Group-level comparisons were conducted by categorizing participants based on gender and age. Mann–Whitney U tests were then used to determine whether there were significant differences in the frequency and duration of physical proximity and interaction attempts between these demographic groups.

A total of 290 participants were included in the analysis. For group-level comparisons, participants were categorized by gender (male = 162, female = 128) and by age group (children and adolescents = 160, adults = 130).

4.2.1. Gender Difference

No significant difference was observed in the duration of maintaining physical proximity between genders (male: M = 20.7, S.D. = 16.6; female: M = 22.9, S.D. = 17.9; p-value = 0.26, r = 0.066), nor for the duration of interaction attempts (male: M = 26.6, S.D. = 43.3; female: M = 21.5, S.D. = 43.7; p-value = 0.16, r = 0.083). There were no significant differences found in the frequency of physical proximity (male: M = 0.097, S.D. = 0.118; female: M = 0.071, S.D. = 0.073; p-value = 0.708, r = 0.022) and interaction attempts (male: M = 0.039, S.D. = 0.053; female: M = 0.029, S.D. = 0.055; p-value = 0.067, r = 0.107) between genders.

4.2.2. Age Difference

A significant difference was found in the duration of interaction attempts between age groups (U-value = 8426.5, p-value = 0.0027, r = 0.176), with children and adolescents (M = 30.2, S.D. = 48.8) exhibiting a higher duration than adults (M = 17.2, S.D. = 34.7). The frequency of interaction attempts also differed significantly between age groups (U-value = 8882, p-value = 0.021, r = 0.135), with children and adolescents (M = 0.039, S.D. = 0.057) engaging more frequently than adults (M = 0.029, S.D. = 0.049). However, the duration of physical proximity did not significantly vary (adults: M = 21.3, S.D. = 14.8; children and adolescents: M = 21.9, S.D. = 19; p-value = 0.353, r = 0.054). No significant difference was also found in the frequency of physical proximity between age groups (adults: M = 0.07, S.D. = 0.08; children and adolescents: M = 0.09, S.D. = 0.11; p-value = 0.666, r = 0.025).

4.3. Individual-Level Behavioral Observation

In the individual-level behavioral observation, we modeled visitor’s engagement levels using a probabilistic model. It focused on distinguishing the time-series behavioral patterns of museum visitors and categorizing the depth of engagement in their interactions with the museum guide robot. It was conducted by utilizing video data, capturing the interaction between visitors and the museum guide robot.

4.3.1. Model Selection and Data Preprocessing

The HMM is a probabilistic model to deduce patterns and information about hidden states that are not directly observable [32]. An HMM can be employed to discover hidden states using observable variables in time-series data [33]. Prior research [34,35,36,37] has successfully applied an HMM in analyzing human behavior patterns. The selection of the HMM is typically determined by the input format: standard HMMs are used for discrete data, Gaussian Hidden Markov Models (GHMMs) for continuous data, and Gaussian Mixture Models (GMMs) for cases that involve both discrete and continuous data. We achieved using a series of observed behavior codes and their durations and frequencies from 290 samples as input variables. In this study, a Gaussian Mixture Hidden Markov Model (GMM-HMM) was developed to infer the engagement levels of individual visitors with the museum guide robot.

The estimation of the hidden state was conducted using three data sources. The variable duration was defined as the length in seconds during which specific individual behaviors such as approach, follow, avoid, etc., were observed. The variable frequency was defined as the occurrences of each behavior occurring divided by the entire observation period for each visitor. The variable code was assigned an integer value ranging from 0 to 5 (approach = 0, avoid = 1, follow = 2, gesture = 3, pass = 4, and touch = 5), serving as a unique identifier for each distinct behavior representing 6 individual codes in Table 3. Although the behavior code none can represent a type of pause in the interaction process, it was excluded to minimize ambiguity, simplify transition dynamics, and reduce noise in the engagement analysis. While none in group-level observation provides a comprehensive view of the behavior of different visitor groups including active interaction and pauses, which is important for understanding differences in interaction styles and designing for diverse visitor needs, the HMM focuses purely on active states, allowing for a more straightforward interpretation of the visitor behavior without introducing additional complexity from pauses that can be challenging to interpret in a consistent manner. This simplification helps in achieving actionable insights into how the visitors engaged with the robot and what improvements can be made to sustain or increase engagement levels.

4.3.2. Model Training

To estimate the parameters of GMM-HMM, we employed K-fold cross-validation to determine the optimal number of hidden states (K = 5). The cross-validation procedure identified the optimal model by selecting the one with the highest average log-likelihood across folds, thereby ensuring that the model could generalize effectively to unseen data. As depicted in Figure 1, the optimal number of hidden states was selected to be five. This approach allowed the model to effectively capture the variability inherent in user behaviors. Emissions from each hidden state were modeled as a combination of several Gaussian distributions, providing a flexible representation capable of accounting for the diverse interaction patterns observed.

The model parameters were refined iteratively using the Expectation Maximization (EM) algorithm. These parameters included the initial state probabilities, transition probabilities between hidden states, and GMM-HMM parameters for emissions. The dataset used for training consisted of individual observation sequences. Each sequence corresponded to a continuous stream of behaviors from a single participant. For instance, a sequence such as approach, touch, avoid, and pass—coded from a participant approaching the robot, interacting briefly, and then leaving—was treated as one unit of analysis. The segmentation relied entirely on observed behavioral continuity and not on fixed time intervals. The training process was set to n_iteration = 100, and the dataset was split into an 8:2 ratio for the train and test sets. The model training converged after several iterations as the log-likelihood values stabilized, indicating that the model had effectively learned the underlying patterns in the data. The final model produced a transition matrix with probabilities that reflected the likelihood of users transitioning between different hidden states, as shown in Figure 2. The optimal number of hidden states was determined based on the highest average log-likelihood achieved during cross-validation, and the final model used these hidden states to classify user behaviors. The observation distributions and state transition dynamics were then further analyzed to understand user interaction patterns.

4.3.3. Model Interpretation

The results provide valuable insights into the different levels of visitor’s engagement during interactions with the robot. The transition diagram in Figure 3 allows us to better understand the behaviors associated with each of the five hidden states. Each state represents a distinct level of engagement, ranging from initial exploration to more focused interactions. The state transitions illustrate how the visitors move between these phases, highlighting typical paths in their engagement.

Figure 4 illustrates the distribution of observed behaviors across five inferred states. The x-axis in each plot represents the feature value intervals without physical units, as both duration and frequency denote normalized or scaled feature values rather than raw time or frequency.

The five states derived from the HMM, namely $L{E}_{entry}$, $L{E}_{observe}$, $M{E}_{trial}$, $M{E}_{focus}$, and $H{E}_{engaged}$, are defined based on distinct behavioral profiles involving differences in action type, duration, and frequency, as illustrated in Figure 4. These states represent various levels and styles of engagement, ranging from minimal physical interaction to sustained immersive behavior. The transitions between these states are governed not by arbitrary shifts but by measurable behavioral thresholds that reflect changes in user attention, proximity, and interaction strategy.

$L{E}_{entry}$ is primarily characterized by a high frequency of touch behavior, while gesture appears only sporadically. The duration of behavior is mostly short, typically around five, and behavior frequency generally falls within the low range, 0.025, as demonstrated in the first row of Figure 4. This pattern indicates that the user engages in minimal, short-term physical contact with the robot without progressing toward cognitively or socially meaningful interaction.

In contrast, $L{E}_{observe}$ is defined by a more diverse behavior set, with pass being the most frequently observed, followed by touch, gesture, and follow. These behaviors typically occur with moderate durations ranging from 5 to 35 and low-to-moderate frequencies between 0.025 and 0.125, as indicated in the second row of Figure 4. This state reflects a scenario in which the user lingers near the robot, expressing ambient interest but without initiating sustained or focused engagement.

As user engagement intensifies, the user transitions into the $M{E}_{trial}$ state, which is characterized by broad behavioral diversity. In this phase, pass and approach behaviors are especially prominent, with behavior durations generally short, around five, and frequency values ranging widely from 0.025 to 0.275, as illustrated in the third row of Figure 4. These behavioral patterns reflect an exploratory interaction mode, in which the user performs a variety of discrete actions including touch, gesture, pass, and approach without yet establishing any repeated or consistent engagement pattern. Functionally, $M{E}_{trial}$ serves as a transitional zone, allowing the user to either escalate into more focused interactions or regress to passive states, depending on the perceived responsiveness of the robot.

The transition into $M{E}_{focus}$ occurs when exploratory behaviors in $M{E}_{trial}$ begin to consolidate into repeated, directed interactions. $M{E}_{focus}$ is almost exclusively dominated by repetitive approach actions, with durations typically moderate, spanning 5 to 35, and frequency values clustered in the moderate range between 0.025 and 0.125, as indicated in the fourth row of Figure 4. This state reflects a form of attentional lock-in, where the user remains near the robot and demonstrates consistent behavior without yet initiating overt social interaction. Touch and gesture are not observed in $M{E}_{focus}$.

Finally, the $H{E}_{engaged}$ state is defined by sustained touch interactions accompanied by a substantial amount of gesture behavior. Other behaviors such as follow, avoid, and pass appear less frequently but are still present. Durations vary widely, ranging from short to extended periods between 5 and over 65, while frequency values are generally low and concentrated around 0.025, as demonstrated in the fifth row of Figure 4. Together, these metrics indicate a sustained, content-focused interaction session, wherein the user exhibits clear attentional commitment and task engagement.

For example, the transition from $L{E}_{entry}$ to $M{E}_{focus}$ is associated with a shift in behavioral profile from short, isolated touch interactions to longer sequences dominated by repeated approach behaviors. This change reflects an increase in spatial attention and physical proximity, as indicated by the transition structure shown in Figure 3 and the distinct behavior distributions in Figure 4. This transition is supported by a probability of 0.33, as shown in Figure 3. Alternatively, when the touch duration exceeds, the user may transition directly into $H{E}_{engaged}$, with a transition probability of 0.16. In the absence of such behavioral escalation, however, the user typically remains in $L{E}_{entry}$ through self-transition, which occurs with a probability of 0.44.

The pathway from $L{E}_{observe}$ to $M{E}_{focus}$ is initiated when repeated approach behaviors are observed, suggesting proximity-based interest. If the user’s actions become increasingly varied rather than converging toward a consistent behavioral pattern by attempting more types of interactions without repetition, the user is more likely to transition into $M{E}_{trial}$ rather than $M{E}_{focus}$. These two pathways, $L{E}_{observe}$ to $M{E}_{focus}$ and $L{E}_{observe}$ to $M{E}_{trial}$, are supported by transition probabilities of 0.54 and 0.13, respectively, and are structurally permitted as shown in Figure 3.

In the case of $M{E}_{trial}$, if users begin to repeat specific behaviors or if behavior durations exceed, the user tends to transition into $M{E}_{focus}$, with a probability of 0.39. Conversely, if the user does not perceive meaningful feedback, they often return to $L{E}_{observe}$, as reflected in the transition probability of 0.35.

While touch and gesture are not observed in $M{E}_{focus}$ according to Figure 4, their emergence at this stage is interpreted as a trigger for transition into $H{E}_{engaged}$, supported by a transition probability of 0.25, as shown in Figure 3. However, if no further interaction occurs within $M{E}_{focus}$, the user may regress to $L{E}_{entry}$ with a probability of 0.32 or $L{E}_{observe}$ with a probability of 0.21. Once in the $H{E}_{engaged}$ state, users may maintain their engagement via self-transition with a probability of 0.25 or shift back to earlier states depending on content depletion or fatigue. Specifically, transitions back to $M{E}_{focus}$ with a probability of 0.15 or to $L{E}_{entry}$ with a probability of 0.15 may occur if the user’s attention diminishes.

4.4. A Guide to Observation Studies Using Our Approach

Based on the procedure used in our observational study, we propose a generalized guideline for conducting observation studies in HRI, as illustrated in Figure 5.

4.4.1. Development of Behavior Coding Scheme

The development of the coding scheme follows a two-stage process. Initially, researchers define a preliminary set of behavior categories informed by observing collected data. Afterward, new interaction patterns may be identified, or the definitions of existing behaviors may require modification, prompting revisions to the coding categories. This iterative refinement process ensures that the coding scheme evolves to remain relevant and accurately reflects the observed behaviors. A detailed account of the process for developing the behavior coding scheme is provided in Figure 5.

4.4.2. Group-Level Observation for Broad Behavior Patterns

Group-level observation is focused on capturing broad behavioral patterns across large groups or populations. It emphasizes the frequency, duration, and proximity of interactions with robots or other stimuli. This approach is beneficial when studying group-level trends or analyzing differences across demographic variables.

To implement group-level observation effectively, researchers begin by employing video surveillance using wide-angle cameras or CCTV systems to record interactions across large environments. This setup facilitates the observation of broad behavioral trends such as how visitors approach or avoid robots. By capturing a comprehensive view of the environment, researchers can gather valuable data on interaction patterns within a diverse population.

Next, researchers apply the behavior coding scheme to categorize general actions, including approaching, avoiding, or interacting with the robot, and then conduct statistical analyses such as t-tests or an ANOVA to identify significant differences in behaviors across demographic groups, including factors like age, gender, or other relevant categories. These statistical methods are well-suited for quantifying broad interaction trends and comparing behavioral responses across different population segments, thereby providing a more comprehensive understanding of group-level engagement dynamics.

4.4.3. Individual-Level Observation for Detailed Interaction Dynamics

Individual-level observation offers a detailed lens for capturing the moment-by-moment dynamics of HRI. This approach focuses on subtle behaviors such as gestures, gaze shifts, and specific physical or verbal responses directed toward the robot. By analyzing these nuanced aspects of interaction, researchers can gain deeper insights into comprehensive interaction processes.

A key strength of this method lies in its use of time-series data to represent behavioral sequences. The process begins with the precise coding of observed behaviors, which are then converted into temporal data streams. This enables the systematic tracking of interaction dynamics over time, capturing shifts in engagement levels and transitions between distinct interaction states.

In this study, we employed an HMM to analyze temporal engagement trajectories. The HMM was particularly well-suited to our context, as it captures latent engagement states and models the probabilistic transitions between them over time. This allowed us to identify meaningful patterns in user behavior that would have been difficult to detect using static or frequency-based methods.

However, it is important to note that the choice of temporal model should be guided by the nature of the data and the research goals. While the HMM proved advantageous for modeling sequential engagement states in our study, alternative models—such as Conditional Random Fields, Dynamic Bayesian Networks or deep-learning-based temporal models—may be more appropriate in other contexts.

Ultimately, the use of temporal modeling in individual-level observation enhances our ability to understand how engagement evolves, identify key turning points in interaction, and inform the design of adaptive systems. This approach contributes to more responsive and personalized robot behaviors, fostering richer and more effective HRIs in real-world environments.

5. Discussion

This study offers meaningful insights into HRI in a real-world setting, focusing on spontaneous interactions between visitors and a museum guide robot. The findings demonstrate the effectiveness of a dual-level observational approach—combining group-level and individual-level analyses—for capturing both overall behavioral trends and dynamic engagement processes.

The results have several implications for the design and implementation of service robots in public spaces. Key themes include the importance of behavior-driven, user-centered design; the value of adaptive interaction strategies; and the utility of time-based engagement modeling for supporting long-term interaction.

5.1. Behavior-Driven, User-Centered Design

A group-level analysis revealed that children and adolescents engaged more frequently and intensely with the robot compared to adults. This aligns with prior research showing higher responsiveness among younger users to interactive technologies [38,39,40,41]. While there is some evidence of gender-related differences in attitudes or perceptions toward robots, research on the influence of gender on actual interaction frequency or duration in real-world HRI settings remains limited. Most existing studies have focused on survey-based assessments rather than direct observation of interaction behaviors, and findings have been somewhat inconsistent [42,43].

These findings underscore the importance of user-centered design principles grounded in actual behavioral patterns. For younger users, playful and exploratory interaction elements may enhance engagement [44,45], while adults may prefer more functional and task-oriented interfaces [46]. This observation aligns with the theory of selective optimization with compensation, which suggests that older users prioritize technologies that enhance everyday functionality.

To increase inclusivity and engagement, robot behaviors should be tailored to demographic and situational characteristics. Flexible interfaces and modular interaction modes can help ensure that service robots remain accessible and engaging for a wide range of users in public environments.

5.2. Adaptive, Dynamic Interaction Strategies

Through the use of an HMM, this study identified three distinct engagement states—low, moderate, and high—capturing the dynamic nature of user interaction. Visitors categorized as HE showed more sustained and complex engagement behaviors, consistent with research linking prolonged interaction to increased curiosity and attention.

These findings emphasize the need for robots to adopt real-time adaptive strategies. For users in LE states, simple prompts such as greetings or visual stimuli could initiate engagement. In contrast, HE users may benefit from content-rich experiences like storytelling, games, or interactive tasks [47]. Tailoring interactions in this way can optimize user experience and maintain engagement over time.

5.3. Utility of Time-Based Engagement Modeling

This study highlights the value of time-based engagement modeling for understanding and supporting user interaction in real-world HRI contexts. By converting behavioral codes into time-series data, we were able to analyze not only the frequency of behaviors but also the transitions between engagement states and the duration of each state. This temporal perspective allows engagement to be treated as a dynamic, context-sensitive process rather than a fixed or binary outcome.

Such an approach provides a meaningful foundation for designing robots that can adapt to the ebb and flow of user behavior in real time. Integrating temporal modeling with intelligent sensing technologies could lead to the development of socially responsive robots capable of recognizing patterns, predicting disengagement, and modifying their behavior to sustain meaningful interaction.

Importantly, this dynamic modeling also offers practical implications for long-term engagement. In public environments such as museums—where users may interact with the same robot across multiple visits—robots equipped with memory and adaptive mechanisms can foster familiarity and build user trust over time [48,49]. Our findings, particularly the transitions from LE to HE, suggest that repeated positive experiences can deepen user–robot relationships. Gradually increasing interaction complexity and personalization can help sustain interest and promote continued engagement [50].

These insights extend beyond the museum context and are relevant to other domains such as healthcare, education, and customer service—settings where long-term engagement and trust are vital for effective and meaningful HRI.

5.4. Limitations

While this study provides valuable insights into HRI within a real-world public setting, several limitations should be acknowledged.

First, although the proposed approach effectively categorized visitor engagement into three levels, it did not investigate how these engagement states could be leveraged to inform real-time robot behavior. As a result, the findings remain limited to post hoc analysis and do not yet support adaptive interactions during live deployments. Future research should explore mechanisms for detecting engagement states in real time and dynamically adjusting robot behaviors to enhance user experience.

Second, the present study focused exclusively on short-term, single-session interactions, without examining how engagement may evolve over repeated encounters. Understanding long-term engagement trajectories is essential for designing robots that can foster sustained interest, trust, and acceptance—particularly in public environments where recurring user contact is common.

Third, the analysis was limited to age and gender as demographic variables. While these factors are relevant, a more inclusive approach would incorporate additional characteristics such as cultural background, educational attainment, prior experience with technology, and cognitive or physical diversity. Expanding the demographic scope would improve the generalizability of the findings and support broader applications in other domains, including healthcare, retail, and public services.

6. Conclusions

In this study, we proposed an observational approach to analyze HRI in real-world public environments, with a specific focus on a science museum setting. By combining group-level and individual-level behavioral observation methods, our approach captured both broad behavioral trends and subtle interactional nuances in visitor interactions with a museum guide robot. The group-level analysis revealed demographic differences, showing that children and adolescents engaged more actively than adults. At the individual level, we applied a GMM-HMM model to classify visitor engagement into three distinct levels, enabling a dynamic view of how engagement evolves over time. This dual-level approach provided valuable insights into interaction dynamics and informed practical design guidelines for optimizing HRI in complex public environments.

Building on these findings, several promising directions for future research emerge. First, long-term engagement patterns—particularly for repeat or returning users—should be explored to understand how sustained interaction can influence user experience and robot effectiveness in public settings. Second, incorporating affective and physiological data using non-intrusive biometric sensing—such as gaze tracking, facial expression analysis, posture estimation, or audio-based crowd response detection—could deepen our understanding of both individual and group-level engagement. These multimodal signals can reveal internal and collective emotional states that are difficult to observe through behavioral cues alone, enabling more adaptive and context-aware HRI in public environments. Finally, given the substantial time and labor required for manual observational studies, future work should explore the use of artificial intelligence to assist in automating behavioral annotation and pattern recognition, thereby enhancing the efficiency and real-world applicability of observational research.