Balancing Personalization and Sustainability in Hotel Recommendation: A Multi-Objective Reinforcement Learning Approach

Abstract

The rapid expansion of the tourism industry underscores the necessity for sustainable hotel recommendation systems that guide user choices while safeguarding the long-term viability of the tourism ecosystem. However, existing methods often struggle to reconcile individual user preferences with sustainable consumption objectives, frequently encountering the “information cocoon” effect and lacking interpretability in their decision-making processes. To address these issues, this study proposes a multi-objective, context-aware hotel recommendation framework that integrates text mining, sequential behavior modeling, and reinforcement learning. The framework begins by employing unsupervised learning to extract multidimensional hotel features from online reviews, with an explicit emphasis on comprehensive sustainability metrics. It subsequently applies a dynamic state representation approach that merges long-term and short-term interests with real-time contextual information to accurately reflect evolving consumer needs. Furthermore, a dynamic feature weighting module is incorporated to enhance interpretability and enable context-adaptive evaluation of both commercial and sustainable attributes. The recommendation process is structured as a Markov Decision Process, leveraging a composite reward function comprising diversity penalties and sustainability incentives. Empirical analysis using real-world data validates the framework, demonstrating its contribution to sustainable tourism and achieving recommendation accuracy that surpasses existing benchmark models.

1. Introduction

The rapid expansion of the online travel agency (OTA) has fueled the robust growth of the global online hotel booking market, positioning online travel agencies as pivotal service providers [1]. An increasing number of consumers now book accommodations and share their experiences on platforms such as Ctrip and Booking.com. These platforms have created a comprehensive closed-loop system, encompassing information browsing, alternative screening, online booking, and post-consumption review sharing. However, from the perspective of tourism ecosystem sustainability, the mission of OTA platforms is evolving. They must fulfill users’ efficiency needs while also promoting sustainable consumption patterns through intelligent recommendations. This approach ensures the long-term vitality of both the hotel industry and tourism resources.

The global shift toward sustainable tourism requires innovative actions beyond traditional operational boundaries. Green recommender systems play a crucial role in narrowing the gap between users’ environmental awareness and their consumption behaviors [2]. Conventional recommendation algorithms target metrics such as click-through rates, commercial conversion, or rating prediction. Sustainability-aware systems, however, have a dual mandate. They address personalized user preferences and promote positive environmental and social results. These results include reducing carbon footprints, increasing resource efficiency, and fostering social inclusivity [3]. Including socio-environmental factors in algorithms enables OTA platforms to become active facilitators of sustainable tourism instead of remaining passive intermediaries.

In the era of the sharing economy and big data, online reviews have become an invaluable resource, offering authentic feedback derived from real guest experiences [4]. Unlike official standard descriptions, user-generated content (UGC) provides objective insights into hotel service quality and the genuine needs of consumers [5]. These reviews not only influence the decisions of potential customers but also serve as critical data for managers seeking to optimize operational strategies [6]. User reviews contain implicit sustainability signals. These include emotional evaluations of hotels’ energy-saving practices, environmental friendliness, and social responsibility. Such insights are essential for building a multidimensional sustainability assessment system. Careful analysis of review data improves recommendation accuracy, boosts platform conversion rates, and supports the green and intelligent transformation of the tourism industry.

Advancements in natural language processing (NLP) have enabled the transformation of unstructured reviews into valuable auxiliary data. Researchers employ models such as Word2Vec and Glove to capture deep semantic relationships [7]. In combination with clustering algorithms like Affinity Propagation (AP), these models facilitate the automatic construction of hotel feature systems across multiple dimensions [8]. Additionally, sentiment analysis techniques have progressed into the deep learning era, allowing for precise quantification of users’ emotional responses to attributes such as location, cleanliness, and service [9]. Building on this foundation, the present study further extends the feature space to incorporate a “sustainability” dimension. By analyzing users’ attention and emotional feedback on low-carbon travel and eco-friendly amenities, this approach provides technical support for achieving user-centric sustainable operations.

Contemporary hotel recommendation approaches on OTA platforms primarily depend on rating data prediction (e.g., matrix factorization) [10] or thematic analysis of review texts [11,12]. Sequential recommendation models (such as RNNs and Transformers) are employed to capture the dynamic evolution of user interests [13]. More recently, studies have integrated spatiotemporal contextual information, which has significantly improved the personalization of recommendations [14].

Despite these advancements, existing hotel recommendation methods—including recent reinforcement learning (RL) and multi-objective systems—still face several fundamental limitations:

• Static and single-scale modeling limitations: Traditional and sequential models often cannot distinguish long-term preferences from short-term situational intentions, such as shifts from business to holiday travel. They also lack a dual-scale architecture designed to capture the deep context dependency inherent in hotel consumption decisions;
• Opaque decision-making processes: Although deep neural network models achieve high accuracy, their complex nonlinear transformations reduce interpretability. This lack of transparency prevents the system from providing clear rationales for recommendations and reduces user trust;
• Insufficient integration of sustainability in multi-objective optimization: Although some advanced RL systems optimize for long-term value or diversity, they typically treat “sustainability” features as binary tags or secondary filters. They lack mechanisms to extract multidimensional sustainability metrics from genuine user interaction feedback and to fully integrate these metrics into the reward system, missing opportunities to promote sustainable tourism.

To address these challenges, this study aims to answer the following key research questions:

(1)

How can a state representation mechanism effectively separate users’ long-term preferences from short-term situational intentions and dynamically integrate them with real-time multidimensional contexts?

(2)

How can the decision-making process in recommendation systems be made transparent and context-adaptive to enhance interpretability without reducing accuracy?

(3)

How can multidimensional sustainability metrics derived from authentic user interaction feedback be effectively integrated into a sequential optimization framework to balance personalization, diversity, and tourism ecosystem sustainability?

To address these challenges, this study develops a dynamic, context-aware personalized hotel recommendation model based on reinforcement learning. The main contributions are as follows:

• Context-specific dual-architecture fusion model: By integrating long-term preferences encoded by a Transformer, short-term memory captured by an LSTM, and multidimensional contextual features, we establish a state representation method tailored to describe the complex evolution of user trajectories in the hotel domain;
• Interpretable recommendation approach: Using an eight-dimensional feature system that explicitly includes a review-derived “sustainability” dimension, a dynamic feature weighting network automatically determines the importance of each attribute in different scenarios. This provides a transparent explanatory foundation for recommendations;
• Advanced multi-objective RL framework: The recommendation process is formulated as a Markov Decision Process (MDP) guided by a composite reward function. This function advances previous RL models by directly incorporating review-based sustainability incentives, diversity penalties, and matching rewards, effectively balancing immediate commercial value with sustainable tourism development.

In summary, the core innovations of this study are as follows:

• A dual-architecture fusion model that integrates Transformer and LSTM to accurately capture the dynamic evolution of user preferences;
• A personalized interpretation mechanism based on dynamic feature weighting, offering a transparent explanatory foundation for recommendations through attribute importance;
• A multi-objective sequential optimization framework that explicitly integrates a review-derived sustainability dimension into its reward system, balancing personalization with sustainable tourism objectives.

The remainder of this study is organized as follows: Section 2 provides a systematic review of the relevant literature; Section 3 details the proposed recommendation model and its core components; Section 4 assesses the model’s effectiveness through case studies and experimental analysis; and Section 5 summarizes the research findings and discusses future directions.

2. Literature Review

The rapid growth of the global tourism industry has made personalized hotel recommendation systems a central element of OTA platforms. These systems must match user preferences with hotel attributes and promote sustainable tourism practices. However, traditional methods have significant limitations in optimizing recommendations for adaptability, interpretability, and long-term ecological outcomes. This section reviews the relevant literature from three perspectives: sequential recommendation systems, interpretable recommendation techniques, and the application of reinforcement learning in recommendation systems. The review synthesizes existing research and identifies key gaps in sustainable hotel recommendation, providing a strong theoretical foundation for the methodology presented in this paper.

2.1. Sequential Recommendation Systems

In the global tourism industry, sequential hotel recommendation systems aim to predict a user’s next booking by analyzing historical consumption patterns. Instead of focusing on individual algorithmic variations, recent advancements in recommendation systems are best understood through several core methodological themes. The first theme is the evolution of temporal and sequence encoding. Early models, such as Context Recurrent Neural Networks (CRNNs) [15] and Transformer-based architectures like SSE-PT [16] and BERT4Rec [17], significantly improved the capture of user-specific sequence patterns. Building on these foundations, later research introduced dynamic time-series modeling by explicitly incorporating timestamps [18] and multi-scale structural learning [19]. These advancements enable systems to quantify the dynamic impact of time intervals on next-item predictions.

Beyond basic sequence encoding, a second major theme addresses the separation of long-term and short-term user interests in recommendation scenarios. User decisions are often influenced by both stable preferences and immediate needs. Models such as STAMP [20] and CLSR [21] employ attention mechanisms and dual encoders to explicitly separate long-term memory from recently triggered short-term intentions. Concurrently, a third line of research emphasizes context integration and theoretical robustness. Recent frameworks show that integrating context-aware layers [22] or applying causal reasoning to reduce contextual interference [23] can significantly improve model accuracy. Additionally, studies find that even simplified architectures perform well when sequence and spatiotemporal context features are comprehensively fused [24], alongside ongoing efforts to address fairness and bias in neural networks [25].

2.2. Interpretable Recommendation Systems

In the hospitality and broader service sectors, interpretable recommendation systems are essential for building user trust through transparent reasoning. Early research in recommendation established foundational interpretability by extracting feature-level rationales from textual data. For example, joint modeling of user ratings and textual reviews [26], combined with attention mechanisms, enabled dynamic assessment of specific item attributes—such as service or cleanliness—and marked a major step toward generating personalized, understandable explanations [27].

Building on these feature-level extractions, a second major research trajectory focuses on dynamic attribute weighting and multimodal integration in recommendation scenarios. Methodologies now include attribute-driven decoupling representation learning [28] and attribute-based matrix factorization [29], which explicitly incorporate user and item properties to refine interaction predictions. Advanced architectures, such as dual-attention networks that process word- and comment-level features [30] and multi-modal contrastive learning frameworks [31], further enhance this process. By enabling joint learning of user preferences and item aspects, these dynamic weighting techniques allow systems to emphasize salient content and construct nuanced representations of user intent.

Alongside algorithmic advances, theoretical and structural frameworks have offered strong methodological support for explainability. The introduction of unified attribution methods such as SHAP [32], comprehensive taxonomies [33], and multi-objective evaluation criteria for trade-offs like readability versus faithfulness [34] has guided scientific assessment of interpretation quality. Additionally, the development of interpretability-constrained matrix factorization techniques [35] and frameworks that ensure structural consistency in dynamic graphs [36] demonstrates the field’s ongoing commitment to producing high-quality, traceable recommendations without exclusive reliance on auxiliary data.

2.3. Reinforcement Learning-Based Recommendation Systems

Reinforcement learning (RL) frames recommendations as sequential decision-making processes, addressing the limitations of traditional models that optimize only for immediate short-term feedback. Early research established the foundation for this paradigm by demonstrating the effectiveness of deep reinforcement learning in handling complex, page-level recommendation tasks [37]. Later work scaled these approaches, deploying the REINFORCE algorithm in large-scale e-commerce environments and providing key engineering insights for practical applications [38].

Building on these foundational implementations, research has shifted toward optimizing comprehensive, long-term objectives and multidimensional rewards. To move beyond short-term metrics, studies have designed RL objective functions that directly promote sustained user engagement [39]. At the same time, multi-objective frameworks like SMORL optimize conflicting goals such as accuracy, diversity, and novelty by adding a dedicated RL layer to standard models [40]. To further enhance ecosystem health, researchers have integrated diversity-aware reward strategies [41] and multi-agent collaborative exploration techniques for conversational settings [42], broadening the action space and mitigating long-tail data distribution issues.

To train these complex strategies without degrading the live user experience, significant progress has been made in environment modeling and offline training techniques. Sophisticated user simulators that provide multi-behavior, cross-session feedback [43] and frameworks using partially observable Markov decision processes (POMDPs) for natural language preference tracking [44] have created robust offline testing environments. Additionally, offline RL approaches leveraging Proximal Policy Optimization (PPO) [45] and constrained MDPs [46] have proven highly effective in reducing policy gradient variance, addressing cold-start problems, and maintaining long-term exposure fairness, thereby mitigating the risks associated with online exploration.

Despite advancements in sequential modeling, interpretability, and reinforcement learning, general-purpose recommendation models continue to struggle with the deep context dependency and evolving sustainability trends in tourism. Traditional sequential models often fail to distinguish stable preferences from abrupt situational shifts, while interpretable and RL frameworks typically treat sustainability as a static filter instead of a dynamic, optimizable objective. To address these gaps, this study proposes a context-aware, DRL-based sequential hotel recommendation framework. By integrating a Transformer and LSTM to capture multi-scale travel intentions and employing a dynamic feature weighting module for transparent decision-making, the framework formulates the recommendation process as a Markov Decision Process. A composite reward function that explicitly incorporates review-derived sustainability metrics effectively balances personalized matching with the long-term viability of the tourism ecosystem.

3. Materials and Methods

To address three core challenges in hotel recommendation within sustainable tourism ecosystems—static preference modeling, lack of transparency in decision-making, and short-term optimization—this study proposes a sequential hotel recommendation framework based on deep reinforcement learning (DRL). The framework models users’ ongoing hotel booking behavior on OTA platforms as a Markov Decision Process (MDP), allowing for real-time adaptation of recommendations that balance user satisfaction with environmental responsibility. As shown in Figure 1, the proposed framework encompasses the entire workflow from data input to recommendation list generation.

This section comprises the following components: (1) collection and preprocessing of hotel review data; (2) comprehensive construction of hotel feature vectors; (3) user preference modeling and state representation; and (4) recommendation strategy learning utilizing the actor–critic framework.

To improve the readability of the methodological formulas and clearly define the mathematical notations used in the reinforcement learning framework, the core notations are summarized in

Table 1. Key notations and descriptions.

Notation	Description
$H,h_i$	Set of candidate hotels, and a specific hotel $h_i.$
$f_{h_i}$	Multidimensional comprehensive feature vector of hotel $h_i.$
$P_u$	Long-term preference representation vector of user $u.$
$z_t$	Short-term memory hidden state at time $t.$
$c_t$	Temporal context vector (e.g., holiday, season) at time $t.$
$s_t$	Fused comprehensive user state vector at decision time $t.$
$a_t$	Action taken at time $t$ (i.e., the recommended hotel).
$r_t$	Composite reward received after taking action $a_t.$
$\alpha$	Weighting coefficients for sub-rewards.
$\gamma$	Discount factor for long-term reward estimation.
$w$	Dynamic attribute attention weights generated by the Actor.
$Q(s_t,a_t)$	Hybrid action-value (matching degree) for hotel $a_t$ in state $s_t.$
${\delta }_t$	Temporal difference (TD) error serving as the advantage signal.
${\pi }_{\theta },V_w$	Policy network (Actor) parameterized by $\theta$ and Value network (Critic) parameterized by $w.$

.

3.1. Collection and Preprocessing of Hotel Reviews

Upon defining the research objectives, data are systematically collected and processed to support the subsequent recommendation algorithm. The detailed workflow is illustrated in Figure 2.

In this study, we used Python (version 3.11.5) to systematically crawl publicly available user-generated hotel reviews from Ctrip, a leading online travel platform in China, with a data time range consistent with the subsequent user interaction dataset (October 2021 to November 2024). Let $H= \{\left.h_i\right|i=1,2,\dots ,m\}$ represent the set of hotels, where $m$ denotes the total number of hotels. For each hotel $h_i$, the collected text reviews are denoted as $R_i=\{\left.r_{i,k}\right|k=1,2,\dots ,n_i\}$, with $r_{i,k}$ indicating the k-th review for hotel $h_i$ and $n_i$ representing the total number of reviews for hotel $h_i$.

To efficiently extract hotel evaluation indicators from a large corpus of reviews, it is essential to assess the validity of individual sentences. Reviews are segmented, and the pre-trained BERT-base-Chinese model (version 4.35.0) generates 768-dimensional semantic representations, constructing a context-aware sentence vector space with 3,237,120 sentences and their vectors. From these, 10,000 samples are randomly selected for manual binary annotation, where each sentence is labeled as valid (1) or invalid (0) based on its relevance to hotel evaluation. This process yields a balanced distribution of approximately 54.4% valid and 45.6% invalid samples. The resulting supervised corpus is used to train a neural network model for comment validity classification, achieving an accuracy and micro F1-score of 85.32% on a 20% hold-out test set. To ensure the robustness of this gating mechanism, we conducted a qualitative audit of the discarded (label 0) sentences. These sentences mainly included metadata-like noise, such as OTA platform logistics (e.g., “The booking was successful”), generic travel itineraries unrelated to hotel quality (e.g., “We took the high-speed rail to Nanjing”), and phatic expressions without evaluative content. Manual inspection confirmed that nuanced or implicit evaluations—such as mentions of “room humidity” (cleanliness) or “reused towels” (sustainability)—were correctly categorized as valid (label 1) due to the strong contextual understanding of the pre-trained BERT model. No systematic exclusion of specific service dimensions was found, ensuring the filtered dataset remains a comprehensive and high-fidelity source for downstream attribute extraction.

Part-of-speech analysis indicates that evaluation indices are primarily represented by nouns. After filtering valid comments, the Stanford CoreNLP tool [47] is used for structural-syntactic parsing. Nouns are extracted using a hybrid word segmentation algorithm and dependency parsing, with unrelated nouns removed. The refined nouns are vectorized using Word2Vec (version 4.3.3; embedding dimension 100, window size 1, minimum count 1 for both Skip-gram and CBOW models, concatenated into 200-dimensional vectors), preserving latent semantic relationships in a low-dimensional space. The Affinity Propagation clustering algorithm [48]—using cosine similarity, a damping factor of 0.5, and the median of input similarities as preference—automatically groups semantically similar nouns through dynamic similarity propagation, yielding a cluster distribution. For clarity, let $C=\{c_j\left|j=1,2,\dots .n\}\right.$ denote the attribute set. The attribute system is constructed in accordance with the Classification and Accreditation for Star-Rated Tourist Hotels (GB/T 14308-2023) [49], which specifies core evaluation dimensions of hotel service quality in China. The subsequent sustainability-oriented analysis is grounded in Environmental-Social-Governance (ESG) sustainable development theory, providing a framework for low-carbon and eco-friendly recommendation strategies.

3.2. Construction of Hotel Feature Vectors

In personalized hotel recommendation systems designed to promote sustainable tourism, precise and comprehensive hotel feature representation serves as a crucial foundation for enhancing recommendation accuracy and personalization. To fully encompass both traditional accommodation requirements and growing environmental awareness, the extracted attribute set incorporates conventional factors (such as infrastructure and living convenience) alongside a dedicated sustainability dimension. Traditional approaches often depend on single scores or manually designed features, which are insufficient for capturing the nuanced, multi-attribute information embedded within user reviews. Such limitations hinder a comprehensive understanding of each hotel dimension and reduce the capacity to accurately align hotel offerings with user preferences. To address these challenges, this study proposes a multidimensional representation methodology that leverages automated extraction from user comments. The approach quantifies the frequency with which attributes are discussed, along with their associated sentiment polarity and intensity, and incorporates macro-level statistical features to construct structured feature vectors. This advanced representation offers a robust foundation for reinforcement learning models to precisely evaluate the alignment between hotel characteristics and user preferences.

3.2.1. Extraction of Feature Reference Frequency

The frequency with which specific hotel attributes are mentioned directly indicates the level of consumer attention toward various aspects of the property. Frequently cited attributes typically serve as key factors in the decision-making process. By computing relative frequencies, the approach mitigates bias arising from disparities in comment volume and produces feature representations that are highly comparable across hotels. The frequency of attribute $c_j$ identified in the reviews for hotel $h_i$ is denoted by $fre{q}_{h_i}^{c_j}$.

$$fre{q}_{h_i}^{c_j}={\displaystyle \frac{\left|\{r_{i,k}:r_{i,k}\in R_{c_j}\}\right|}{\left|R_i\right|}}$$

(1)

Here, $R_{c_j}$ denotes the set of reviews referencing feature $c_j$, and $\left|•\right|$ represents the cardinality of this set. Accordingly, the reference frequency vector is constructed as follows:

$$fre{q}_{h_i}=[fre{q}_{h_i}^{c_1},fre{q}_{h_i}^{c_2},\dots ,fre{q}_{h_i}^{c_n}]$$

(2)

3.2.2. Extraction of Multi-Dimensional Emotional Features

Relying solely on existence statistics is insufficient to accurately capture consumers’ true perceptions of various attributes. Sentiment analysis techniques enable the extraction of subjective emotional tendencies from textual data. By quantifying both sentiment scores and emotional intensity, it becomes possible to discern not only whether consumers’ evaluations of specific attributes are positive or negative, but also the strength of those evaluations—an aspect crucial for understanding user preferences. For sentiment analysis, the SnowNLP toolkit can accurately interpret Chinese-language context (e.g., classifying “not bad” as neutral), determine sentiment polarity via probabilistic modeling, and generate a continuous sentiment score in the range [0, 1]. Scores closer to 1 indicate a more positive sentiment expressed in the text [50].

Utilizing the SnowNLP sentiment analysis tool, customer reviews are analyzed to compute the average sentiment score for each hotel attribute. The resulting sentiment scores are then represented as a vector corresponding to the hotel’s performance across these attributes:

$$sen{t}_{h_i}^{c_j}={\displaystyle \frac{{\displaystyle \sum _{k=1}^{n_i}sentiment(T_{i,k}^j)}}{n_i}}$$

(3)

$$sentimen{t}_{h_i}=[sen{t}_{h_i}^1,sen{t}_{h_i}^2,\dots ,sen{t}_{h_i}^n]$$

(4)

Here, $sentiment(T_{i,k}^j)$ denotes the sentiment score assigned to a sentence in which feature $c_j$ is referenced within review $r_{i,k}$ as determined by the SnowNLP sentiment analysis model.

The emotional intensity of user comments is quantified by assessing the polarity of sentiments expressed. Specifically, emotional intensity (denoted as $intensit{y}_{h_i}^{c_j}$) is calculated as the absolute difference between the average sentiment score and the neutral sentiment score.

$$intensit{y}_{h_i}^{c_j}=\left|sen{t}_{h_i}^{c_j}-0.5\right|$$

(5)

The emotional intensity vector is defined as follows:

$$intensit{y}_{h_i}=[intensit{y}_{h_i}^1,intensit{y}_{h_i}^2,\dots ,intensit{y}_{h_i}^n]$$

(6)

3.2.3. Extraction of Statistical Features

Macro-level statistical features characterize a hotel’s overall performance and offer valuable additional context for the recommendation system. The total number of reviews serves as an indicator of the hotel’s popularity, the average sentiment score reflects overall customer satisfaction, and the proportion of positive reviews signifies the quality of the hotel’s reputation. These macro features, when combined with fine-grained attribute-level features, contribute to constructing a comprehensive profile of each hotel. Accordingly, the statistical features comprise: (1) the total number of hotel reviews $review\_count$, (2) the overall average sentiment score of the reviews $avg\_sentiment$, and (3) the proportion of positive reviews $pos\_ratio$. The specific calculation methods are as follows:

$$review\_count=\left|R_i\right|$$

(7)

$$avg\_sentiment={\displaystyle \frac{{\displaystyle \sum _{r\in R_i}sentiment(r)}}{\left|R_i\right|}}$$

(8)

$$pos\_ratio={\displaystyle \frac{\left|\{r\in R_i:sentiment(r)>0.5\right|}{\left|R_i\right|}}$$

(9)

The specific statistical feature vectors are constructed as follows:

$$stat{s}_{h_i}=[review\_count,avg\_sentiment,pos\_ratio]$$

(10)

Finally, features with distinct properties are concatenated along the vector dimension to construct an information-rich representation of each hotel. This comprehensive feature vector provides robust data support for subsequent similarity calculations and recommendation decision-making. To rigorously prevent data leakage, all attribute-level metrics—especially multidimensional sustainability features—are calculated using only user reviews published before each interaction. This temporal constraint ensures the model’s evaluation of a hotel’s “sustainability” performance relies solely on information genuinely available to the user and system at the time of decision-making.

$$f_{h_i}=Concat(fre{q}_{h_i},sentimen{t}_{h_i},intensit{y}_{h_i},stat{s}_{h_i})$$

(11)

To improve the reproducibility of the text-mining and attribute extraction process, we present high-level pseudo-code (Algorithm 1) that summarizes the pipeline from raw review corpus to structured hotel feature vectors.

Algorithm 1. Text Mining and Multidimensional Feature Construction Pipeline

Input: Raw hotel review set $R_i$, Pre-trained BERT-base-chinese model, Word2Vec model. Output: Structured hotel feature vectors $f_{h_i}$ for each hotel $h_i\in H$. Sentence Filtering 1: Segment reviews $r_{i,k}$ into sentences. 2: Use BERT classifier to identify valid evaluative content (label 1) and filter metadata-like noise (label 0). Linguistic Extraction 3: Perform structural-syntactic parsing with Stanford CoreNLP. 4: Extract nouns $N$ using a hybrid segmentation and dependency parsing. Semantic Mapping 5: Project refined nouns $N$ into a 200-dimensional semantic space via Word2Vec. 6: Group similar nouns $N$ into 51 fine-grained clusters using Affinity Propagation (AP). Attribute Set $\mathit{C}$ Construction 7: Map 51 clusters to 8 primary dimensions in set $C$ (e.g., Infrastructure, Sustainability). 8: Ensure alignment with GB/T 14308-2023 and ESG theoretical framework. Feature Quantification (for each hotel ${\mathit{h}}_{\mathit{i}}\in \mathit{H}$ and attribute ${\mathit{c}}_{\mathit{j}}\in \mathit{C}$): 9: Calculate ${freq}_{h_i}$ using Equation (2). 10: Calculate average sentiment score ${sent}_{h_i}^{c_j}$ using SnowNLP and Equation (3). 11: Calculate ${intensity}_{h_i}^{c_j}$ using Equation (5). Statistical Feature Extraction 12: Calculate $review\_count$ (total reviews), $avg\_sentiment$ (average sentiment), and $pos\_ratio$ (positive ratio) for hotel $h_i$. Vector Finalization 13: Concatenate ${freq}_{h_i}$, ${sent}_{h_i}^{c_j}$, ${intensity}_{h_i }^{c_j}$, and statistical vectors. 14: Ensure all metrics are calculated using reviews published before time $t$ to prevent data leakage.

3.3. User Preference Modeling and State Representation

Capturing the evolving nature of user preferences remains a core challenge for sustainable sequential hotel recommendations on OTA platforms. Traditional static profiling methods struggle to distinguish between long-term preferences—such as a consistent interest in high-end amenities—and short-term intentions, like a temporary need for budget accommodations during holidays. These methods also lack integration of real-time contextual data. As a result, recommendations are often misaligned, leading to reduced user retention on OTA platforms. To address these issues, this framework uses a hierarchical fusion architecture tailored for sustainable travel. A Transformer encoder [51] extracts stable patterns from long-term booking history. An LSTM network [52] dynamically encodes recent interactions and incorporates real-time context. The fused state vector combines multi-scale preference information, providing a detailed understanding of user needs across different travel scenarios, including environmental comfort and accessible facilities. This approach improves the temporal modeling of recommendation systems. It allows strategies to adapt to changing user preferences and balance immediate conversions with long-term user experience and environmental sustainability. Both goals are crucial for the ecological and commercial success of OTA platforms.

3.3.1. Encoding of Users’ Long-Term Preferences

To capture stable, long-range dependencies in user histories, we use a Transformer encoder. Through self-attention and positional encoding, the Transformer extracts long-term preference representations, such as seasonal travel trends, that remain robust to short-term fluctuations. These high-dimensional embeddings are essential for the recommendation framework. For each user, the feature vectors from their historical hotel stays are arranged chronologically to form a sequence:

$$H_u=[f_{h_1},f_{h_2},\dots ,f_{h_T}]$$

(12)

Here, $T$ represents the sequence length, and $f_{h_t}$ denotes the complete feature vector of hotel $h_t$. Notably, the calculation of this vector utilizes only data available prior to the time point $t$ in the sequence.

In sequential recommendation tasks, standard Transformer positional encoding assumes that time intervals within user behavior sequences are uniformly distributed. However, actual hotel reservation behaviors exhibit significant variability in time intervals—for instance, periods of frequent bookings during holidays and sparse interactions on weekdays. This discrepancy limits the model’s ability to accurately capture real-world sequence dynamics. To address this, we propose an adaptive positional encoding mechanism that enhances sinusoidal encoding by incorporating a time decay factor, ${\omega }_t^{time}$. Specifically, $\Delta t_t$ denotes the actual time interval between time steps $t$ and $t+1$, and ${\omega }_t^{time}=\exp (-{\displaystyle \frac{\Delta t_t}{{\tau }_{time}}})$ is set, where ${\tau }_{time}$ is the time attenuation constant that can be learned. The improved positional encoding formula is as follows:

$$P_{t,2i}=\sin ({\displaystyle \frac{t\cdot {\omega }_t^{time}}{{\mathrm{10,000}}^{2i/d}}})$$

(13)

$$P_{t,2i+1}=\cos ({\displaystyle \frac{t\cdot {\omega }_t^{time}}{{\mathrm{10,000}}^{2i/d}}})$$

(14)

Here, $d$ denotes the dimensionality, which is designed to capture deep, abstract semantic information from hotel features. The variable $i$ takes values in $\{0,1,\dots ,d/2-1\}$. Each dimension, $2i$ and $2i+1$, of the positional encoding corresponds to a coordinate in the $d$-dimensional space. This ensures that each time step $t$ is associated with a unique $d$-dimensional positional encoding vector, which can be added to the projected hotel feature vector.

Simultaneously, the time attenuation factor ${\omega }_t^{time}$ nonlinearly scales the time interval $\Delta t_t$ via an exponential function. When $\Delta t_t$ is large—for instance, when the interval between user behaviors spans several weeks—${\omega }_t^{time}$ approaches zero, significantly reducing the rate of change in the positional encoding relative to the time step. This diminishes the model’s ability to differentiate among long-term historical behaviors. Conversely, when $\Delta t_t$ is small, such as during periods of intensive interactions over consecutive days, ${\omega }_t^{time}$ approaches one, thereby preserving the high-frequency oscillatory characteristics of standard encoding and enabling fine-grained discrimination of recent behaviors.

Using the optimized time-aware sequential input, the user’s long-term and stable preference representation is extracted. The detailed algorithmic procedure is as follows:

First, the user’s historical accommodation sequence $H_u=[f_{h_1},f_{h_2},\dots ,f_{h_T}]$ is mapped to a unified dimensional space via a linear projection layer:

$$\mathrm{Linear}(H_u)=H_u\cdot W_e+b_e$$

(15)

Here, $W_e$ and $b_e$ denote learnable weight and bias vectors, respectively. The linear projection layer standardizes scale differences among heterogeneous features by projecting and compressing the multidimensional hotel attributes into a unified semantic space. This transformation ensures that attribute semantics are preserved, independent of their original dimensional representations.

Subsequently, the projected results are combined with the adaptive positional encoding through element-wise addition:

$$X=\mathrm{Linear}(H_u)+P$$

(16)

$X$ encapsulates both content semantics and dynamic temporal weights, offering a unified representational foundation for subsequent modeling.

Next, global dependencies are captured using the transformer encoder:

$$Z=\mathrm{Encoder}(X)$$

(17)

This method efficiently computes semantic associations across the entire sequence and overcomes the local dependency limitations of traditional models. Finally, the encoder output is aggregated to produce a long-term preference vector:

$$P_u={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T{z}_t}$$

(18)

Here, $z_t$ denotes the t-th row of the encoder output matrix, representing the context-aware embedding at time step $t$. The resulting vector $P_u$, a high-dimensional and dense representation, comprehensively encodes the user’s stable preference tendencies throughout their history and serves as the core input for subsequent dynamic user-state modeling.

3.3.2. Short-Term Memory

In sequential recommendation tasks, user decisions display dual time-scale dynamics: they are guided by long-term, stable preferences (such as a persistent inclination toward high-end hotels) while also being strongly influenced by recent interactive behaviors. To effectively capture the dynamic evolution of these recent behavioral patterns, this study introduces the Long Short-Term Memory (LSTM) network for dynamic behavior modeling.

By leveraging the LSTM’s gating mechanism, the system dynamically adjusts how it retains historical information and integrates new interactions. This approach effectively captures short-term needs and changing preferences for spatial comfort and inclusivity in sustainable travel. For example, it can identify a temporary preference for family-friendly hotels during holidays. As a result, the model improves real-time adaptability without the computational overhead of global attention for short sequences.

Specifically, the user’s interaction history with the recommendation system is systematically encoded as a dynamically evolving memory vector. At each time step $t$, the action $a_{t-1}=f(z_{t-1})$, recommended by the system at the preceding time step—generated by the policy function based on the previous hidden state $z_{t-1}$—is used as input to both the long-term and short-term memory networks. The LSTM integrates this new input with the existing memory via its gating mechanism, thereby updating its internal hidden state:

$$z_t=LSTM(a_{t-1},z_{t-1})$$

(19)

Here, $z_{t-1}$ represents the hidden state of the LSTM at time $t-1$. As a compact and informative vector, it efficiently encodes the user’s recent behavioral sequence and interaction context up to the current time. This hidden state preserves long-term dependencies while also capturing nuanced dynamics in short-term behavior patterns.

In summary, this approach constructs a comprehensive understanding of user intent through two complementary modules. First, the transformer-based encoder extracts the long-term preference $P_u$ from users’ complete historical behaviors, capturing their consistent interests and consumption patterns. Meanwhile, the LSTM module focuses on recent interaction sequences, dynamically capturing short-term behavioral patterns within the current session or task by recursively updating the hidden state $z_t$.

3.3.3. Contextual Information Fusion

Users’ hotel choices show clear temporal and contextual patterns. Preferences change over time. During statutory holidays like the Spring Festival and National Day, users prefer family-oriented accommodations and popular tourist spots. On weekdays, especially Monday to Thursday, business needs take priority. Users value transportation convenience, quick check-in, and office amenities. Seasonal changes also affect demand. In summer, users prefer seaside locations and air conditioning. In winter, heating and indoor comfort become more important. Weekends and workdays reveal further differences, such as price sensitivity, length of stay, and advance booking times.

If these nuanced temporal context signals are overlooked and the model relies solely on static long-term preferences $P_u$ or short-term memory $z_t$, it becomes challenging to discern whether a user is planning a May Day holiday trip or making a last-minute arrangement for tomorrow. This disconnect can result in recommendations that do not align with the user’s actual, immediate intentions.

To address this, we explicitly introduce a temporal context vector $c_t$ at decision time $t$, encoding the date as a periodic feature comprising the following elements:

$$c_t=[\sin {\displaystyle \frac{2\pi m}{12}},\cos {\displaystyle \frac{2\pi m}{12}},\sin {\displaystyle \frac{2\pi w}{7}},\cos {\displaystyle \frac{2\pi w}{7}},{\mathrm{I}}_{holiday},e_{season}]$$

(20)

Here, m and w represent the month (1–12) and weekday (1–7), respectively. These values are mapped into a four-dimensional space using sine and cosine transformations, effectively eliminating the discontinuity often associated with traditional numerical encoding at periodic boundaries. ${\mathrm{I}}_{holiday}$ is a binary indicator variable set to 1 if the time falls on a legal holiday or weekend, and 0 otherwise. This strictly calendar-based indicator is generated using the Python Chinese calendar library (version 1.11.0), which accurately identifies official Chinese statutory holidays (e.g., Spring Festival, National Day) and standard weekends. It does not include unofficial peak travel periods or empirical demand fluctuations observed on the platform. This variable differentiates between users’ tourism-oriented and business-oriented preferences. $e_{season}$ is a four-dimensional one-hot vector representing spring, summer, autumn, and winter, designed to capture fluctuations in hotel demand resulting from seasonal climate variations.

This vector allows the recommendation system to learn context-aware decision-making patterns, such as increased user focus on leisure facilities during holidays or prioritization of transportation convenience for business travel.

3.3.4. State Vector Construction

The final state vector, $s_t$, is derived by integrating and fusing users’ long-term preferences, recent behavioral dynamics, and the current temporal context. Specifically, the long-term interest representation $P_u$ (extracted by the Transformer), the short-term memory $z_t$ (encoded by the LSTM), and the periodic temporal feature vector $c_t$ are jointly projected into a unified state space:

$$s_t=Concat(P_u,c_t,z_{t-1})$$

(21)

The state vector, $s_t$, serves as an informative signal for the subsequent actor–critic network, guiding the generation of personalized recommendations and the evaluation of the state’s long-term value.

By employing this multi-level and dynamic user state representation approach, the system achieves a comprehensive description of user preferences—spanning from macro to micro and from static to dynamic dimensions. This methodology provides a robust foundation for generating truly context-aware and forward-looking personalized recommendations.

3.4. Actor–Critic-Based Recommendation Strategy Learning

Personalized sustainable hotel recommendations on OTA platforms involve a continuous, sequential decision-making process. User interaction feedback reveals immediate booking intentions and shapes long-term platform engagement. Traditional static models often focus on short-term click-through rates. They may overlook changes in user preferences and fail to balance the exploration of latent needs. As a result, recommendations become homogenized, and user retention on OTA platforms declines.

To address these challenges, this study models the sustainable hotel recommendation sequence on OTA platforms as a Markov Decision Process (MDP) and employs the Actor–Critic deep reinforcement learning framework. The policy network (Actor) creates personalized recommendation distributions tailored to eco-friendly scenarios. The value network (Critic) estimates expected cumulative rewards. It balances immediate user preferences with long-term sustainability goals, such as promoting eco-friendly and socially inclusive accommodation options. This framework balances exploration and exploitation. It shifts the system from a passive matching tool to an active decision-making agent that optimizes short-term accuracy and long-term sustainable values on OTA platforms.

3.4.1. MDP Formalization

Formalizing recommendation problems as Markov Decision Processes (MDPs) provides a solid foundation for applying reinforcement learning. This method turns vague objectives, such as improving user satisfaction, into structured and optimizable sequential decision-making tasks. Key elements include state representations, which encode user behaviors, context, and recommendation history to reflect real-time user conditions. Actions are the recommended hotel options, forming the decision space. Rewards quantify user feedback and measure the immediate value of each interaction. The state–action–reward loop creates an interactive process that allows the system to learn from experience and optimize long-term strategies through ongoing interaction. This shift enables proactive planning instead of passive response.

Within this overarching framework, the elements of the MDP are further instantiated for the hotel recommendation scenario. The state $s_t$ represents all observable user information. It integrates long-term interests, temporal context, and recent interactions up to the previous moment. This structure gives the reinforcement learning recommendation system a complete view of the user’s current state. The action $a_t$ is a candidate hotel recommended to the user. Each hotel is uniquely identified by a specific hotel ID. The model maps each action to the hotel’s full feature vector $f_{h_i}$. The reward $r_t$ is based on the user’s response to the recommendation, specifically whether the user accepts the suggestion and completes a booking. This feedback serves as a key signal for evaluating each decision and guides ongoing optimization of the recommendation strategy. State transitions occur naturally through user interactions. When a user acts on a recommendation, the behavior updates both their historical sequence and short-term memory. Combined with the new temporal context, this forms the next state $s_{t+1}$. This process creates a closed-loop, data-driven, sequential decision-making flow. The reinforcement learning recommendation system can continually learn from real user feedback and move toward an optimal recommendation strategy.

3.4.2. Actor Network

The actor network is the strategic core of the system. It moves beyond traditional fixed-feature weighting by learning parameterized policy functions that adapt recommendations to the user’s current state. The network uses a dynamic feature adjustment mechanism. Real-time analysis of the user’s state and intent generates attention weights, assigning specific importance to hotel attributes such as price, location, and rating. This process captures users’ immediate preferences in various scenarios. For example, it can emphasize room capacity during holidays or prioritize transportation convenience on weekdays. As a result, recommendations shift from static profile matching to context-aware, adaptive responses. This improvement increases both the flexibility and personalization of the strategy.

• Dynamic Weight Generation

Traditional recommendation models often use fixed feature weights for all users and scenarios. They overlook the fact that the same user may prioritize different factors at different times. To address this limitation, we introduce a lightweight network that generates attention weights for $n$ hotel attributes in real time, tailored to the current user state $s_t$:

$$w_t=\mathrm{Softmax}(W_w{s}_t+b_w)$$

(22)

Here, $w_t$ is an $n$-dimensional vector representing the relative attention a user assigns to $n$ hotel attributes in the current state $s_t$; $W_w$ is a learnable weight matrix used to project the state vector into the attribute attention space; and $b_w$ is the bias term. This approach allows the system to detect contextual changes and adjust attribute importance in real time. It addresses the main limitation of static weight models, which lack context awareness. As a result, the system delivers more timely and personalized recommendations.

• Matching Degree Calculation

For a candidate hotel $h_i$, its attractiveness is determined by both a fundamental matching score and a deep adjustment component.

First, the basic matching score integrates the mention frequency and sentiment orientation of hotel attributes, applying the dynamic weight $w_t$ for aggregation:

$$bas{e}_{h_i}=w_t^{T}fre{q}_{h_i}+\gamma w_t^{T}sentimen{t}_{h_i}$$

(23)

Here, $\gamma >0$ is a hyperparameter that adjusts the relative influence of the sentiment signal compared to the frequency signal, and $w_t^T$ denotes the transpose of $w_t$, enabling weighted summation. By integrating both objective behavior (mention frequency) and subjective attitude (sentiment tendency) within a unified weighting framework, and dynamically modulating the contribution of each attribute through $w_t$, this approach preserves the interpretability of traditional content matching. Additionally, it mitigates excessive reliance on fixed weights, reducing susceptibility to noise or outdated signals, and thereby enhances the robustness and semantic validity of the basic scoring mechanism.

To more effectively capture high-order nonlinear interactions between user states and hotel attributes, two independent neural encoders are introduced to compute the deep adjustment component. Specifically:

$$adjus{t}_{h_i}={\varphi }_s{(s_t)}^T{\varphi }_a(f_{h_i})$$

(24)

Here, ${\varphi }_s(\cdot )$ denotes a state encoding network that projects user states $s_t$ into a fixed-dimensional semantic space, while ${\varphi }_a(\cdot )$ refers to an action encoding network that maps the original feature vector $f_{h_i}$ of hotel $h_i$ into the same semantic space. In this study, both networks are implemented as three-layer fully connected architectures, and their output vectors are combined via inner product to assess semantic compatibility within the hidden space. Traditional linear or shallow models struggle to capture the complex, coupled relationships between state and action (e.g., a specific user state may be sensitive only to certain hotel types). By leveraging implicit semantic alignment, the deep interaction module can autonomously uncover high-order patterns, thereby compensating for the limited expressiveness of explicit weighted models and significantly enhancing the Q-value’s ability to discriminate subtle preference differences.

Ultimately, the comprehensive Q value for hotel $h_i$ is computed as the weighted sum of two components:

$$\mathrm{Q}(s_t,a_{h_i})=bas{e}_{h_i}+\beta \cdot adjus{t}_{h_i}$$

(25)

Here, $\beta \ge 0$ is the balance coefficient that controls the contribution of the deep adjustment component (which can be set as a hyperparameter or learned), and $a_{h_i}$ denotes the action of recommending hotel $h_i$. To prevent the deep interaction term from overwhelming the interpretable base score, the output vectors of the dual encoders—${\varphi }_s(s_t)$ and ${\varphi }_a(f_{h_i})$—are L2-normalized before their inner product is computed. This bounds the deep adjustment component within [−1, 1]. The balance coefficient $\beta$ is also constrained (empirically set to 0.5), ensuring the deep interaction serves only as a bounded residual adjustment. This approach maintains explicit semantic matching as the dominant and interpretable driver of the recommendation. This hybrid architecture accounts for both interpretability (base term) and generalization capability (adjustment term), thereby avoiding the debugging challenges of pure black-box models as well as the performance limitations of strictly rule-based approaches. By tuning $\beta$, the system can flexibly balance these aspects under varying data sparsity conditions or business objectives, achieving an optimal trade-off between stability and expressiveness.

• Policy Function

To generate the final recommendation distribution, the Q values of all candidate hotels are converted into probabilities using the $\mathrm{softmax}$ function:

$${\pi }_{\theta }(a_{h_i}|s_t)={\displaystyle \frac{\exp (\mathrm{Q}(s_t,a_{h_i}))}{{\displaystyle \sum _{j=1}^m\exp (\mathrm{Q}(s_t,a_j))}}}$$

(26)

Here, $m$ denotes the size of the candidate hotel pool; ${\pi }_{\theta }(a_{h_i}|s_t)$ represents the probability of selecting action $a_{h_i}$ (i.e., recommending hotel $h_i$) in state $s_t$; and $\theta$ comprises the learnable parameters of the entire actor network (including weight generators, encoding networks, etc.). The softmax-based strategy output enables random exploration. Hotels with higher Q values are more likely to be recommended, while those with lower Q values still receive some exposure. This approach mitigates premature convergence in reinforcement learning. The resulting probability distribution also provides a smooth gradient for the critic network. This stability ensures effective policy updates, which are essential for successful end-to-end training.

This approach uses a five-tiered design: dynamic weighting, interpretable base scoring, deep interaction components, hybrid Q-value computation, and probabilistic strategies. It addresses the limitations of traditional recommendation systems, including limited contextual adaptability, shallow interaction modeling, and inadequate exploration. As a result, the system becomes an intelligent recommendation engine with accuracy, flexibility, and robust learning capacity.

3.4.3. Critic Network

In deep reinforcement learning, policy gradient methods often face unstable training and slow convergence due to high variance in feedback signals. To address these issues, this study uses a Critic network to assess values. The Critic accurately predicts long-term cumulative returns based on the current user state. The system compares user interaction feedback with the Critic’s baseline value for each state. This comparison produces an advantage signal that quantifies the merit of the current recommendation against average performance. The mechanism removes random fluctuations caused by environmental noise. It provides a low-variance, discriminative gradient direction for optimizing Actor networks. As a result, the learning process becomes more stable and efficiently identifies recommendation behaviors that enhance long-term user experience. This approach accelerates the development of effective strategies and improves overall recommendation performance.

The critic network serves as the central component for state-value assessment within this framework. Architecturally, it is implemented as a multi-layer perceptron (MLP) that takes the current user’s state vector $s_t$ as input and outputs a scalar value $V(s_t;w)$, representing the long-term expected cumulative discounted return from state $s_t$.

Here, $w$ denotes all learnable parameters of the critic network; $V(\cdot ;w)$ is the parametric approximation of the true state value function $V^{\pi }(s_t)={\mathrm{{\rm E}}}_{\pi }[{\displaystyle \sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}}]$; and $\gamma$ represents the discount factor.

To ensure efficient and stable policy learning, the system employs the temporal difference (TD) learning mechanism. The core principle of TD learning is to utilize the value prediction of the subsequent state to construct the return target for the current state, thereby eliminating the need to wait until the conclusion of the entire interaction sequence. Specifically, after the recommendation system executes action $a_t$ in state $s_t$, it observes the immediate reward $r_t$ and transitions to the next state $s_{t+1}$, constructing the following one-step TD target value:

$$V_{target}=r_t+\gamma \cdot V(s_{t+1};w)$$

(27)

Here, $r_t$ denotes the immediate reward signal derived from quantifying user behavior; $\gamma$ is the discount factor used to balance short-term gains and long-term experience; and $V(s_{t+1};w)$ represents the critic’s current estimate of the value of the next state.

Next, compute the temporal difference (TD) error:

$${\delta }_t=V_{target}-V(s_t;w)$$

(28)

This error essentially quantifies the deviation between the observed one-step return and the current value prediction. In this approach, the TD error is directly utilized as an estimate of the advantage function:

$$A_t={\delta }_t$$

(29)

This design offers clear behavioral interpretability: when $A_t\ge 0$, the total return from executing action $a_t$ exceeds the average for the current state, indicating that the recommendation outperforms random selection and should be reinforced in the policy. Conversely, when $A_t\le 0$, the action’s performance falls short of expectations and should be suppressed within the policy.

Unlike Monte Carlo methods, which require complete interaction sequences to compute returns, temporal difference (TD) learning allows online updates without full trajectories. This approach reduces variance and improves sample efficiency. It is especially well-suited for hotel recommendation scenarios with frequent interactions and sparse feedback. The Critic and Actor networks work in close synergy: the Critic provides advantage signals $A_t$ to guide updates of the Actor’s parameters, while the new trajectories generated by the Actor facilitate ongoing calibration of the Critic’s value estimates. This bidirectional mechanism fosters a dynamic balance between exploration and exploitation, supporting the system’s steady convergence toward an optimal strategy.

The current framework operates in an offline reinforcement learning setting with a fixed set of logged bandit feedback. Value estimation relies only on these logged interaction trajectories and does not use explicit off-policy corrections, such as importance sampling or conservative value estimation. As a result, the critic network may be limited by distribution shifts or may overestimate out-of-distribution actions. To address this, the framework uses an imitation learning (behavioral cloning) warm start during initialization, as detailed in Algorithm 2. This step anchors the initial policy close to the behavioral policy and reduces extreme variance before the main TD-learning loop.

In summary, the critic network provides gradient signals with low variance and a high signal-to-noise ratio for policy optimization through efficient TD learning. It also models users’ long-term experience directly. This capability allows the recommendation system to plan for the future by continuously optimizing and increasing user satisfaction during dynamic interactions. The critic network is a foundational pillar for achieving high-performance and sustainable personalized recommendations.

3.4.4. Reward Function Design

The reward function shapes the system’s optimization objectives and strategic direction. Relying only on immediate feedback, such as clicks or conversions, can lead to overly homogeneous recommendations. This may briefly boost metrics but often reduces diversity, speeds up user interest convergence, and lowers conversion rates. To address this, the study adopts a multidimensional reward mechanism. Immediate relevance is measured by integrating explicit and implicit feedback. Long-term value is supported by encouraging exploration of new or rarely recommended high-quality hotels and discouraging repetitive recommendations. This ensures balanced category coverage. The approach balances short-term performance with long-term ecosystem health. It enables the collaborative optimization of user satisfaction and environmental sustainability.

To enable the recommendation system to learn strategies that balance immediate accuracy, semantic relevance, and sustained diversity, this study introduces a composite reward function. The total reward at time $t$ is defined as follows:

$$r_t={\alpha }_m\cdot r_t^{match}+{\alpha }_s\cdot r_t^{similarity}+r_t^{diversity}+{\alpha }_{\mathrm{sust}}\cdot r_t^{sust}$$

(30)

Here, $r_t$ denotes the comprehensive reward signal generated after the recommendation system executes action $a_t$ (i.e., recommends a hotel) in state $s_t$. ${\alpha }_m\ge 0$, ${\alpha }_s\ge 0$ and ${\alpha }_{sust}\ge 0$ are adjustable hyperparameters that control the relative weights of the matching reward, similarity reward and sustainability reward, respectively. The four sub-rewards provide learning signals across the dimensions of exact matching, semantic proximity, and behavioral diversity, collectively forming a multi-objective optimization framework.

• Basic Matching Rewards

This reward delivers the most direct and substantial positive feedback, defined as follows:

$$r_t^{match}=\mathrm{I}[a_t=f_{h_{t+1}}]$$

(31)

Here, $\mathrm{I}(\cdot )$ denotes an indicator function that returns a value of 1 when the specified condition is satisfied and 0 otherwise. When the recommendation result exactly matches the user’s final selection, a distinct high-value reward is assigned. Although this “all or nothing” signal is sparse, it offers strong discriminative power, effectively guiding the learning direction of the strategy and preventing the model from deviating from genuine user intent.

• Similarity Reward

To address the sparsity of matching rewards, a continuous semantic similarity reward is incorporated:

$$r_t^{similarity}=\mathrm{sim}(a_t,f_{h_{t+1}})$$

(32)

Here, $\mathrm{sim}(\cdot ,\cdot )$ denotes cosine similarity, which measures the directional alignment between two hotel feature vectors within the semantic space. Even if a recommendation is not ultimately selected by the user, as long as it is semantically similar to the user’s actual choice, the system receives partial positive feedback. This continuous, smooth gradient signal greatly enhances the stability and sample efficiency of policy learning, playing a particularly crucial role in cold-start or long-tail scenarios.

• Diversity Penalty

To mitigate repetitive recommendations and encourage the exploration of novel options, a diversity penalty is incorporated:

$$r_t^{diversity}=-{\lambda }_d\cdot {\max }_{h_i\in H_u^{recent}}\mathrm{sim}(a_t,f_{h_i})$$

(33)

Here, ${\lambda }_d>0$ denotes the hyperparameter controlling the strength of the diversity penalty, and $H_u^{recent}$ represents the set of hotels that user $u$ has recently visited. ${\max }_{h_i\in H_u^{recent}}\mathrm{sim}(a_t,f_{h_i})$ calculates the similarity between the currently recommended hotel, $a_t$, and the most semantically similar hotel within the user’s recent history. This approach effectively mitigates the information cocoon effect by penalizing high similarity in recent user interactions. For instance, after a user stays at a business chain hotel, the system avoids recommending similar properties. While this may temporarily decrease click-through rates, it enhances the freshness of the user experience and promotes recommendation diversity. The strategy encourages exploration of differentiated, high-value hotel options to optimize long-term user satisfaction and sustain the health of the platform ecosystem.

• Sustainability Reward

To proactively guide users toward sustainable accommodation choices, this study introduces a dedicated sustainability reward ($r_t^{sust}$). Unlike conventional binary green tags, the sustainability dimension within this framework is comprehensively derived from multidimensional user reviews. It broadly encompasses key aspects of sustainability, including indoor environmental quality (IEQ), spatial comfort, environmental integration, and social inclusivity. The detailed composition and statistical analysis of these sub-attributes will be presented in Section 4.

To precisely assess a hotel’s sustainability performance based on historical feedback, the reward evaluates the target hotel $h$ using both the mention frequency feature ($S_{h,sust}$) and the sentiment feature ($F_{h,sust}$) associated with sustainability attributes. This reward is formally defined as follows:

$$r_t^{\mathrm{sust}}=(S_{h,sust}-0.5)\log (1+{\beta }_{sust}\cdot F_{h,sust})$$

(34)

where the frequency feature is subjected to a non-negative truncation, such that $F_{h,sust}=\max (F_{h,sust},0)$, and ${\beta }_{sust}$ is a scaling coefficient (set to 10.0 in this study) that controls the logarithmic growth rate of the frequency impact.

This formulation establishes a bidirectional incentive mechanism grounded in user sentiment polarity. When the overall sentiment toward a hotel’s sustainability attributes is positive $S_{h,sust}>0.5$, a higher mention frequency $F_{h,sust}$ amplifies the positive reward, motivating the system to prioritize hotels recognized for their sustainable, comfortable, and inclusive practices. Conversely, if the sentiment is negative $S_{h,sust}<0.5$, the logarithmic frequency term imposes a penalty, effectively discouraging the recommendation of hotels with subpar environmental or social performance.

To provide a clear and practical interpretation of this multi-objective mechanism, consider the real-world operational logic of a recommendation agent: The basic matching reward serves as a direct “conversion bonus”. When the system predicts the exact hotel a user books, it receives the maximum reward, ensuring the policy prioritizes accuracy. The semantic similarity reward functions as a “near-miss incentive”. If a user books a specific five-star resort, but the system recommends a different resort with similar amenities, it still receives partial credit. This prevents learning stagnation when exact matches are rare. The diversity penalty acts as an “anti-fatigue nudge”. For example, if a user has recently stayed at several standard business chains, the system is penalized for recommending another identical chain. This penalty encourages exploration of new categories and helps break the information cocoon. Finally, the sustainability reward functions as a “green premium”. When choosing between two equally relevant hotels, the system receives extra reinforcement for recommending the one with strong eco-friendly feedback, such as excellent public transit access or high energy efficiency. This approach proactively guides users toward sustainable choices.

In summary, the composite reward function provides strong supervisory signals through precise matching. It reduces sparsity with semantic similarity, encourages exploration with diversity penalties, and promotes sustainable well-being and social inclusivity through sustainability rewards. Together, these four components enable the Actor–Critic framework to learn an intelligent recommendation strategy. This approach captures users’ immediate intentions and preserves long-term experience quality throughout sequential interactions. Ultimately, it maximizes personalized service value. The model training process includes three core stages: initialization with imitation learning warm-start, end-to-end reinforcement learning main training, and model evaluation with optimal weight preservation. The logic, mathematical definitions, and iterative steps for each stage are detailed in Algorithm 2.

Algorithm 2. Reinforcement Learning-based Hotel Recommendation Strategy Learning

Input: User interaction dataset $D$, discount factor $\gamma$, reward weights ${\alpha }_m$, ${\alpha }_s$, ${\alpha }_{sust}$, coefficients ${\lambda }_c$, ${\lambda }_e$, ${\lambda }_{margin}$. Output: Optimized Actor network parameters $\theta$ and Critic network parameters $w$. Initialization and Warm-start 1: Initialize Actor network ${\pi }_{\theta }$ and Critic network $V$ with random weights. 2: Imitation Learning (Behavioral Cloning): Pre-train ${\pi }_{\theta }$ by minimizing the cross-entropy loss $L_{BC}$ between the policy distribution and the actual historical hotel selections $h_{t+1}$ to mitigate initial high variance. Reinforcement Learning Training Loop 3: for each training epoch do 4:Sample a mini-batch of user sequences from $D$. 5: for each time step $t$ in the sequence do 6: State Representation: Construct the fused state vector $s_t$ by integrating long-term preference $P_u$ (via Transformer), short-term memory $z_t$ (via LSTM), and temporal context $c_t$: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{s}_t=Concat(P_u,z_t,c_t )$ 7: Action Generation: The Actor network ${\pi }_{\theta }$ computes the probability distribution over candidate hotels using a Softmax strategy based on the hybrid Q-values $Q(s_t,a_t)$: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\pi }_{\theta }\left(a_t|s_t\right)={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(Q\right(s_t,a_t\left)\right)}{{\sum }_{j=1}^m\mathrm{e}\mathrm{x}\mathrm{p}\left(Q\right(s_t,a_j\left)\right)}}$ 8: Execute and Observe: Recommend hotel $a_t$, observe the user interaction feedback, and calculate the composite reward $r_t$: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{r}_t={\alpha }_m·r_t^{match}+{\alpha }_s·r_t^{similarity}+r_t^{diversity}+{\alpha }_{sust}·r_t^{sust}$ 9: Next State: Transition to $s_{t+1}$ by updating the user’s behavioral memory with the new interaction. 10: Value Assessment: Use the Critic network to estimate the Temporal Difference (TD) target $V_{target}$ and the advantage signal ${\delta }_t$ (TD error): $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{V}_{target}=r_t+\gamma V\left(s_{t+1};w\right)$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\delta }_t=V_{target}-V(s_t;w)$ 11: end for 12: Optimization: Update network parameters $\theta$ and $w$ by minimizing the multi-objective loss function $L(\theta ,w)$: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{L}_{total}=L_{policy}+{{\lambda }_c·L}_{critic}+{\lambda }_e·L_{entropy}+{\lambda }_{margin}·L_{margin}$ 13: end for

4. Results

To assess the effectiveness and feasibility of the proposed deep reinforcement learning-based sequential hotel recommendation framework, this section presents an empirical analysis using real user behavior sequence data from the Ctrip platform. The study follows the methodological process outlined in Section 3. It systematically reproduces each stage, from constructing multi-dimensional hotel features—including infrastructural, experiential, and sustainability indicators—to user state encoding, strategy training, and evaluation. The analysis examines key intermediate states, compares results with leading benchmark methods, and quantitatively assesses multidimensional performance metrics. These steps validate the framework’s strengths in capturing dynamic user preferences, promoting sustainable travel choices, and optimizing long-term outcomes. The following discussion highlights the approach’s advancements and practical value across four dimensions: data and experimental settings, effectiveness of state representation, interpretability of the decision mechanism, and offline recommendation performance comparison.

4.1. Dataset and Experimental Settings

This section details the dataset composition, preprocessing procedures, experimental environment, and evaluation protocols employed to validate the DRL-based recommendation framework. The aim is to ensure research reproducibility and enhance the credibility of the results.

4.1.1. Data Sources and Collection

To establish a realistic sequential decision-making environment for hotel recommendation in OTA platforms, this study employs publicly available user-generated user interaction feedback from Ctrip, a leading OTA platform in China. The dataset encompasses temporal elements (precisely recording hotel IDs and stay dates), content richness (synchronously acquiring hotel review texts for feature vector construction), and contextual features (extracting month, week, holiday status, and seasonal information from timestamps). Collectively, these elements comprehensively characterize the dynamic evolution and environmental dependencies of user behavior sequences.

Table 2. Basic statistical information of the dataset.

Statistic	Value
number of users	14,522
number of hotels	806
total number of interaction records	94,634
user average sequence length	6.52
time span	2021/10–2024/11
data sparsity	0.9919
minimum value of sequence length	3
maximum sequence length	28
median sequence length	5

summarizes the key statistical characteristics of the dataset. Of particular importance is the average user sequence length, as it directly influences the complexity of training episodes and the model’s capacity to capture long-term dependencies.

As shown in Table 2, the dataset contains 94,634 interaction records from 14,522 users and 806 hotels, spanning October 2021 to November 2024. The average user interaction sequence length is 6.52, with a median of 5 and a minimum of 3, indicating that most sequences are short and only a small subset of users have interaction trajectories extending up to 28 events. The dataset is highly sparse, with a sparsity level of 0.9919, highlighting the pronounced sparsity of the user–hotel interaction matrix. These characteristics align with real-world business scenarios on OTA platforms: short sequences reflect the inherently low-frequency nature of hotel bookings in OTA platforms, while high sparsity is a common challenge in e-commerce user–item interaction matrices. These issues not only affect the accuracy of sequence modeling but also hinder the deployment of recommendation systems on platforms managing large-scale sparse data. To address these challenges, this study utilizes a Transformer encoder to integrate global interaction information, facilitating the identification of latent, multifaceted preference associations—including users’ intrinsic tendencies toward sustainable accommodations—within short sequences. Furthermore, the composite reward mechanism, which integrates both similarity-based and sustainability-oriented components, delivers smooth learning signals, effectively reducing training instability in sparse data scenarios while simultaneously guiding the system toward eco-friendly recommendation strategies.

4.1.2. Data Preprocessing and Sequence Partitioning

• Hotel Feature Vector Construction

Following the methodology outlined in Section 3.1, this study collected online review texts from 806 hotels listed on the Ctrip platform and automatically extracted eight principal features (encompassing infrastructural, experiential, and sustainability dimensions), as illustrated in Figure 3. Concurrently, leveraging the approach described in Section 3.2, a static feature vector $f_{h_i}\in {\mathbb{R}}^{27}$ was constructed for each hotel $h_i$. This vector comprehensively represents the key attribute dimensions influencing user decision-making, including price level, geographical location, facility configuration, and attribute-level metrics such as mention frequency and sentiment tendency derived from historical user reviews.

Affinity Propagation clustering automatically generated 51 fine-grained semantic clusters from valid hotel review nouns. To create a structured and interpretable attribute system, these clusters were systematically mapped to eight high-level dimensions: Infrastructure, Convenience Facilities, Recreation, Hospitality, Accessibility, Livability, User Experience, and Sustainability. This mapping used a predefined semantic dictionary aligned with the GB/T 14308-2023 standard and the ESG theoretical framework. The complete mapping between the eight dimensions and their underlying sub-clusters is detailed in

Table 3. Mapping of clusters to hotel dimensions.

Primary Dimension	Associated Sub-Clusters
Infrastructure	Door & Window; Balcony; Room Size; Bedding; Interior Decor & Furnishings; In-room Entertainment.
Convenience Facilities	Furniture; Wireless Network; Bathroom Facilities; Daily Necessities; Laundry Room; Parking Facility; Takeaway & Delivery Service.
Recreation	Swimming Facility; Gym; Other Sports; Amusement; Show; Leisure & Relaxation; Bar Service.
Hospitality	Front Desk Service; Luggage Handling; Gift & Complimentary Items; Airport Pick-up/Drop-off; Extended Stay/Late Service; Dining Service; Room Upgrade.
Accessibility	High-speed Rail Station; Airport; Scenic Attractions; Business District.
Livability	Basic Odor Perception; In-Room Brightness; Spatial Breathability; Auditory Quietness; Window View Quality; External Disturbance Perception.
User Experience	Value for Money; Safety & Security; Privacy; Comfort.
Sustainability	Low-Carbon Mobility; Energy Efficiency & Indoor Climate; Resource & Light Environment; Acoustic & Olfactory Health; Biophilic Environment; Social Inclusivity

.

To embed sustainable development goals into the recommendation logic, the eighth dimension, Sustainability, is intrinsically integrated into the hotel feature system. This dimension transcends conventional service metrics by quantifying the hotel’s performance in low-carbon operations, environmental health, and social inclusivity. Specifically, it clusters fine-grained attributes derived from guest reviews into thematic categories that reflect a hotel’s commitment to sustainable practices.

Table 4. Categories and Indicators of the Sustainability Dimension.

Category	Keywords	Sustainability Rationale
Low-Carbon Mobility	Subway, light rail, public bus, shuttle bus	Encouraging public transport to reduce the carbon footprint of travel.
Energy Efficiency & Indoor Climate	Air conditioning, heating, thermal comfort, ventilation, air quality	Optimizing HVAC energy use while maintaining high indoor environmental quality.
Resource & Light Environment	Lighting, natural daylight, sunlight, LED fixtures	Leveraging natural light and energy-efficient lighting systems.
Acoustic & Olfactory Health	Noise control, sound insulation, air cleanliness, odor elimination	Ensuring a healthy, non-toxic, and quiet living environment for guests.
Biophilic Environment	Landscape, greenery, scenic views, courtyard, lakefront, eco-trails	Promoting mental well-being and biodiversity through natural integration.
Social Inclusivity	Child-friendly, elderly friendly, parental facilities, multi-generational accessibility	Enhancing social sustainability by catering to vulnerable and diverse age groups.

illustrates the hierarchical structure and keyword logic of the sustainability attribute.

To illustrate the mapping process from unstructured textual data to structured sustainability metrics,

Table 5. Illustrative mapping from raw review phrases to sustainability metrics.

Raw Review Phrase (Translated)	Extracted Keyword(s)	Mapped Sustainability Category	Sentiment Score	Metric Impact
Public bus and subway nearby make travel extremely convenient.	Subway; Bus	Low-Carbon Mobility	0.86 (Positive)	Frequency + 1; Positive sentiment aggregated
The advantage is that it gets plenty of sunlight in the afternoon.	Sunlight	Resource & Light Environment	0.90 (Positive)	Frequency + 1; Positive sentiment aggregated
The hotel’s sound insulation is really poor.	Sound insulation	Acoustic & Olfactory Health	0.22 (Negative)	Frequency + 1; Negative sentiment aggregated
The exhaust fan doesn’t work well, and the odor was not eliminated.	Odor elimination	Acoustic & Olfactory Health	0.23 (Negative)	Frequency + 1; Negative sentiment aggregated

presents real-world examples of how raw review phrases are parsed. It shows how targeted keywords are extracted, assigned to specific sustainability subcategories, and used to generate frequency counts and sentiment scores.

To further validate the robustness and empirical validity of the constructed sustainability measure, we conducted a sanity-check analysis on the dataset. Descriptive statistics show that the average sustainability sentiment score across the 806 hotels is 0.5456 (SD = 0.1275), indicating a generally positive and adequately dispersed distribution of green and inclusive feedback. A Pearson correlation analysis reveals a moderate positive correlation (r = 0.4896, p < 0.001) between a hotel’s sustainability sentiment score and its average user rating. This finding aligns with intuitive expectations: hotels that invest in sustainable practices, indoor comfort, and social inclusivity consistently deliver higher-quality overall guest experiences. These statistics confirm that our text-derived sustainability construct is reliably grounded in real-world user perceptions.

By mapping these keywords to frequency and sentiment vectors, the model captures each hotel’s sustainability performance. This approach ensures that the reinforcement learning strategy delivers recommendations that are both personalized and aligned with the principles of responsible, sustainable tourism.

A critical aspect of this process is the strict enforcement of temporal consistency constraints: for any user check-in record, the corresponding hotel feature vector $f_{h_i}$-including fine-grained sustainability scores based on user-generated content- is generated solely from publicly available review data with timestamps preceding time $t$. Consequently, when constructing samples for training or inference, the system ensures that no information from the current or future time points is utilized, thereby eliminating prospective bias and preventing data leakage.

This design guarantees that the model’s recommendation strategies are grounded exclusively in authentic, available information, thereby enhancing the reliability of offline evaluations and ensuring generalizability and business credibility during online deployment.

• User Sequence Partitioning and Episode Construction

In reinforcement learning-based recommendation systems, transforming raw user behavior logs into training samples that adhere to the Markov Decision Process (MDP) framework is a critical prerequisite for effective model training. Accordingly, this study introduces a rigorous sequential data processing pipeline comprising two core steps: user-level data partitioning and episode construction.

First, the training, validation, and test sets are partitioned at the user level. Specifically, all users are randomly assigned to the training, validation, and testing sets in an 8:1:1 ratio, ensuring strict user isolation—meaning all interaction records for a given user appear exclusively in one set. This approach effectively mitigates evaluation bias arising from cross-set user information leakage, resulting in model performance evaluations that more accurately reflect real-world deployment scenarios.

Subsequently, for each user’s accommodation sequence $H_u=[f_{h_1},f_{h_2},\dots ,f_{h_T}]$—where $h_t$ denotes the hotel stayed at during the $t$-th occasion, with the sequence strictly ordered by ascending timestamp—a sliding window approach is applied to generate multiple training episodes. Each episode represents a complete interaction trajectory, formalized as a sequence of transition tuples: state, action, reward, and next state.

$$[(s_1,a_1,r_1,s_2),(s_2,a_2,r_2,s_3),\dots (s_{T-1},a_{T-1},r_{T-1},s_T)]$$

where the state $s_t$ is generated as described in Section 3.3, integrating the user’s $t$ historical check-in records, current temporal context (such as week and season), and a long-term preference representation to comprehensively capture the observable user state at time t. The action $a_t$ is determined during the offline training phase based on the user’s actual check-in sequence: given state $s_t$, the policy network outputs a probability distribution over hotel recommendations, and the user’s subsequent check-in to hotel $h_{t+1}$ serves as the target action for supervision and reward calculation. The next state, $s_{t+1}$, is dynamically updated to reflect this new check-in behavior. The reward $r_t$ is computed using the composite function defined in Section 3.4, incorporating the characteristics of action $a_t$ and hotel $h_{t+1}$ to produce a comprehensive value reflecting matching degree, semantic similarity, diversity and sustainability alignment. After the user completes their $t+1$-st check-in, the system updates their behavioral memory and context to generate a refreshed state representation.

It is important to emphasize that the entire construction process rigorously adheres to temporal causality: states $s_t$ depend solely on information available up to and including time $t$, actions $a_t$ reflect the actual choice at time $t$, and rewards $r_t$ are calculated using only data available up to and including time $t+1$. This approach fully prevents future information leakage and ensures that every transition tuple is consistent with the temporal logic of real-world interactions.

Through this process, the initially sparse user accommodation logs are converted into structured reinforcement learning training trajectories. This transformation provides high-quality, temporally consistent learning signals for the end-to-end optimization of the Actor–Critic network.

Furthermore, from an implementation perspective, because the maximum sequence length in our dataset is short, we retained each user’s complete historical trajectory during episode construction without truncation or resampling. This approach ensures no information loss for long-term preference modeling.

• Action Space Definition

To balance exploration and computational efficiency, this approach restricts the candidate action space at each step to a manageable subset of hotels. The candidate set is composed of two components: (1) hotels from the user’s historical interactions, and (2) the top 200 most similar hotels from the complete hotel library, identified based on the user’s profile. The user profile is generated by averaging the static feature vectors of the user’s past hotel stays. Using cosine similarity, 200 hotels with which the user has little or no interaction—but which are most similar to the user profile—are selected. The final candidate action space is the union of these two components, resulting in a set slightly larger than 200 hotels (due to the inclusion of previously interacted hotels), yet substantially smaller than the full set of available hotels.

To ensure this action space restriction does not introduce selection bias against sustainable options, we analyzed candidate generation for users with no prior sustainable booking history. Statistical analysis shows that even for these users, the Top-200 pool remains inclusive: at a moderate sustainability threshold (${sent}_h^{sust}>0.5$), sustainable hotels make up 66.66% of the pool; at stricter thresholds of 0.65 and 0.70, they represent 24.64% and 15.32%, respectively. This confirms that candidate generation, based on eight-dimensional comprehensive similarity, does not prematurely exclude sustainable opportunities.

4.1.3. Hyperparameter Setting and Training Process

To ensure experimental reproducibility and rigor, this section details the rationale for hyperparameter selection and the training evaluation process within the proposed DRL-based recommendation framework. All hyperparameters—including learning rate, discount factor, reward weight, and network architecture—are optimized using an independent validation set by maximizing long-term cumulative rewards. This approach ensures both model performance stability and the reliability of results.

• Hyperparameter Settings and Analysis

The hyperparameters within this framework are classified into three categories: model structure parameters, core reinforcement learning parameters, and training optimization configurations.

Table 6. Core hyperparameters of the DRL-based recommendation framework.

Parameter Category	Parameter Name (Symbol)	Value
Model structure parameters	number of features ($n$)	8
	Transformer hide dimensions ($d_{model}$)	128
	LSTM hide state dimension ($d_h$)	64
	Total dimension of state vector ($d_{state}$)	201
	Dimensionality of the query and key vectors ($d_k$)	32
Core parameters of reinforcement learning	discount factor ($\gamma$)	training: 0.95 evaluate: 0.9
	Basic matching reward weight (${\alpha }_m$)	1.0
	Similarity reward weight (${\alpha }_s$)	0.3
	Sustainability reward weight (${\alpha }_{sust}$)	0.15
	Depth adjustment item weight $(\beta )$	0.5
	Diversity penalty coefficient (${\lambda }_d$)	0.1
	Emotional score weight (${\gamma }_s$)	0.3
	Critic loss weight (${\lambda }_c$)	0.3
	Margin ranking loss weight (${\lambda }_{margin}$)	0.2
Training optimization parameters	Actor network learning rate (${\eta }_{\theta }$)	0.0005
	Critic network learning rate (${\eta }_{\omega }$)	0.0002
	Batch size (B)	128
	Entropy regularization coefficient (${\lambda }_e$)	0.005

presents the final values of key hyperparameters and provides a brief explanation of their underlying rationale and design considerations, thereby ensuring methodological transparency and reproducibility.

The hidden dimension ($d_{model}$) of the Transformer is set to 128, the discount factor ($\gamma$) is 0.95, and the diversity penalty coefficient (${\lambda }_d$) is 0.1. These values are chosen to optimize the model’s representational capacity, long-term objective alignment, and the balance between exploration and exploitation, respectively. Comparative experiments demonstrate that $d_{model}$ = 128 offers the optimal trade-off between preference representation accuracy and computational efficiency; a dimension of 64 fails to capture complex relationships, while 256 increases computational overhead without improving performance. A discount factor of $\gamma$ = 0.95 mitigates short-sighted decisions and conservative strategies, supporting the goal of maximizing user satisfaction throughout the entire lifecycle. The diversity penalty coefficient ${\lambda }_d$ = 0.1, determined through offline evaluation, achieves a dynamic balance between leveraging known user preferences and encouraging exploration of potential needs by applying moderate diversity constraints. Furthermore, the sustainability reward weight is deliberately set to a moderate value of 0.15, ensuring that the model effectively promotes environmentally friendly accommodation choices without compromising fundamental recommendation accuracy metrics such as NDCG.

• Training and Evaluation Procedure

This framework adheres strictly to the offline reinforcement learning paradigm, conducting policy learning and evaluation using historical behavior logs without any online interaction with real users. The overall process consists of four distinct stages:

The first stage is imitation learning warm-start. To mitigate the high variance associated with pure policy gradient methods during the cold start phase, behavior cloning (BC) is initially employed to pre-train the policy network ${\pi }_{\theta }$. In this approach, the next hotel actually selected in the user’s historical sequence is treated as an expert demonstration action. An initial policy with fundamental recommendation capability is established by minimizing the cross-entropy loss ($L_{BC}$) between the policy output distribution and the target action.

Following the warm-start phase, the system transitions to the end-to-end reinforcement learning optimization stage: reinforcement learning primary training. During each iteration, counterfactual simulations are conducted on user sequences from the training set: recommendation actions are generated according to the current policy (or rewards are computed using the target action), and transition tuples $(s_t,a_t,r_t,s_{t+1})$ are constructed using the composite reward function defined in Section 3.4. Small batches of data are then randomly sampled to jointly optimize the overall loss function:

$$L_{total}=L_{policy}+{\lambda }_c\cdot L_{critic}+{\lambda }_e\cdot L_{entropy}+{\lambda }_{m\arg in}\cdot L_{m\arg in}$$

Here, ${\lambda }_c$, ${\lambda }_e$, and ${\lambda }_{margin}$ are the weight coefficients for the Critic loss, entropy regularization, and Margin loss, respectively. The individual loss components are formally defined as follows:

(1)

Policy Loss ($L_{policy}$): The policy loss is updated using the advantage-based policy gradient to encourage actions with positive advantages:

$$L_{policy}=-{\displaystyle \frac{1}{B}}{\displaystyle \sum _{t=1}^B\log {\pi }_{\theta }(a_t\left|s_t\right.){\delta }_t}$$

(35)

Here, $B$ is the batch size, ${\delta }_t$ is the temporal difference (TD) error representing the advantage signal, and $a_t$ is the action taken at time $t$ (i.e., the recommended hotel).

(2)

Critic Loss ($L_{critic}$): Fits the temporal difference (TD) target value by optimizing the Value network through minimization of the Mean Squared Error (MSE):

$$L_{critic}={\displaystyle \frac{1}{B}}{\displaystyle \sum _{t=1}^B{(V(s_t;w)-V_{t\arg et})}^2}$$

(36)

(3)

Entropy Regularization ($L_{entropy}$): Adds an entropy regularization term to encourage the policy to maintain exploration and prevent premature convergence:

$$L_{entropy}=-{\displaystyle \frac{1}{B}}{\displaystyle \sum _{t=1}^B{\displaystyle \sum _{a\in C}{\pi }_{\theta }(a\left|s_t\right.)\log }{\pi }_{\theta }(a\left|s_t\right.)}$$

(37)

where $C$ denotes the candidate hotel set for the current decision step.

(4)

Margin Ranking Loss ($L_{margin}$): Enhances discrimination between positive and negative samples by enforcing a margin $m=0.1$ between the Q-values of the user-selected hotel and a randomly sampled unselected hotel:

$$L_{m\arg in}={\displaystyle \frac{1}{B}}{\displaystyle \sum _{t=1}^B\frac{1}{C-1}{\displaystyle \sum _{j=1}^{C-1}\max (0,m-Q(s_t,a_t^{+})+Q(s_t,a_{t,j}^{-}))}}$$

(38)

Here, $a_t^{+}$ denotes the positive action (the hotel selected by the user from the candidate set), and $a_{t,j}^{-}$ denotes the j-th negative action (an unselected hotel from the same candidate set).

The strategy network parameter $\theta$ and value network parameter $w$ are updated synchronously by minimizing the total loss during this process.

The third stage involves model checkpointing and learning rate scheduling. After each training epoch, the current policy is evaluated on the validation set using NDCG@K for model selection and early stopping. The test set is used only once for final reporting. Concurrently, adaptive learning rate scheduling is applied to dynamically adjust the learning rate based on evaluation results, thereby enhancing training stability.

The final stage is final evaluation. All reported performance metrics—including long-term cumulative rewards, NDCG@K, and diversity scores—are calculated on a test user set that is entirely separate from the training and validation processes. This design ensures that the evaluation results objectively reflect the model’s generalization capability and prevent performance inflation due to data leakage.

This process balances training stability, strategy exploration, and evaluation rigor, providing a reliable foundation for the effectiveness verification of the proposed method.

4.2. Analysis of the Validity of User State Representation

In sequential hotel recommendation for OTA platforms, the quality of user state representation is critical to the policy network’s ability to deliver real-time, scenario-adaptive, and sustainability-aware recommendations. Traditional static user profiling methods struggle to distinguish between long-term preferences, such as business travelers’ focus on transportation, and short-term intentions, like family travel needs during holidays. They also lack integration of real-time contextual signals and emerging sustainable consumption trends. These limitations often result in misaligned recommendations, missed opportunities to encourage green travel, and reduced user retention on OTA platforms.

To address these limitations and support sustainable tourism objectives, this study introduces a dynamic state modeling mechanism that constructs a joint state vector $s_t$. Notably, this framework expands the traditional feature representation into an eight-dimensional space, newly incorporating a “Sustainability” dimension to capture hotel performance and user intent regarding eco-friendly, energy-saving, and low-carbon practices. This joint vector dynamically integrates expanded long-term preferences, short-term memory, and temporal contextual information. This approach enables fine-grained, multidimensional characterization of the user’s current state, accurately reflecting both traditional accommodation needs and evolving environmental awareness.

This section empirically evaluates the effectiveness of the enhanced state representation through visualization and quantitative analysis. It demonstrates the model’s capacity to distinguish user groups with significant behavioral differences at the cross-user level—including the identification of cohorts with pronounced sustainability preferences—and to evolve continuously and in a semantically interpretable manner within individual user interaction sequences as behavioral trajectories and contextual factors (such as transitions from weekdays to holidays) change. All analyses are performed on test set users under strict temporal constraints, ensuring that the state evolution path reflects authentic interaction logic and is not influenced by future information leakage.

4.2.1. Comparative Analysis of Long-Term Preference Encoding Effectiveness

To assess the adaptability of preference encoding in OTA platform contexts scenarios, we compared the clustering performance of Transformer output vectors (128 dimensions, as described in Section 3.3, which fundamentally incorporates the 8th dimension of Sustainability) with that of arithmetic mean vectors (27 dimensions, corresponding to the 27-dimensional hotel feature vector in Section 3.2). The Transformer-based approach captures global preference associations, whereas the arithmetic mean aggregates only local features—aligning with the practical user segmentation requirements of OTA platforms.

For the analysis, a standardized process was applied to both types of vectors. First, the K-means algorithm was employed to cluster user vectors in their original space, with the number of clusters predetermined. Next, the t-SNE algorithm was utilized to project the high-dimensional vectors onto a two-dimensional plane—its primary advantage being the preservation of local neighborhood relationships, enabling semantically similar users to cluster naturally in the low-dimensional visualization. The t-SNE projection results were then visualized as scatter plots, with each point representing a test user and colored according to the K-means clustering results. To deepen the analysis, attribute statistics were compiled for representative users in each cluster and their historical hotel stays, including facility preferences and geographical distribution. This allowed for the assignment of clear semantic labels to each clustering structure.

Contour coefficient analysis reveals that the optimal number of clusters is K = 3 for Transformer encoding and K = 2 for mean encoding. This distinction demonstrates that Transformer encoding can more precisely identify user group structures. Its global context modeling capability significantly enhances the granularity and discriminative power of preference representation.

Table 7. Clustering quality of long-term preference representations.

Representation Method	Optimal Number of Clusters K	Silhouette Coefficient	Average Distance Within the Cluster
Transformer encoding	3	0.2899	7.8616
Historical mean encoding	2	0.2044	4.0335

presents the internal evaluation metrics for both long-term preference representation methods at their respective optimal cluster counts. The contour coefficient for Transformer encoding (0.2899) is notably higher than that of historical mean encoding (0.2044), indicating that the Transformer’s non-linear modeling of global interaction sequences via the self-attention mechanism significantly enhances inter-cluster separation. This highlights its ability to accurately capture temporal dependencies and latent correlation structures in user preferences. Although the average intra-cluster distance for Transformer encoding (7.8616) is higher than that for mean encoding (4.0335), this can be attributed not only to the finer granularity of clustering but also to dimensionality effects: the 128-dimensional feature space of Transformer encoding naturally yields higher Euclidean distances, while the 27-dimensional space of mean encoding results in lower values. After controlling for the effect of dimensionality, Transformer encoding still maintains reasonable intra-cluster cohesion and greatly improves inter-cluster discrimination. This underscores its advantage in dynamic weight allocation through the self-attention mechanism, enabling the effective extraction of implicit preference patterns from short sequences and avoiding the limitations of coarse representations and information loss associated with simple linear aggregation.

To maintain consistency in evaluation criteria, the number of clusters was fixed at K = 3 for the comparative analysis of the two preference representation methods.

As illustrated in Figure 4, for the clustering task with K = 3, the three clusters in Figure 4a (Transformer-encoded long-term preference vector t-SNE clustering visualization) display distinct distribution structures, with clear boundaries and strong intra-cluster cohesion. In contrast, Figure 4b (t-SNE clustering visualization of long-term preference vectors from mean encoding) reveals significant overlap between clusters, ambiguous cluster structures, and mixed distribution characteristics. These observations clearly demonstrate the superiority of Transformer encoding in distinguishing long-term preference representations and in structural expressiveness.

The quantitative metrics in this setting further corroborate the visual findings (

Table 8. Cluster quality for long-term preferences (K = 3).

Representation Method	Silhouette Coefficient	Average Distance Within the Cluster
Transformer encoding	0.2899	7.8616
Historical mean encoding	0.1404	3.8337

). Transformer encoding substantially outperforms mean encoding in both contour coefficient values and intra-cluster distances.

Finally, leveraging the cluster semantic tag mining module, representative tags are automatically identified using both the frequency of feature keywords within each cluster’s hotel reviews and their relative importance as determined by TF-IDF. As illustrated in Figure 4, the three user groups display a classic clustering pattern: high intra-cluster density and clear inter-cluster separation. This spatial distribution closely aligns with the semantic preferences indicated by each cluster’s core keywords, effectively validating the effectiveness of clustering based on 128-dimensional long-term preference vectors.

1.

Cluster 1 (Economical and Low-Carbon Transportation Segment)

This cluster is concentrated within t-SNE dimension 1 (ranging from −100 to 0) and dimension 2 (ranging from −50 to 50), forming a high-density region with clearly defined boundaries. Keyword analysis reveals a strong dependence on transportation infrastructure and smart conveniences. The most frequent keywords—”South Station” (489 occurrences), “High-Speed Railway” (197), and “Tourism” (463)—demonstrate a preference for public transit hubs, strongly aligning with low-carbon travel tendencies within the sustainability dimension. Additionally, the high occurrence of keywords such as “Robot” (264), “Takeout” (426), and “Front Desk” (8936) underscores a demand for smart, efficient, and potentially paperless or contactless hotel services. Users in this segment are highly sensitive to location and practical amenities, representing a pragmatic group whose behavioral patterns naturally coincide with sustainable, resource-efficient commuting habits.

2.

Cluster 2 (High-End and Sustainable Quality Segment)

This cluster is distinctly positioned within the projection space, highlighting its unique preferences and strong internal consistency. High-frequency infrastructure terms—such as “Decoration” (857 occurrences), “Style” (368), “Design” (345), and “Space” (395)—reflect a keen pursuit of spatial aesthetics and high accommodation quality. When combined with service and leisure indicators like “Breakfast” (4110), “Upgrade” (876), “Pool” (344), and “Bathtub” (175), these features create a comprehensive semantic profile of a premium experience. Regarding sustainability, this group’s demand for superior “Design” and advanced facilities often aligns with premium hotels that actively implement green building standards, energy-efficient designs, and sustainable hospitality practices. Their isolated grouping in the preference vector space demonstrates a strong correlation between premium experiential demands and sustainable quality.

3.

Cluster 0 (Mid- to High-End Comprehensive Segment)

This cluster is primarily concentrated in the high-density core areas, reflecting a combination of clear core commonalities and diverse individual preferences. High-frequency keywords such as “Golden Eagle” (1068 occurrences) and “Shopping Mall” (452) indicate a strong reliance on established urban commercial districts. From a sustainability perspective, this preference demonstrates a significant demand for urban walkability, allowing users to access dining, shopping, and entertainment venues without the need for additional vehicular transport, thus reducing their travel-related carbon footprint. Additionally, terms like “Space” (495) and “Large Room” (410) underscore the importance placed on accommodation comfort, while “Tourism” (264) and “Attractions” (171) reveal varied travel scenarios. This composite preference structure—integrating walkable locations, accommodation quality, and scenario diversity—strongly aligns with the “Mid- to High-End Comprehensive” user segment.

In summary, t-SNE visualization not only intuitively demonstrates the effectiveness of clustering structures but also validates the accuracy of semantic labels from both qualitative and quantitative perspectives through strong alignment with core keywords. This provides a theoretical foundation that is both interpretable and practical for subsequent user-based hotel recommendations and targeted operational strategies.

This analysis not only validates the semantic integrity of $P_u$, but also establishes an interpretable basis for subsequent strategy networks to discern the long-term intentions—particularly the latent green consumption tendencies—of various user groups. It clearly demonstrates the value of integrating sustainability into preference representation learning, enabling platforms to accurately identify and serve eco-conscious travelers.

4.2.2. Comparative Analysis of Short-Term Memory Mechanism Effectiveness

To assess the adaptability of short-term memory in OTA platform contexts scenarios—for example, when users transition from conventional business travel to eco-friendly accommodations for leisure, or suddenly display a heightened preference for sustainability features due to contextual changes—this section compares state representations with and without LSTM modules. The analysis centers on the model’s capacity to rapidly capture scenario-driven preference shifts, a capability essential for real-time recommendations and the advancement of green consumption on OTA platforms.

A comparative experiment was conducted using a control group and an experimental group: the control group incorporated short-term memory encoded by LSTM within the state vector, while the experimental group excluded this encoding. Both models utilized pre-trained Transformer encoders and identical Actor–Critic network architectures with consistent hyperparameters, ensuring that any observed differences were solely attributable to the presence or absence of short-term memory. First, 109 user sequences exhibiting diverse behavioral patterns (defined by significant changes in accommodation purposes or hotel types within a single sequence) were selected from the test set. Next, representative sequences were chosen, and their state evolution trajectories were projected onto the first principal component via PCA, enabling the compression of high-dimensional dynamic information and the highlighting of key trends. Finally, by comparing the trajectory differences between the two types of visualized states, the modeling role of short-term memory in capturing temporal dependencies in decision-making is intuitively revealed.

As illustrated in Figure 5, at the user behavior switching point (t = 1.00), the complete state curve (which includes the LSTM component) demonstrates a rapidly rising and stable dynamic response. The slope peaks within 0.5 time units after the switch (Δ = 1.82), clearly reflecting the short-term memory mechanism’s ability to instantly capture changes in user intent. In contrast, the memoryless state curve (containing only long-term preferences and contextual information) displays a more gradual upward trend, lasting 1.2 time units with a substantially lower amplitude (Δ = 0.97). This indicates that the absence of temporal modeling results in delayed responses and reduced sensitivity to abrupt user behavior changes. This comparison clearly highlights the critical role of LSTM components in integrating recent interaction information. By dynamically maintaining short-term dependencies within behavioral sequences, recommendation strategies can more accurately adapt to the real-time evolution of user intent, rather than relying solely on static long-term preference representations. This capability is especially valuable for addressing scenarios involving variable user behavior.

To quantify the sensitivity of the state vector to changes in user intent, state responsiveness is defined as the increment in Euclidean distance of the state vector at behavior category switching points (i.e., between adjacent time steps $t-1$ and $t$). A higher value of this metric indicates greater system sensitivity to changes in user intent. Statistical analysis of 363 valid behavior category switching events in the test set (

Table 9. Comparative analysis of state responsiveness.

Status Representation	Average State Responsiveness	Proportion of High Responsiveness Events	NDCG@10
Complete state	2.869	50.0%	0.754
No memory state	2.788	13.9%	0.546

) demonstrates that this metric effectively quantifies the critical role of short-term memory mechanisms in capturing user decision dynamics.

Quantitative analysis in Table 9 indicates that the average state responsiveness of the complete state (2.8698) is slightly higher than that of the memoryless state (2.7887), with a substantially greater proportion of high-responsiveness events (50.00% vs. 13.89%). This confirms that the LSTM mechanism effectively enhances the state vector’s real-time sensitivity to abrupt user behavior changes. At the decision-making level, recommendations based on the complete state in user intention-switching scenarios yield an NDCG@10 value of 0.7539—a 38.18% improvement over the memoryless state (0.5456). These findings highlight that integrating a short-term memory component substantially enhances the recommendation strategy’s ability to respond to users’ evolving needs by accurately capturing temporal dependencies. Within the realm of sustainable OTA platform contexts, this mechanism is especially important: it enables the system to promptly identify and reinforce users’ emerging green consumption intentions. Instead of being limited by historical inertia, the platform can dynamically provide personalized, eco-friendly hotel recommendations precisely when a user’s short-term focus shifts toward sustainability, thereby promoting sustainable choices without compromising overall recommendation accuracy.

4.2.3. Analysis of the Dynamic Evolution Trajectories of State Vectors

An optimal dynamic state representation should encode a user’s long-term identity while responding in real time to interactive behaviors and contextual changes. A valid state representation will exhibit evolutionary trajectories in continuous decision-making that align with behavioral logic and remain interpretable—for example, shifting toward “leisure” when transitioning from weekdays to holidays or reinforcing “efficiency orientation” following repeated business hotel bookings. This analysis empirically evaluates dynamic adaptability by tracking the evolution of typical user state trajectories.

To illustrate this, two users with extended sequence lengths and distinct behavioral patterns are selected. User A is a high-frequency business traveler whose accommodations are primarily concentrated in standardized mid- to high-end hotels in city centers, often for business conferences. In contrast, User B is a mixed-type traveler who alternates between various activities, such as Spring Festival holiday gatherings and weekend short trips, resulting in a broad spectrum of hotel choices. For each user’s complete interaction episode, we first extract their fused state vector $s_t$ at each decision point $t$, dynamically integrating long-term preference $P_u$, short-term memory $z_t$, and temporal context $c_t$. Principal component analysis (PCA) is then applied to project the high-dimensional state sequence onto a two-dimensional plane defined by the first two principal components, enabling visualization of continuous state evolution trajectories and their temporal drift and inflection points. To further enhance interpretability, key contextual information and user behaviors are annotated on the trajectory map, including the time semantics of each moment (e.g., “workday”, “National Day holiday”, “weekend”), the actual hotel $h_{t+1}$ to be checked in at the next time step, and its core attributes (such as the presence of parent–child facilities or location in a scenic area), as illustrated in Figure 6.

As illustrated in Figure 6, User A’s trajectory exhibits a three-stage evolution—initial stability, mid-term transition, and later aggregation—highlighting the synergistic interplay of long-term preferences, short-term memory, and temporal context. During the initial stage, the trajectory remains stable at PC1 = −10 to 0 and PC2 ≈ 0, indicating that long-term preferences anchor the underlying behavioral tendency. At this point, the temporal context is steady, and short-term memory gradually accumulates weekly behavior patterns, resulting in only gradual state changes. In the mid-term stage, PC2 exceeds 30, coinciding with the Spring Festival annotation. Here, abrupt changes in temporal context and instant updates from short-term memory drive a pronounced shift in the state vector along the temporal dimension. PC1 remains within 0–10, suggesting that long-term preferences continue to serve as the evolutionary foundation despite the strong influence of the scenario. In the final stage, PC2 returns to approximately 0, and PC1 converges to 10–20. This shift indicates that short-term memory has consolidated a stable behavioral pattern, while long-term preferences resume dominance in the state distribution, guiding the trajectory toward regions aligned with core user preferences.

User B’s trajectory follows an evolutionary path of initial anchoring, mid-term fluctuations, and eventual convergence, illustrating a shift from scenario-driven to preference-driven behavior. The initial state is anchored at PC1 ≈ −15 and PC2 ≈ −5, integrating Spring Festival context with long-term preferences. During transitions between weekdays and weekends, periodic changes in temporal context modulate short-term memory, causing PC2 to fluctuate between −5 and 0—reflecting dynamic adaptation to weekly behavioral shifts. Meanwhile, PC1 gradually moves toward positive values, signifying the increasing influence of long-term preferences. In the final sequence stage, the trajectory converges to PC1 = 10–15 and PC2 ≈ 0, indicating alignment between short-term memory and long-term preferences and a transition from fluctuation to stable aggregation in state evolution.

User A is classified within Cluster 0 (Mid- to High-End Comprehensive Segment), and their trajectory—characterized by overall aggregation and mid-term transition—closely aligns with the cluster’s hallmark features of clear core commonalities and diverse preferences. The keywords “Golden Eagle” and “Shopping Mall” highlight a reliance on urban commercial districts, which explains the stable aggregation of PC1 resulting from consistent selection of downtown hotels. The multi-scenario demands, indicated by terms such as “scenic spots” and “public transportation”, correspond to the PC2 transitions triggered by family reunions during the Spring Festival. If, following this transition, the selected hotel is indeed family-friendly, it further validates the evolutionary logic of “core anchoring” and “contextual response”.

User B is categorized within Cluster 1 (Economical and Low-Carbon Transportation Segment), and their trajectory—characterized by convergence following fluctuation—reflects the highly homogeneous preference structure of this group. The dependence on transportation hubs, as indicated by keywords such as “South Station”, “High-Speed Railway”, and “Metro”, aligns with the locations of the selected hotels, explaining the evolutionary direction of PC1. Economic and convenience-oriented demands, represented by terms like “cost-effectiveness”, “food delivery”, and “robots”, correspond to the eventual convergence toward hotels offering affordable prices and practical facilities. Despite scene changes between daily and short-distance travel, core preferences remain stable, confirming the high behavioral consistency typical of users in Cluster 1.

The trajectory feature system described above validates the effectiveness of the fused state vector design. PC1, as a projection of long-term preference, consistently constrains state evolution within a limited range, thereby preventing defocusing from short-term disturbances. PC2 reflects the combined influence of temporal context and short-term memory, enabling the state to respond to strong contextual signals—such as the Spring Festival—by triggering transitions, and to transition from fluctuations to stability through behavioral accumulation. Together, these components establish a hierarchical state evolution mechanism characterized by long-term anchoring, scenario-driven modulation, and short-term adaptation.

4.3. Analysis of the Decision-Making Mechanism for the Recommendation Strategy

Interpretability and rational decision-making are essential for recommendation systems to build user trust and enhance long-term satisfaction. Traditional deep recommendation models are often regarded as “black boxes” that cannot satisfactorily answer the fundamental question: “Why was this hotel recommended?” This lack of transparency can result in users feeling disconnected from and unable to control the system. By contrast, the strategy network developed in this study—based on the Actor–Critic framework—naturally achieves greater transparency and interpretability through dynamic feature weight generation and multi-objective composite reward optimization. Dynamic weights reveal the relative importance assigned to different hotel attributes (such as price, location, and facilities) in specific contexts, while composite rewards explicitly encode multidimensional optimization objectives, including matching degree, semantic similarity, diversity, and sustainability incentives. This design ensures that the recommendation logic remains traceable and accountable.

This section examines the core decision-making mechanisms of strategy networks and demonstrates the necessity of three key components—dynamic weight generation, diversity reward guidance, and sustainability-driven reward mechanisms—through internal comparative experiments. The analysis reveals how recommendation systems achieve personalized, state-aware matching while effectively balancing long-term user satisfaction and environmental sustainability goals.

4.3.1. Necessity of Dynamic Weight Generation

Traditional recommendation models typically assign fixed weights to all attributes, making it challenging to accommodate the personalized preferences of different users across diverse scenarios. As a result, recommendations often lack contextual sensitivity. In this study, the Actor network dynamically generates weights based on the current state, allowing the model to adaptively adjust the importance of each attribute according to the user’s real-time behavioral context and achieve genuinely context-aware recommendations. This analysis systematically evaluates whether dynamic weight generation more accurately captures evolving user preferences at different decision points compared to static weight allocation, thereby enhancing the rationality of recommendation decisions and the precision of personalized matching.

Figure 7’s grouped violin plot enables direct, side-by-side comparison of each attribute. The dynamic weight distribution (shown by blue violins) reveals considerable heterogeneity. Key attributes such as “Hospitality”, “User Experience”, and “Sustainability” show significant fluctuations and can reach high weight levels, while other attributes remain concentrated at lower values. The weight for sustainability adjusts dynamically across a broad range, as indicated by its wide density distribution. It peaks for users with strong eco-friendly preferences or in contexts that encourage green consumption. This demonstrates the model’s ability to tailor attribute importance based on personalized contexts.

In contrast, the static weight distribution (shown by flat orange markers next to each blue violin) is highly concentrated and rigid. It maintains consistently low values and lacks attribute discrimination across different scenarios. This disparity results from the dynamic weighting mechanism, which models state–attribute associations in real time using attention networks. This approach allows the model to emphasize key preference dimensions based on the user’s current decision scenario. Static weights, derived from a global mean allocation, cannot accommodate contextual changes and cause recommendations to lose sensitivity to users’ dynamic needs. This comparison highlights the need for a dynamic weighting strategy in effective context-aware recommendations.

Quantitative analysis (

Table 10. Variance comparison of weight distributions.

Strategy	Attribute	Variance
dynamic weight	User Experience	0.018
dynamic weight	Hospitality	0.028
dynamic weight	Sustainability	0.004
Static weight	User Experience	0.000
Static weight	Hospitality	0.000
Static weight	Sustainability	0.000

) demonstrates that the dynamic weighting mechanism delivers pronounced attribute differentiation in decision-making: key attributes consistently receive higher weight allocations, while secondary attributes are assigned lower weights, resulting in distinct attention distributions. Notably, the “Sustainability” attribute exhibits a significant variance of 0.0041 under the dynamic strategy, indicating that the proposed model surpasses the rigid neglect of green features observed in static models (variance of 0) and effectively captures the nuanced environmental demands of various user segments. In contrast, static weights assign nearly equal, low values to all attributes, with distribution variance approaching zero. This difference stems from the dynamic mechanism’s use of a learnable attention function to establish a conditional mapping between states and attributes, enabling the model to accurately identify the core factors driving each decision. In contrast, static weights—relying on a fixed allocation strategy—cannot account for inherent variations in attribute importance, leading to inflexible recommendation logic. These results firmly establish the critical role of dynamic weighting in enhancing the granularity and effectiveness of decision-making.

To demonstrate how the dynamic weight generation mechanism systematically adapts to varying decision contexts on a global scale, we further analyzed the average attribute weights assigned by the model across two different contextual scenarios.

Figure 8 clearly illustrates the systematic differences in user decision-making preferences across two typical scenarios: “holidays” and “workdays”. The heatmap illustrates the dynamic weight distribution across eight hotel attribute categories, where higher weights indicate greater decision importance within the given context. This analysis demonstrates that Actor networks can adaptively adjust their focus based on contextual cues, accurately capturing the core demands specific to each scenario.

In weekday scenarios, user decisions exhibit a pronounced reliance on core experiential and environmental factors. “Sustainability” ranks highest, with a weight of 0.357, indicating that during business or routine travel, users—or corporate travel policies—place significant value on green-certified or eco-friendly hotel practices. This is closely followed by “User Experience” (0.273) and “Hospitality” (0.240), highlighting a strong demand for efficient and responsible accommodations.

In contrast, the focus of decision-making during holidays shifts markedly to address leisure-oriented needs. “Hospitality” rises to a dominant weight of 0.548, underscoring the heightened significance of hospitality, breakfast offerings, and personalized care during vacation periods. Notably, “Sustainability” continues to hold a substantial weight of 0.331 during holidays. This indicates that, while the Actor network dynamically reallocates significant attention to service quality to fulfill leisure-related experiential demands, sustainability remains a consistent and fundamental preference across different contexts. This dynamic adjustment mechanism highlights the model’s situational awareness, enabling it to flexibly reconfigure attribute priorities according to temporal context while steadfastly maintaining the long-term goal of sustainable recommendations.

It is noteworthy that the weight assigned to “Infrastructure” remains relatively consistent between holidays (0.001) and workdays (0.002), indicating that fundamental features such as air conditioning, decor, and space represent baseline requirements across all scenarios. Shifts in the weighting of other attributes clearly reflect the distinct objectives of each context: weekday decisions emphasize efficiency and functional priorities, while holiday decisions focus on experiential needs and elevate the importance of service and leisure attributes. This dynamic adjustment mechanism demonstrates the Actor network’s strong situational awareness, as it flexibly reconfigures attribute priorities according to temporal context, enabling personalized decision modeling that closely aligns with actual user behavior.

4.3.2. Necessity of Diversity and Sustainability Reward Mechanisms

In reinforcement learning-based recommendation systems, a reward function that exclusively targets users’ existing preferences may give rise to a greedy strategy. This often results in repetitive recommendations and content homogenization, while neglecting broader societal values such as environmental responsibility. Consequently, this approach can undermine users’ long-term interests as well as the sustainable development of the platform. To address this issue, a composite reward mechanism is introduced, incorporating a diversity penalty term ($r_t^{diversity}$) and a sustainability reward ($r_t^{sust}$). This method explicitly discourages repetitive recommendation behavior while promoting eco-friendly choices, dynamically balancing the trade-off between leveraging known preferences, exploring novel items, and advancing environmental sustainability.

This section aims to empirically verify whether the multi-objective reward signal effectively guides the recommendation system’s decision-making as intended by analyzing its actual distribution and collaborative dynamics. The complete reward strategy is implemented on the test set, with immediate rewards collected at each step and decomposed into four components: a basic matching reward $r_t^{match}$, a similarity reward $r_t^{\mathrm{s}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}}$, a diversity penalty $r_t^{\mathrm{d}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{y}}$, and a sustainability reward $r_t^{\mathrm{s}\mathrm{u}\mathrm{s}\mathrm{t}}$. The intentional integration of the sustainability reward explicitly directs the policy to prioritize eco-friendly hotels, effectively balancing the trade-off between leveraging known preferences, exploring novel items, and fulfilling environmental responsibilities.

Figure 9 and Figure 10 illustrate the distribution and temporal dynamics of the reward function components on the test set, providing a systematic evaluation of their roles in long-term optimization. The distribution plots (Figure 9) highlight the feedback properties and functions of the four reward components. Matching rewards are largely concentrated at zero, producing non-zero signals only when the recommendation aligns precisely with the user’s actual check-in, demonstrating their sparsity. Thus, matching rewards serve as precise calibration signals at critical points, validating whether the current strategy meets user needs. In contrast, similarity rewards remain consistently high, with most values at 1.0, though the time-series reveals occasional drops. These rewards provide continuous, strong positive feedback, ensuring each recommendation stays closely aligned with the user’s long-term preferences and serving as the primary gradient signal for learning.

Diversity penalties appear as a constant, concentrated distribution at −0.1, acting as a persistent, gentle regulatory mechanism that reduces content homogenization without imposing severe penalties. The sustainability reward serves as a dynamic environmental constraint and incentive, with a broad, mixed-sign distribution. Predominantly positive, with a median around 0.1 and clustering between 0.1 and 0.2, it can also be negative when recommendations have a high environmental impact, as shown by the test set range from −0.3 to 0.6. This component continuously applies reinforcement to encourage eco-friendly options, ensuring the policy is guided toward sustainable choices while maintaining high similarity rewards.

The time series curve from a representative episode (Figure 10) further clarifies the dynamics among the four reward components during interactions. Similarity rewards remain consistently high, with minor fluctuations during exploration and adjustment phases, indicating the system prioritizes preference consistency in decision-making. This results in coherent recommendation behavior and a stable user experience, with rare drops to zero. Instantaneous peaks in matching rewards occur at select points—such as the beginning or midpoint of the episode—and remain at zero otherwise, confirming this sparse but high-value signal reinforces the current decision path. Diversity penalties stay at −0.1 from the outset, suggesting the system introduces subtle recommendation changes (like hotels with slightly different amenities in the same district), reducing redundancy without compromising adaptability. The sustainability reward typically ranges from 0.05 to 0.28, with occasional dips to −0.1, functioning as a dynamic environmental constraint. The varying magnitude of these signals shows the system continuously evaluates each hotel’s sustainability, penalizing poor performers and rewarding good ones. This mechanism guides exploration toward sustainable choices, achieving long-term sustainability goals without sacrificing matching quality.

Together, the four reward components guide the recommendation system’s decision-making by providing stable alignment, precise calibration, moderate adjustment, and dynamic environmental incentives. They keep the strategy aligned with users’ long-term interests, validate accuracy at key points, encourage exploration, and penalize environmentally harmful choices to ensure sustainable options are favored. This design balances short-term user experience with long-term optimization. Relying solely on sparse signals may lead to instability, while optimizing a single objective can cause repetitive recommendations. Integrating all four components, especially the dynamic sustainability reward, allows the system to explore high-quality options. It maintains user adaptability and ecological responsibility. This approach maximizes cumulative rewards for both the platform and the environment.

4.4. Comparative Analysis of Recommendation Performance

To quantitatively assess the recommendation performance of the reinforcement learning framework proposed in this paper, an offline comparative experiment was conducted. The primary objective is to address the core question: Does the DRL-based recommendation framework offer significant advantages in real-time recommendation accuracy and ranking quality compared to classical recommendation and sequential decision models? All experiments were performed using the unified test set and experimental environment detailed in Section 4.1, ensuring fairness and comparability of results.

To provide a comprehensive performance evaluation, three representative baseline methods were selected: (1) Item-based Collaborative Filtering (Item CF), which calculates similarity using item feature vectors; (2) TODIM and VIKOR, multi-criteria decision-making methods that employ the entropy weight method to determine static criterion weights and use the user’s average historical preferences as the ideal point; and (3) GRU4Rec, a deep recommendation model that captures interactive sequences using gated recurrent units. All methods utilize only historical information available up to the current step and adhere to identical candidate pool filtering rules to ensure a fair comparison. Performance is assessed using the Top-10 recommendation list NDCG@10 indicator, which measures ranking quality and reports the average value at each decision point in the test sequence (

Table 11. NDCG@10 Performance Comparison of Recommendation Methods.

Metric	This Research Method	Item-Based CF	TODIM	VIKOR	GRU4Rec
NDCG@10	0.7539	0.6282	0.6392	0.6988	0.6838

).

The NDCG@10 value achieved by the proposed method (0.7539) significantly surpasses all baseline approaches: it represents an improvement of approximately 0.1257 over Item-based CF (0.6282) and about 0.0701 over GRU4Rec (0.6838). To rigorously validate these improvements, we conducted a paired t-test comparing the NDCG@10 scores of our method and the best-performing baseline (VIKOR) across all test sequences. The results confirm that the performance gains are highly significant (p < 0.01). These results confirm the effectiveness of dynamic state perception and sequential decision optimization in enhancing the ranking quality of recommendation lists at a macro level.

A cross-category comparison reveals that among traditional collaborative filtering methods, Item-based CF performs the weakest. This approach relies solely on static item co-occurrence or feature similarity, entirely overlooking the sequential and dynamic nature of user preferences and the importance of decision context, and thus exemplifies a “stateless” recommendation paradigm. The multi-criteria decision-making methods TODIM (0.6392) and VIKOR (0.6988) outperform Item-based CF, demonstrating the utility of multi-attribute trade-offs in leveraging hotel feature information. However, the use of static weights restricts adaptability to diverse users and scenarios—such as the inability to dynamically adjust the importance of sustainability according to context. Additionally, their single-step decision-making framework does not consider the impact of current recommendations on users’ subsequent states and long-term satisfaction. The deep sequential model GRU4Rec (0.6838) is not weaker than static methods, validating the importance of behavioral history modeling for next-interaction prediction. Nevertheless, as a “next prediction” model, it optimizes only for single-step accuracy and does not explicitly account for long-term, multi-step interaction benefits. Additionally, it lacks a multi-objective constraint to balance sequence matching with sustainability objectives. In contrast, the proposed method extends the optimization objective to encompass the long-term cumulative benefits of interaction sequences within a Markov decision process framework, thereby providing a superior global perspective for ranking decisions.

To assess the empirical impact of our proposed sustainability reward, we performed a post hoc comparison against traditional baselines. We calculated two key metrics: (1) the average sustainability sentiment score of the top-10 recommended hotels (${\overline{S}}_{rec}@10$) and (2) the proportion of recommended hotels in the top 20% of sustainability scores ($P_{sust}@10$), reflecting high-tier sustainable options. As shown in

Table 12. Sustainability performance comparison by model.

Metrics	This Research Method	Item-Based CF	TODIM	VIKOR	GRU4Rec
${\overline{S}}_{rec}@10$	0.70	0.58	0.61	0.62	0.49
$P_{sust}@10$	71%	40%	50%	20%	6%

, conventional recommendation models—including Item-based CF, TODIM, VIKOR, and GRU4Rec—tend to recommend hotels with lower average sustainability scores and fewer top-quantile properties. These models typically optimize for static feature similarity or immediate accuracy and cannot balance multi-objective constraints. In contrast, our DRL-based sequential hotel recommendation framework achieves substantially higher ${\overline{S}}_{rec}@10$ and dominant $P_{sust}@10$ values. This comparison shows that the sustainability reward effectively guides the recommendation policy toward environmentally friendly choices, particularly by prioritizing top-tier “sustainability” hotels without post-processing.

In addition to assessing sustainability performance, we evaluated the model’s ability to promote genuine exploration beyond historical recall. Analysis of the final Top-10 recommendation lists shows that about 95% of recommended hotels are new properties not present in the user’s historical interaction set. This high exploration ratio confirms that the RL policy, guided by the diversity penalty ($r_t^{diversity}$), effectively breaks the “information cocoon” and uncovers latent user preferences—an especially important outcome in the low-frequency hotel booking context.

Beyond the impact of the multi-objective reward, the empirical results clarify the specific contributions of the dual-architecture design for user state representation. Models relying mainly on short-term sequential memory (such as the GRU4Rec baseline) achieve an NDCG@10 of 0.6838, while those depending solely on Transformer-based long-term encoding (the “no-memory state” in Section 4.2.2) yield 0.5456. In contrast, the complete dual architecture significantly outperforms both at 0.7539. In hotel recommendations, both components are essential. The Transformer serves as a stable anchor, capturing intrinsic consumption habits (e.g., a consistent preference for high-end amenities or sustainable brands). The LSTM acts as a real-time sensor for situational triggers (e.g., a sudden shift from weekday business trips to weekend family resorts). Without long-term memory, recommendations become short-sighted and overly responsive to recent clicks. Omitting short-term memory results in rigid strategies that fail to adapt to immediate contextual shifts.

The performance comparison above quantitatively affirms the central proposition of this study: compared to static collaborative filtering and multi-criteria decision-making methods, the primary advantage of the proposed approach lies in its dynamic state perception. Relative to GRU4Rec, which offers sequence modeling capabilities, the key strength is the optimization of long-term sequential decisions through a composite multi-objective reward. Collectively, these features—dynamic state perception, sequential decision optimization, and sustainability-aware reward constraints—form the core foundation for this method’s superior performance in complex, dynamic, and personalized recommendation tasks.

4.5. Case Study: Recommendation List Comparison

To illustrate how the proposed reinforcement learning framework balances personalized relevance, diversity, and environmental objectives, we present a micro-level case study. We examine a specific decision step for a representative user in a “Holiday” context. This user has historically shown a strong preference for “Hospitality” (service-oriented) attributes, rather than being strictly eco-driven.

Table 13. Comparison of Top-5 Recommendations for a Specific User Context.

Rank	GRU4Rec Main Tag	GRU4Rec Sust. Score	This Research Main Tag	This Research Sust. Score
1	Sustainability	0.7351	Sustainability	0.7649
2	Hospitality	0.6983	Sustainability	0.8523
3	Hospitality	0.6581	Hospitality	0.6602
4	Hospitality	0.6513	Sustainability	0.7485
5	Sustainability	0.5890	Sustainability	0.7400
Average		0.6664		0.7532

compares the top-5 recommendation lists generated by the strongest baseline, GRU4Rec, and our RL model for the same context.

As shown in Table 13, the baseline GRU4Rec model relies heavily on historical inertia, filling the list primarily with traditional “Hospitality” options (Ranks 2, 3, and 4) to match the user’s past behavior. Although it includes some sustainable features, the overall list is conservative, with an average sustainability score of 0.6664.

In contrast, our RL model overcomes historical inertia while maintaining user relevance. Unlike the black-box GRU4Rec, our Actor network offers transparent interpretability. Analysis of the dynamic weights at this step shows the model adaptively recognized the “Holiday” context and shifted its focus, assigning the highest weights to “Sustainability” (0.4376) and “User Experience” (0.3037), followed by the user’s core preference, “Hospitality” (0.1904).

This weight distribution explains the RL model’s strategic output. Driven by the sustainability reward and dynamic weights, the RL model recommends high-quality eco-friendly hotels, raising the average sustainability score to 0.7532. It maintains relevance by allocating 0.1904 to “Hospitality” and 0.3037 to “User Experience”, ensuring the recommended green hotels deliver exceptional experiences and retaining a premium “Hospitality” option at Rank 3 to meet the user’s core needs.

To quantitatively confirm that the RL model generates a more diverse set of hotels, rather than simply homogenizing the list with “green” tags, we calculated Intra-List Diversity (ILD@5) and History-Novelty@5 using the cosine distances of the hotels’ multi-dimensional feature vectors. Compared to GRU4Rec, our RL model achieves a 16.67% relative improvement in both ILD@5 and History-Novelty@5. This indicates the recommended hotels are more dispersed across attribute dimensions (e.g., price, location, infrastructure) and that the model actively explores high-quality options beyond the user’s history. These results validate the effectiveness of the diversity penalty ($r_t^{diversity}$) in breaking the “information cocoon”, providing a recommendation list that is personalized, diverse, and sustainable.

5. Discussion

Building on the empirical analysis presented in the previous section, this study provides not only a data-driven analytical tool but also a strategic operational framework grounded in dynamic user preferences, specifically tailored for the sustainable tourism ecosystem. The empirical findings reveal that user preferences can be algorithmically segmented into distinct groups, highlighting the necessity of moving beyond traditional positioning based solely on star ratings and price. This approach advocates for precision service allocation that explicitly integrates sustainability performance.

5.1. Operational Strategies for Hotels in Sustainable Tourism

For hotels targeting segments with high sensitivity to basic facilities and transportation, eco-friendly practices should be seamlessly integrated into core services—such as paperless check-in or energy-efficient smart room controls—highlighting these sustainable attributes to enhance competitive advantage. The model’s “dynamic weight” mechanism demonstrates that user decision priorities shift markedly across different scenarios. Accordingly, hotels should develop scenario-adaptive sustainable operational strategies: (1) On weekdays, when efficiency is emphasized, hotels can promote “green business travel packages” (e.g., carbon-offset airport transfers, low-carbon office amenities) to align with algorithmic recommendation criteria. (2) During holidays, when experiential attributes are prioritized, hotels should spotlight eco-friendly leisure activities and organic dining options to meet the rising demand for responsible tourism. Additionally, hotels should leverage real-time data feedback—such as the fluctuating weight of the sustainability attribute—to establish an environmental performance monitoring loop, ensuring that resource investments are aligned with evolving societal expectations for green hospitality.

5.2. Strategic Optimization for Online Travel Agencies (OTAs)

For online travel agencies (OTAs), this study provides advanced algorithms and evaluation frameworks for intent-aware recommendation systems that directly address the core challenge of balancing personalized user satisfaction with promoting sustainable tourism. Experimental results indicate that static models are inadequate for capturing the real-time evolution of user intentions. In contrast, the dynamic state representation mechanism enables platforms to transition from passive matchmakers to proactive advocates for green consumption. Platforms should therefore redesign their optimization frameworks to be explicitly multi-objective, prioritizing “environmental ecosystem health” as a core metric alongside user retention. By leveraging the model’s sustainability reward and diversity penalty mechanisms, platforms can prevent over-optimization for immediate clicks, which often results in homogenized, carbon-intensive travel recommendations. Instead, platforms can gradually and consistently direct traffic toward eco-certified or low-emission properties. Additionally, platforms should utilize the model’s interpretability to strengthen user trust. For instance, transparently displaying recommendation rationales on product pages (e.g., “Recommended for you due to its exemplary green certification and your preference for sustainable travel”) reduces information asymmetry and emphasizes the environmental value of the selection. This strategic approach fosters a mutually beneficial ecosystem, enabling OTAs to drive transaction volume while simultaneously fulfilling their corporate social responsibility (CSR) objectives.

5.3. Empowering Eco-Conscious Consumer Decisions

The findings of this study also empower consumers to make more responsible and autonomous choices within the complex landscape of online booking. The model’s interpretable decision-support features—such as dynamic weight heatmaps that highlight prioritized attributes, including sustainability—enable consumers to shift from passive recipients of algorithmic output to active participants in sustainable tourism. For example, when browsing hotel listings, users can utilize the platform’s “preference focus” prompts to efficiently assess the environmental impact of their selections. By leveraging the model’s state evolution mechanism, OTA platforms can serve as “eco-conscious travel exploration consultants”, fulfilling core functional needs while proactively introducing high-quality, sustainable alternatives to stimulate latent green interests. This approach allows users to discover environmentally friendly accommodations without sacrificing their primary preferences. Ultimately, this strategy not only enhances the efficiency of immediate booking decisions but also cultivates long-term ecological awareness, encouraging consumers to embed sustainability into their ongoing travel behaviors.

6. Conclusions

This study presents a sequential hotel recommendation framework based on deep reinforcement learning that advances beyond previous multi-objective RL systems. Existing RL recommenders often optimize for general long-term value and diversity but frequently fail to capture complex, domain-specific needs, such as those in sustainable tourism. To address this gap and distinguish our work from conventional models, our framework introduces two key contributions. First, it employs a hotel-specific dual architecture that integrates a Transformer and an LSTM to clearly separate and combine long-term preferences with short-term memory. Second, it explicitly extracts multidimensional sustainability metrics from authentic user reviews and embeds them into both the dynamic feature representations and the composite reward system. This approach effectively addresses the limitations of traditional methods in modeling temporal dependencies and promoting sustainable adaptability.

Empirical analysis rigorously validated the effectiveness of each core component within the framework. The Transformer-based encoding and LSTM modules effectively captured the dynamic trajectories of users’ transitions across diverse scenarios. The dynamic weight generation mechanism facilitated personalized situational awareness by accurately reallocating attribute priorities—such as maintaining the significance of sustainability alongside fluctuating service demands—across varying contexts. Notably, the sustainability reward component functioned as a persistent environmental constraint, collaborating with diversity penalties to guide the exploration–exploitation balance toward sustainable options without compromising recommendation accuracy. Comparative experiments revealed that this framework significantly outperformed benchmark methods in NDCG@10 metrics, confirming the superiority of modeling recommendation problems as sustainability-aware sequential decision-making processes.

Theoretically, this study establishes a novel recommendation paradigm that addresses the limitations of generic RL applications by formalizing review-derived sustainability as a core optimizable objective rather than a secondary filter. By showing how a specialized dual-architecture effectively disentangles multi-scale user intents, it bridges the gap between advanced sequential personalization algorithms and sustainable tourism objectives. Practically, it offers a reusable, modular technological pathway that forms the engineering foundation for dynamic platform responses, precise customer segmentation, and the advancement of sustainable consumption in the hospitality industry.

Despite these theoretical and practical contributions, a key limitation remains regarding the generalizability of this study. Because our empirical analysis relies exclusively on data from Ctrip within a single national market, the applicability of the proposed framework to other international OTA platforms is uncertain. Consumer perceptions, sensitivities, and priorities related to “sustainability” may vary significantly across cultural contexts.

Looking ahead, future research can be expanded in several directions. From a data perspective, incorporating more granular environmental metrics—such as real-time carbon footprint tracking, energy consumption reports, or ESG (Environmental, Social, and Governance) ratings—could enable richer sustainability representations. Algorithmically, developing reward-shaping methods grounded in counterfactual reasoning may further optimize the balance between user satisfaction and ecological impact. Integrating robust off-policy correction mechanisms, such as conservative Q-learning or importance sampling, into the offline RL training loop is a crucial next step to address distribution shifts and improve value estimation accuracy on fixed datasets. Additionally, deeply embedding robust interpretability and fairness mechanisms within sustainability objectives is essential for building trustworthy recommendation systems. Evaluating and adapting this framework in diverse international markets to validate its global applicability and examine cross-cultural differences in sustainable tourism behaviors will be an important direction for future research. Advancing these avenues is expected to unlock the full potential of data-driven recommendations in promoting sustainable tourism behaviors, providing robust methodological support for a greener and more intelligent hospitality sector.