Session2vec: Session Modeling with Multi-Instance Learning for Accurate Malicious Web Robot Detection

Abstract

This study addresses the side effect of the rapid development of the Internet, positioning botnets within digital ecosystems as a very serious potential threat to the Internet users. Malicious web robot might facilitate Web/data scraping, DDoS attacks, and data theft yielding serious cybersecurity threats. Modern botnets are advanced and have unique browser fingerprints, making their detection a real challenge. Traditional feature extraction methods heavily depend on expert knowledge. They also struggle with dimensional inconsistency when processing sessions of varying lengths, failing to counter evolving camouflage attacks. To approach such challenges, we propose Session2vec, a session representation framework based on multi-instance learning (MIL), which pioneers the MIL approach for Web session modeling. In this approach, we treat each request as an instance and the entire session as an instance collection, and then we use the FastText model to convert each URL request into a vector representation. Then, we utilize two innovative multi-instance aggregation methods: SARD (Session-level Aggregated Residual Descriptors) and SFAR (Session-level Fisher Aggregated Representation) to aggregate variable-length sessions into fixed-dimensional vectors capturing spatiotemporal features and distributional information within sessions. Simulation results of the proposed SARD and SFAR methods on public datasets show accuracy improvement of 5.2% and 16.3% on average, respectively, compared to state-of-the-art baselines. They also enhance F1 scores by 8.5% and 19.7%, respectively.

1. Introduction

Web robots or crawlers have become a major source of network traffic. While some robots (such as those used by search engines) behave well, other robots may perform DDoS [1,2] attacks and carry out low-rate threats [3], posing a serious security threat to websites. Web robots (also known as “Web crawlers”), as a key component supporting data collection and indexing in the modern Internet, are technologically neutral and are thus widely used in building search engine indexes, scanning for network vulnerabilities, and archiving Web content [4,5,6]. According to the latest research [7], for Web applications with known vulnerabilities, 47.4% of the network traffic is Web robot traffic, with malicious traffic accounting for as much as 63.57%. This trend highlights the urgency of strengthening cybersecurity defenses against such network security events.

Assessing whether a website bot harbors malicious intent is often challenging; it is relatively straightforward to identify such bots only when they display clear danger signals (e.g., frequent access to restricted resources within a short time span or initiating numerous connection requests). However, more covert malicious bots tend to evade detection by mimicking human behavior, spoofing trusted IP addresses, or adhering to typical website navigation structures, thereby posing potential threats to both the ethical and security dimensions of websites and their hosted information [8]. Currently, the detection of Web robots primarily relies on Web log data to identify and intercept potential malicious activities [7,9,10,11,12]. Consequently, extracting effective and comprehensive data representations from Web logs is crucial for reliable Web robot detection. Because these detection models heavily depend on how log data are represented, a lack of high-quality feature extraction and representation can significantly hinder the accurate differentiation between legitimate and malicious access.

Despite notable progress in Web bot detection research, significant limitations persist in feature representation generalizability and temporal modeling capabilities. For instance, methods based on static statistical features (e.g., request frequency and error rate) achieve high detection precision (98.7%) in specific scenarios [13], yet their reliance on manually defined feature sets fails to capture the temporal semantics of request sequences, resulting in degraded cross-platform generalization performance. To address this limitation, subsequent studies have attempted to integrate multi-source heterogeneous data, such as combining mouse dynamics with request metadata to enhance detection robustness [12]. However, these approaches heavily depend on manual feature engineering and prior knowledge, making it difficult to handle diverse attack patterns. Recent real-time detection frameworks leveraging multimodal behavioral analysis [14] have improved temporal modeling by defining 43 browser event features for LSTM classifiers. Nevertheless, their feature design heavily depends on prior knowledge of HTTP protocols, struggles to adapt to emerging communication protocols like WebSocket, and incurs substantial deployment costs due to the requirement for fully annotated event sequences. Notably, semantic-enhanced methods (e.g., LDA2Vec topic feature extraction) that combine Web logs with content data have achieved an F1-score of 96.58% in e-commerce contexts [7]. However, their session characterization remains confined to static topic distributions and handcrafted statistics, failing to model dynamic contextual relationships in request sequences, which creates detection blind spots for temporally camouflaged bots.

To avoid reliance on manual feature engineering and to better capture cross-request spatiotemporal relationships, this paper proposes an end-to-end session representation method based on multi-instance learning (MIL), termed Session2vec. Under the MIL paradigm [15], each session is regarded as a bag of multiple request instances, and a comprehensive, scalable session vector representation is constructed through character-level subword embeddings and spatiotemporal clustering. Specifically, we first employ FastText to embed each request URL into a vector. Subsequently, drawing inspiration from the classic VLAD (Vector of Locally Aggregated Descriptors) and Fisher Vector algorithms [16], we propose two innovative aggregation strategies: SARD and SFAR. SARD extracts spatiotemporal relationships from clustering residuals based on hard assignments, while SFAR captures richer distributional features using soft assignments based on Gaussian Mixture Models (GMMs) [17]. Compared to traditional manual statistical features or sequence analysis methods, Session2vec focuses more on spatiotemporal associations among requests and the concealed distributional characteristics that malicious activities may exhibit, thereby significantly enhancing Web robot detection performance and robustness. Our five-fold cross-validation results demonstrate that this method achieves optimal performance in terms of both accuracy and F1-score.

In the field of network and information security, multi-instance learning has been widely applied to enhance the representation of unstructured data, such as text and logs. For example, in the threat behavior extraction framework SeqMask, cyberthreat intelligence (CTI) texts are treated as bags of behavioral phrase instances, and a Mask Attention mechanism is employed to mine key behavioral terms, effectively revealing the adversaries’ strategies, tactics, and procedures (TTPs) and achieving excellent classification performance in distant supervision scenarios [18]. This work demonstrates that under weakly supervised or semi-supervised conditions, semantic aggregation and attention filtering via multi-instance learning can enhance the extraction of critical information while reducing labeling costs. Inspired by this idea, Session2vec extends the multi-instance learning framework to dynamic session behavior modeling: it not only focuses on the content or features of individual requests but also captures the latent associative structures among requests through spatiotemporal distribution modeling, thereby effectively countering complex camouflage attacks.

Furthermore, the MIL-based text representation model BOS proposed by He et al. segments texts into sentence-level instances and employs sentence similarity measures along with an improved KNN algorithm to perform document classification [19]. This method overcomes the bag-of-words independence assumption inherent in conventional vector space models (VSMs) [20], thereby preserving to some extent the internal semantic structure of texts. With regard to semantic structure modeling, the syntax-aware entity embedding model developed by He et al. [19] employs tree-structured neural networks (e.g., Tree-GRU/Tree-LSTM) to represent entity contexts and leverages both sentence-level and entity-level attention mechanisms to enhance the accuracy of distant supervision relation extraction [21]. Their work indicates that incorporating richer hierarchical structures (such as syntactic trees or graph structures) can provide more refined contextual information for multi-instance learning in complex texts. However, in Web security detection scenarios, request patterns are typically more dynamic and diverse—randomized URL paths, adversarial request fingerprints, and significant quantities of noisy data render static tree structures unsuitable. For this reason, Session2vec does not rely on predefined syntactic trees; rather, it employs FastText [22] subword embeddings to capture the character-level semantics and structure of request paths, and then applies the SARD and SFAR algorithms to aggregate the entire session sequence, thereby achieving a weakly supervised representation of variable-length request streams.

The main contributions of this work are as follows:

• Session representation innovation within the multi-instance learning framework:This research pioneers the systematic application of the MIL paradigm to Web session representation, treating each request as a semantic instance and the entire session as an instance collection. By proposing the SARD algorithm to perform clustering residual encoding on FastText-generated request vectors, we effectively model spatiotemporal behavior patterns within sessions. This approach significantly improves detection accuracy for complex camouflage attacks and provides a novel perspective for session representation in the Web security domain.
• Session vectorization and end-to-end detection framework: Traditional Web robot detection typically relies on handcrafted features or synthetic data, which struggle to comprehensively capture real attack scenarios. Session2vec implements cross-request end-to-end detection through session-level vectorization, reducing over-reliance on manual features while naturally integrating contextual information to improve overall malicious behavior recognition. The proposed SARD and SFAR aggregation strategies enable the model to adapt to variable-length sessions while preserving temporal relationships between requests.
• Unsupervised embedding and dual aggregation for spatiotemporal modeling: Session2vec addresses the challenges of high annotation costs and data diversity in network security tasks. It combines unsupervised FastText subword embeddings with two complementary aggregation strategies, SARD and SFAR, to achieve efficient spatiotemporal feature capture.

The remainder of this paper is organized as follows: In Section 2, we review the related work on Web robot detection and session modeling. Section 3 formulates the problem within a multi-instance learning framework. In Section 4, we detail our proposed approach, including the request embedding process and the design of the SARD and SFAR aggregation methods for session-level representation. Section 5 presents the experimental setup, dataset statistics, and performance evaluation. Section 6 concludes the paper and outlines potential avenues for future research. Finally, Section 7 discusses future work.

2. Related Work

Recent advancements in Web robot detection have explored diverse methodologies to address the evolving challenges posed by malicious bots. Traditional approaches, such as rule-based systems and classical machine learning algorithms (e.g., KNN, decision trees, and neural networks), have shown limitations in handling variable-length session data and inconsistent feature dimensions, particularly when bots mimic human-like behaviors or employ sophisticated evasion tactics [13]. For instance, while neural networks achieve high precision in bot detection, their computational overhead and reliance on fixed-dimensional feature vectors hinder scalability.

Semantic-based detection methods leverage content and behavioral patterns to distinguish bots. Studies like [7,23] utilize semantic features from Web content or logs, assuming human users exhibit topic-specific interests. However, these methods often neglect the temporal and contextual relationships within session-level requests, limiting their ability to model complex bot behaviors. Similarly, dynamic metadata approaches (e.g., mouse dynamics [12]) enhance accuracy but focus on biometric traits rather than session-structured log analysis, which is critical for detecting bots operating at scale.

Semi-supervised learning has emerged to address labeled data scarcity. For example, Web-S4AE [11] employs a stacked sparse autoencoder to exploit unlabeled data, yet its dependency on content–log hybrid features may not fully resolve the challenges of variable request counts within sessions. In contrast, Session2vec introduces a multi-instance learning (MIL) framework coupled with FastText [22] and miVLAD and miFV algorithms [16]. This approach uniquely converts individual requests into fixed-dimensional vectors (via FastText) and aggregates them into session-level representations (via miVLAD and miFV), effectively addressing feature inconsistency while capturing temporal and contextual dependencies. Unlike the study [11], which requires extensive labeled data, MIL reduces annotation costs by treating sessions as bags of instances, aligning with the trend toward weak supervision in bot detection.

Furthermore, while real-time detection methods [24,25] prioritize early classification of active sessions, they often overlook the structural heterogeneity of session data. Session2vec bridges this gap by unifying session modeling and representation learning, offering a robust solution for detecting advanced bots with dynamic behavioral patterns.

3. Problem Formulation

In Web behavior analysis, each session can be treated as a “multi-instance” bag composed of multiple requests. Traditional single-instance approaches often focus on the features of individual requests while neglecting the broader temporal and semantic patterns within a session, making it difficult to detect sophisticated malicious bot activities. To address this issue, we model session-level detection as a multi-instance learning problem.

Formally, consider a Web log dataset containing multiple sessions $\{S_1,S_2,\dots ,S_m\}$. Each session $S_i$ comprises requests $\{r_{i1},r_{i2},\dots ,r_{in}\}$ and carries a single label (e.g., “benign” or “malicious”). In the multi-instance setting, each request $r_{ij}$ is treated as an instance, whereas each session $S_i$ is regarded as a bag. Our objective is to make a session-level classification decision to distinguish between human and bot traffic.

Under this framework, we first embed each request into a vector representation. We then deploy an aggregation step that converts all instance vectors within a bag into a fixed-dimensional session vector, which can be fed into supervised or unsupervised methods to uncover potential malicious behaviors. Because the number of requests (instances) per session (bag) can vary, multi-instance learning naturally accommodates this variability, making it highly suitable for our problem setting.

4. Methodology

In this study, we propose a structured framework for Web robot detection, illustrated in Figure 1. First, both human and robot users generate significant numbers of Web logs by interacting with a server. We preprocess these logs, extracting relevant features such as the request URL field. After data filtering and feature extraction, the Session2vec approach is employed to obtain session-level representations.

The representation process involves two stages: (1) Converting each request URL into a fixed-dimensional embedding using a FastText model; (2) Treating all requests within a session as a multi-instance bag, which is subsequently aggregated into a session-level vector through two methods: SARD and SFAR. The resulting vectors can then be fed into a classifier to differentiate between human and robot activities.

4.1. Request Embedding

In this study, we first perform text preprocessing and vectorization on each raw data request to construct a session-level representation. Specifically, the URL path in each request is preprocessed by replacing the forward slash (“/”) with a space, thereby generating a continuous text string. We then apply a pretrained FastText-300 model to convert the text into a 300-dimensional vector representation. Since the model is trained on large-scale corpora, it effectively mitigates issues such as spelling errors and out-of-vocabulary words. The complete process of request embedding generation is detailed in Algorithm 1.

For a single session S, multiple requests are contained within. By stacking the vector of each request in the original chronological order, we obtain an $n\times 300$ matrix (n is the number of requests), as shown in Algorithm 1. This matrix preserves the local semantic features of each request and simultaneously captures the temporal relationships within the session, laying the foundation for detecting complex behavior patterns.

The following pseudocode illustrates how to build request embeddings using a pretrained FastText-300 model and stack them into matrices grouped by session.

Algorithm 1 Request embedding generation

Require:  Dataset D, where each record contains the fields $\mathtt{session}\mathtt{id}$ and $\mathtt{request}\mathtt{path}$; pretrained FastText model M (with 300-dimensional output). Ensure:  A vector representation for each request and a request matrix $E\in {\mathbb{R}}^{n\times 300}$ grouped by session.   1:   for each record $r\in D$ do   2:      Extract the request path: $\mathrm{uri}\leftarrow r.\mathtt{request}\_\mathtt{path}$   3:      Preprocess uri: replace the forward slash with a space to obtain the string $\mathtt{processed}\_\mathtt{uri}$   4:      Use the pretrained FastText model to compute the sentence vector: $v\leftarrow M.\mathtt{get}\_\mathtt{sentence}\_\mathtt{vector}(\mathtt{processed}\_\mathtt{uri})$   5:      Save the vector v as a new field for the record (e.g., $\mathtt{uri}\_\mathtt{vector}$)   6:   end for   7:   for each session s (grouped by $\mathtt{session}\_\mathtt{id}$) do   8:      Collect all request vectors for session s: $\{v_1,v_2,\dots ,v_n\}$   9:      Stack these vectors using $\mathtt{np}.\mathtt{vstack}$ to form the matrix $E_s\in {\mathbb{R}}^{n\times 300}$ 10:   end for 11:   Return the collection of request matrices $\left\{E_s\right\}$ corresponding to all sessions

4.2. Session-Level Representations: SARD and SFAR

In this section, we propose two innovative session-level aggregation methods, SARD and SFAR, to build fixed-dimensional session representations from request embeddings. Both methods follow a multi-instance aggregation principle, effectively deriving statistical and distributional features from session requests while maintaining a fixed output dimension.

4.2.1. SARD (Session-Level Aggregated Residual Descriptors)

SARD is inspired by the classical VLAD approach. It utilizes hard assignment to cluster each request vector and accumulates their residuals as the final representation. Let

$$E_s\in {\mathbb{R}}^{n\times d}$$

be the request embedding matrix of session s, where n is the number of requests and d is the embedding dimension (e.g., 300). Assume there is a pretrained KMeans [26] model C with K centroids

$$\{c_1,c_2,\dots ,c_K\},c_k\in {\mathbb{R}}^d.$$

For each request vector $x_i$ in $E_s$, we identify its nearest centroid index $k_i^{*}$ via

$$k_i^{*}=\underset{k}{argmin}{\parallel x_i-c_k\parallel }^2.$$

We then accumulate the residual into the matrix $V\in {\mathbb{R}}^{K\times d}$:

$$V\left[k_i^{*}\right]\leftarrow V\left[k_i^{*}\right]+(x_i-c_{k_i^{*}}).$$

By row-wise flattening V, we obtain

$$v_{\mathrm{sard}}\in {\mathbb{R}}^{K·d}.$$

Finally, we apply power normalization and $L_2$ normalization:

$$v\leftarrow sign\left(v\right)\sqrt{\left|v\right|},v_{\mathrm{sard}}\leftarrow {\displaystyle \frac{v_{\mathrm{sard}}}{\parallel v_{\mathrm{sard}}{\parallel }_2}}.$$

Algorithm 2 illustrates the complete SARD procedure.

Algorithm 2 Session embedding—SARD generation

Require:  Set of request embeddings $\left\{E_s\right\}$, where $E_s\in {\mathbb{R}}^{n_s\times d}$; number of clusters K; pretrained KMeans model C. Ensure:  Fixed-dimensional session representation $v_{\mathrm{sard}}\in {\mathbb{R}}^{K·d}$ for each session.   1:   for the matrix $E_s$ for each session s do   2:      Initialize the residual matrix $V\leftarrow \mathtt{zeros}(K,d)$   3:      for each request vector $x\in E_s$ do   4:         $k^{*}\leftarrow C.\mathtt{predict}\left(x\right)$   5:         $V\left[k^{*}\right]\leftarrow V\left[k^{*}\right]+(x-C.\mathtt{cluster}\_\mathtt{centers}\left[k^{*}\right])$   6:      end for   7:      Flatten V to obtain $v_{\mathrm{sard}}\in {\mathbb{R}}^{K·d}$   8:      Apply power normalization and $L_2$ normalization   9:      Save $v_{\mathrm{sard}}$ as the final representation of session s 10:   end for 11:   Return $\left\{v_{\mathrm{sard}}\right\}$

4.2.2. SFAR (Session-Level Fisher Aggregated Representation)

Unlike the hard assignment in SARD, SFAR leverages a Gaussian Mixture Model (GMM) to capture the underlying probability distribution of request embeddings. The GMM is parameterized by

$$\lambda ={\{{\pi }_k,{\mu }_k,{\Sigma }_k\}}_{k=1}^K,$$

where ${\pi }_k$ denotes the prior probability of the kth component, and ${\mu }_k$ and ${\Sigma }_k$ are its mean and covariance (often diagonal). For each request vector $x_i$, we compute the posterior probability:

$${\gamma }_k\left(x_i\right)={\displaystyle \frac{{\pi }_k\mathcal{N}(x_i\mid {\mu }_k,{\Sigma }_k)}{{\sum }_{j=1}^K{\pi }_j\mathcal{N}(x_i\mid {\mu }_j,{\Sigma }_j)}}.$$

We then calculate the first-order and second-order gradients with respect to the GMM parameters:

$${\nabla }_{{\mu }_k}={\displaystyle \frac{1}{n_s\sqrt{{\pi }_k}}}\sum _{i=1}^{n_s}{\gamma }_k\left(x_i\right)\left({\displaystyle \frac{x_i-{\mu }_k}{{\sigma }_k}}\right),{\nabla }_{{\sigma }_k}={\displaystyle \frac{1}{n_s\sqrt{2{\pi }_k}}}\sum _{i=1}^{n_s}{\gamma }_k\left(x_i\right)\left[{\displaystyle \frac{{(x_i-{\mu }_k)}^2}{{\sigma }_k^2}}-1\right].$$

All these gradients are concatenated to form the Fisher vector

$$v_{\mathrm{sfar}}\in {\mathbb{R}}^{2K·d},$$

which is then power-normalized and $L_2$-normalized to enhance robustness and comparability. Algorithm 3 provides the complete pseudocode for the SFAR approach.

Algorithm 3 Session embedding—SFAR generation

Require:  Set of request embeddings $\left\{E_s\right\}$, where $E_s\in {\mathbb{R}}^{n_s\times d}$; number of Gaussian components K; pretrained GMM model G. Ensure:  Fixed-dimensional session representation $v_{\mathrm{sfar}}\in {\mathbb{R}}^{2K·d}$ for each session.   1:   for the matrix $E_s$ for each session s do   2:      Initialize the Fisher vector $v_{\mathrm{sfar}}\leftarrow \mathtt{zeros}(2K·d)$   3:      for each request vector $x\in E_s$ do   4:        Compute posterior probabilities ${\gamma }_k\left(x\right)$ for all components k   5:        Compute gradients ${\nabla }_{{\mu }_k}$ and ${\nabla }_{{\sigma }_k}$ and accumulate them in $v_{\mathrm{sfar}}$   6:      end for   7:      Apply power normalization and $L_2$ normalization   8:      Save $v_{\mathrm{sfar}}$ as the final representation of session s   9:   end for 10:   Return $\left\{v_{\mathrm{sfar}}\right\}$

5. Experiments

5.1. Experiment Setup

In this study, we used a high-performance computing system with two NVIDIA RTX A6000 GPUs (48 GB memory each) and 80 virtual CPUs powered by an Intel Xeon Platinum 8383C processor (2.70 GHz). The experiments ran on Ubuntu 24.04.1, with Python 3.8.8 as the main programming language. Key Python libraries included Numpy 1.20.1, Pandas 1.2.4, XGBoost 2.1.2, Fasttext 0.9.2, and Scikit-Learn 0.24.1 [27]. This setup enhanced computational efficiency and improved model evaluation.

5.2. Datasets

Table 1. Session and request statistics for each fold in the dataset.

Fold	Total Sessions	Total Requests	Avg. Requests/Session
1	186	118,172	635.33
2	188	118,883	632.36
3	188	117,270	623.78
4	190	124,466	655.08
5	190	122,862	646.64
Average	188.40	120,330.60	638.64

presents the basic statistics of the five-fold cross-validation dataset. We used the dataset from [8], which was established as a multimodal detection benchmark comprising session-level Web logs and mouse trajectory data. Its original architecture aimed to improve adversarial bot recognition by fusing spatiotemporal features. The Web logs recorded HTTP request types (GET/POST), resource categories (HTML/CSS/image files), status codes, and timestamps, which together captured visitors’ temporal behavioral patterns (e.g., request frequency, navigation paths). Meanwhile, the mouse trajectory data, collected through embedded frontend scripts, captured spatial interaction features (e.g., movement speed, click coordinates). Because mouse data collection is often limited in real-world scenarios, we focused on single-modal Web log analysis. We retained semantic features in request sequences (e.g., resource-type preferences, error-status distribution) and behavioral patterns (e.g., high-frequency access, abnormal jumps) to examine the viability of a lightweight detection framework.

In this study, we merged data from Phase 1 and Phase 2 and divided them into training and testing sets at a 7:3 ratio, consistent with the method used by [8] for comparison with their statistical feature representation approach, while ensuring that each category (human, advanced bot, and moderate bot) maintained the same proportion in both sets. As shown in the table, the merged dataset contained an average of 188.40 sessions, with the number of sessions per fold ranging from 186 to 190. The total number of requests averaged 120,330.60, ranging from 117,270 to 124,466. Notably, each session contained an average of 638.64 requests, indicating highly intensive session interactions.

Figure 2 illustrates the distribution of requests per session across the five folds of the cross-validation dataset. The boxplots revealed relatively consistent request distribution patterns across folds, with median values of approximately 600 requests per session, indicating frequent and intensive session interactions. Notably, all folds contained outliers representing sessions with exceptionally high request counts, potentially indicating particularly complex interaction patterns or prolonged sessions. This consistency demonstrated the appropriateness of the data partitioning and indicated stability in session behavior patterns throughout the dataset. This characteristic is especially important for building robust detection models, as it ensures that training and testing data have similar statistical properties, thereby enhancing the model’s generalization capability.

5.3. Comparison Method

In this study, we compared the performance of the proposed Session2vec method with traditional session representation methods based on statistical features [8]. These statistical features are commonly used in Web robot detection tasks and can describe various aspects of a session, such as the number of requests, response status codes, and resource types requested. However, these methods heavily rely on manually defined statistical features, which may fail to capture the complex and dynamic behaviors of bots, especially when the bots mimic human interactions, where traditional methods perform poorly. In contrast, the Session2vec method, through its multi-instance learning framework and subword embeddings, better captures the temporal and behavioral dynamics of Web sessions, overcoming these limitations. The statistical features used for comparison are shown in

Table 2. Statistical features for session representation [8].

Feature	Short Description and Literature
session_id	Unique identifier for each session, distinguishing different sessions.
user_id	Unique identifier for the user, extracted from the training data.
total_requests	Total number of HTTP requests made during the session [5,28,29,30,31,32,33].
total_bytes	Total size of the transferred data during the session (in bytes) [5,10,24,28,30,33].
get_requests	Number of HTTP GET requests made in the session [24,30,31,33,34,35].
post_requests	Number of HTTP POST requests made in the session [24,30,31,33,35].
head_requests	Number of HTTP HEAD requests made in the session [5,10,24,29,30,31,32,33,35].
%_http_3xx	Proportion of requests with HTTP 3xx status codes (redirection) [24,30,33,34].
%_http_4xx	Proportion of requests with HTTP 4xx status codes (client errors) [5,10,24,29,30,32,33,34].
%_image_requests	Proportion of image requests in the session [30,31,36,37].
%_css_requests	Proportion of CSS requests in the session [30,37].
%_js_requests	Proportion of JavaScript requests in the session [30,36,37].
html_to_image_ratio	Ratio of HTML requests to image requests [29,30,32,33].
depth_sd	Standard deviation of URL path depth, reflecting the diversity in page access [5,30,32,33].
max_requests_per_page	Maximum number of requests to a single page.
avg_requests_per_page	Average number of requests to each page in the session [30].
max_consecutive_sequential	Maximum number of consecutive requests with a parent–child URL relationship [30,33].
%_consecutive_sequential	Proportion of consecutive requests with a parent–child URL relationship [5,29,30,32].
session_time	Duration of the session from the first to the last request (in seconds) [10,28,30,31,33,34].
browsing_speed	Rate of page requests per second during the session [30,34].
sd_inter_request_times	Standard deviation of the time intervals between consecutive requests [30,34].

.

These statistical features are frequently used in the existing literature for Web robot detection. However, these methods are often limited in their ability to generalize to complex and dynamic attack behaviors. They fail to model the temporal and spatiotemporal relationships between requests, which are crucial for detecting sophisticated bots. To address this, Session2vec incorporates a multi-instance learning framework with FastText subword embeddings and the miVLAD algorithm to aggregate session-level information into a fixed-length vector representation. This approach not only resolves inconsistencies in request counts but also captures the temporal dynamics of the session, providing a more robust and scalable solution for bot detection.

To evaluate the performance of Session2vec in comparison to statistical feature-based methods, we applied several machine learning models: Random Forest [38], Decision Trees [39], MLP [40], XGBoost [41], and KNN [42]. The models were evaluated using two key performance metrics: accuracyand F1-score. These metrics allow for a thorough evaluation of classification performance, accounting for both the overall accuracy and the balance between precision and recall, which is essential for detecting imbalanced bot detection tasks. The results demonstrated that Session2vec outperformed traditional methods, providing superior detection accuracy and robustness against sophisticated camouflage attacks.

5.4. Performance Metrics

In our experiments, we evaluated the performance of our methods using four key metrics: accuracy (Acc), precision (P), recall (R), and the F1-score (F1). These metrics provided a comprehensive analysis of the model’s performance by assessing its correctness, completeness, and balance. Below are the detailed definitions and formulas for each metric:

• True positives (TP): the number of instances correctly classified as positive by the model.
• False positives (FP): the number of instances incorrectly classified as positive by the model.
• False negatives (FN): the number of instances incorrectly classified as negative by the model.
• True negatives (TN): the number of instances correctly classified as negative by the model.

Accuracy ($Acc$): Accuracy measures the overall correctness of the model by computing the ratio of correctly classified instances to the total number of instances. The formula is:

$$Acc={\displaystyle \frac{\sum \mathrm{TP}+\sum \mathrm{TN}}{\sum \mathrm{TP}+\sum \mathrm{FP}+\sum \mathrm{FN}+\sum \mathrm{TN}}}$$

Precision (P): Precision quantifies the proportion of correctly predicted positive instances out of all predicted positive instances. The formula is:

$$P={\displaystyle \frac{\sum \mathrm{TP}}{\sum \mathrm{TP}+\sum \mathrm{FP}}}$$

Recall (R): Recall measures the model’s ability to identify all actual positive instances. The formula is:

$$R={\displaystyle \frac{\sum \mathrm{TP}}{\sum \mathrm{TP}+\sum \mathrm{FN}}}$$

F1-score (F1): The F1-score combines precision and recall into a single metric by computing their harmonic mean. It is particularly useful when dealing with imbalanced datasets. The formula is:

$$F1=2·{\displaystyle \frac{P\times R}{P+R}}$$

By incorporating these metrics, we ensured a holistic evaluation of the model’s performance, balancing its accuracy, precision, and recall, while addressing potential class imbalances.

5.5. Experiment Results

To evaluate the effectiveness of our proposed methods, we conducted extensive experiments comparing the performance of SARD and SFAR with baseline approaches across multiple classification algorithms.

5.5.1. Cross-Fold Validation Performance Analysis

To evaluate the stability and generalization capability of the proposed methods, we conducted a 5-fold cross-validation comparison among three approaches: the baseline method, SARD, and SFAR. For each fold, the performance was averaged over five commonly used classifiers: Random Forest, Decision Trees, MLP, XGBoost, and KNN. This averaging was designed to highlight the robustness and general applicability of the proposed methods across different learning algorithms. Figure 3 illustrates the performance of these three methods across different folds in terms of precision, recall, F1-score, and accuracy. It is evident from the figure that our proposed SFAR method consistently outperformed both the baseline and SARD methods across all evaluation metrics. In particular, in terms of precision, the SFAR method achieved an average improvement of approximately 24.6% compared to the baseline.

Table 3. Performance comparison across folds averaged over Random Forest, Decision Trees, MLP, XGBoost, and KNN (The best results are in bold).

Fold	Metric	Baseline	SARD (Ours)	SFAR (Ours)
1	Precision	0.603	0.724	0.829
1	Recall	0.637	0.717	0.807
1	F1-score	0.596	0.717	0.810
1	Accuracy	0.637	0.717	0.807
2	Precision	0.687	0.731	0.873
2	Recall	0.728	0.724	0.872
2	F1-score	0.692	0.724	0.869
2	Accuracy	0.728	0.724	0.872
3	Precision	0.615	0.734	0.864
3	Recall	0.650	0.732	0.861
3	F1-score	0.616	0.728	0.860
3	Accuracy	0.650	0.732	0.861
4	Precision	0.652	0.737	0.847
4	Recall	0.695	0.737	0.842
4	F1-score	0.662	0.727	0.839
4	Accuracy	0.695	0.737	0.842
5	Precision	0.707	0.820	0.892
5	Recall	0.747	0.807	0.888
5	F1-score	0.715	0.808	0.888
5	Accuracy	0.747	0.807	0.888

provides a detailed breakdown of the performance metrics for each fold. The data revealed that the SFAR method achieved the best results (highlighted in bold) across all folds and all evaluation metrics. Notably, the SFAR method performed exceptionally well in fold 5, achieving a precision of 0.892, while the baseline method only reached 0.707. This result suggests that incorporating factor analysis for session representation enabled the more effective capture of latent structures in user behavior patterns, thereby enhancing anomaly detection performance. Moreover, even in fold 1, where the baseline method performed the poorest (precision of only 0.603), the SFAR method maintained a high precision value (0.829), further demonstrating the robustness of the proposed approach.

5.5.2. Overall Performance Comparison

Figure 4 illustrates a comprehensive comparison of classification performance—based on the average results from 5-fold cross validation—across five different models: Decision Tree, Random Forest, MLP, XGBoost, and KNN. For each model, three methods were benchmarked: the traditional baseline approach, our proposed SARD, and SFAR. The evaluation was carried out using four key metrics—precision, recall, F1-score, and accuracy—which are presented in separate panels to enable a multi-dimensional assessment of performance.

As evident from Figure 4, our proposed methods demonstrated significant improvements over the baseline in most scenarios. Particularly noteworthy is the performance enhancement observed in MLP and KNN models, where both SARD and SFAR substantially outperformed the baseline. The improvement was more pronounced with SFAR, which consistently achieved the highest scores across most models and metrics.

Table 4. Performance comparison of different methods across various models (The best results are in bold).

Model	Method	Precision	Recall	F1-Score	Accuracy
Decision Tree	Baseline	0.806	0.799	0.798	0.799
	SARD (ours)	0.788	0.786	0.782	0.786
	SFAR (ours)	0.809	0.803	0.800	0.803
Random Forest	Baseline	0.835	0.827	0.826	0.827
	SARD (ours)	0.851	0.844	0.842	0.844
	SFAR (ours)	0.895	0.886	0.887	0.886
MLP	Baseline	0.130	0.361	0.192	0.361
	SARD (ours)	0.586	0.576	0.571	0.576
	SFAR (ours)	0.833	0.820	0.818	0.820
XGBoost	Baseline	0.867	0.862	0.862	0.862
	SARD (ours)	0.837	0.830	0.830	0.830
	SFAR (ours)	0.897	0.893	0.893	0.893
KNN	Baseline	0.625	0.608	0.604	0.608
	SARD (ours)	0.683	0.681	0.679	0.681
	SFAR (ours)	0.873	0.868	0.868	0.868

provides the detailed numerical results corresponding to Figure 4, with the best performance for each model and metric highlighted in bold. The most remarkable improvements were observed with the MLP model, where SFAR enhanced the precision from a baseline of 0.130 to 0.833, representing an improvement of over 500%. For KNN, SFAR achieved a precision of 0.873 compared to the baseline’s 0.625, demonstrating a substantial enhancement of approximately 40%.

Even for models that performed relatively well with the baseline approach, such as XGBoost and Random Forest, our methods still achieved meaningful improvements. SFAR increased XGBoost’s precision from 0.867 to 0.897 and Random Forest’s from 0.835 to 0.895. Notably, SFAR consistently outperformed both SARD and the baseline across all tested models, except for the Decision Tree model’s precision, where the baseline achieved 0.806 versus SFAR’s 0.809, a negligible difference that was offset by SFAR’s superior performance in recall, F1-score, and accuracy for that model.

These results demonstrate that our session-based feature extraction techniques, particularly the feature aggregation approach employed in SFAR, effectively capture the temporal and behavioral patterns in session data, leading to more accurate anomaly detection across various classification algorithms.

5.5.3. Performance Improvement Analysis

This section presents a quantitative analysis of the performance improvements achieved by the SARD and SFAR methods compared to the baseline approach. We employed Absolute Improvement ($AI$) as the evaluation metric, defined as the direct difference between the performance indicator values of the improved method and the baseline method, calculated as follows:

$$AI=P_{improved}-P_{baseline}$$

(1)

where $P_{improved}$ represents the performance metric value of the improved method, and $P_{baseline}$ represents the performance metric value of the baseline method. As shown in

Table 5. Performance improvement of SARD and SFAR methods compared to baseline.

Method	F1-Score Improvement	Accuracy Improvement
SARD	0.0846	0.0521
SFAR	0.1971	0.1627

, regarding the F1-score, the SARD method demonstrated an improvement of 0.0846 over the baseline, while the SFAR method achieved an improvement of 0.1971, with the latter being approximately 2.33 times greater than the former. In terms of accuracy, SARD improved by 0.0521, whereas SFAR improved by 0.1627, with the latter again significantly outperforming the former (by approximately 3.12 times). These results indicate that while both methods effectively enhanced the performance of the baseline approach, the SFAR method exhibited more substantial advantages across all evaluation metrics. Notably, the improvement in F1-score (0.1971) achieved by the SFAR method exceeded its improvement in accuracy (0.1627), suggesting that this method demonstrated superior performance in balancing precision and recall. This characteristic is particularly significant for addressing class imbalance problems, as it can effectively maintain classification performance for both minority and majority classes simultaneously.

6. Conclusions

This paper pioneered a novel session representation learning framework from the perspective of multiple instance learning, effectively addressing the inherent limitations of traditional statistical methods in Web robot detection. We designed and implemented two innovative approaches: SARD and SFAR. These methods automatically extract latent semantic features from high-dimensional sparse session data, enabling precise representation of Web behaviors. Through comprehensive five-fold cross-validation experiments, we thoroughly evaluated the performance of the proposed methods under different data partitioning conditions. The experimental results demonstrated that the SFAR method significantly outperformed traditional baseline approaches across all evaluation metrics, particularly achieving an average improvement of 24.6% in precision. Notably, even on datasets where the baseline method underperformed, the SFAR method maintained high precision and stability, showcasing its exceptional robustness and generalization capability. While the SARD method showed more modest improvements, it consistently outperformed traditional methods across all evaluation metrics, validating the effectiveness of our proposed framework.

7. Future Research Directions

Building on the promising results of this study, several avenues for future work could further advance research in this area: (1) Expanding application domains: investigate the applicability of the proposed session representation framework to additional and more diverse datasets, potentially including other types of Web logs or heterogeneous data sources, to further validate the model’s robustness and generalizability. (2) Real-time detection and adaptation: develop real-time detection systems that leverage the proposed techniques and incorporate online learning strategies to adapt dynamically to evolving attack patterns and emerging camouflaged bot behaviors. (3) Scalability and efficiency: investigate computational optimization techniques to improve the scalability and efficiency of the framework for deployment in large-scale, real-world environments with massive Web traffic.