ArabicEduCrawler: AI-Assisted Focused Crawling and Corpus Construction for Arabic Educational Web Content

Abstract

Arabic natural language processing (NLP) faces major difficulties due to the language’s rich morphological structure and the scarcity of high-quality datasets, especially for educational material distributed across diverse online platforms. Many existing large-scale corpus construction methods depend on extensive web crawling followed by substantial post-processing. This process may introduce irrelevant or low-quality data and often fails to represent the target domain adequately. As a result, a robust approach to developing corpora tailored for domain-sensitive educational NLP systems and linguistic depth is critical, as most current resources are inadequate. This paper presents ArabicEduCrawler, an AI-assisted focused crawling framework designed to improve the acquisition, discovery, and organization of Arabic educational web content. The framework integrates domain-aware source selection, in-crawl Arabic language detection using FastText, large language model (LLM)-assisted XPath extraction, and metadata retrieval to support corpus quality and traceability. Its two-layer architecture combines dynamic web crawling using Scrapy-Playwright with advanced NLP processing, including automatic linguistic annotation with GateNLP and Stanza and a sentence-aware chunking strategy designed for transformer-compatible token limits. Experiments across four major Arabic educational domains resulted in the creation of the Arabic Educational Web Corpus (AraEdu-WC), which consists of 101,770 documents segmented into approximately 286 k text chunks, with more than 50 million tokens, 289,778 sentences, and nearly 3.5 million named entities. The system achieved a harvest ratio of 95.25%, indicating its effectiveness in filtering and retaining relevant content. The sentence-aware chunking evaluation showed consistent improvements in top-ranked retrieval, achieving the highest Hit Rate@10 and MRR@10 across all four embedding models. In particular, the multilingual-E5-large model achieved a Hit Rate@10 of 70%, Precision@10 of 18%, and MRR@10 of 57%. These findings demonstrate that the proposed approach provides an effective balance between crawl efficiency, language purity, and content richness, offering a high-quality Arabic educational corpus for downstream NLP and retrieval research.

1. Introduction

Recently, building text corpora for medium- to low-resource languages (LRLs) has become a popular way to improve natural language processing (NLP) and information retrieval (IR) approaches. Despite these limited resources, there has been a continuous rise in interest in this field. Pre-trained language models (PLMs) depend heavily on the scale, diversity, and quality of their raw textual corpora. Recent work demonstrates that the curation of the pretraining corpus, in addition to architecture and training compute, shapes downstream performance [1,2,3]. Many prominent large-scale corpora are built on foundations like Common Crawl (CC) or web archives [4]. These resources are typically refined through post-processing, rather than being collected with task awareness during crawling. Examples include CCNet [5], C4 [6], OSCAR [7], FineWeb and FineWeb-Edu [8], ArabicWeb-Edu [9], and LanguageCrawl [10], all of which begin with broad web snapshots and subsequently apply cleaning, language identification, filtering, classification, or deduplication.

This paradigm has become indispensable for general-purpose pretraining and is particularly effective for scalability. However, it allocates a substantial amount of quality control to the post-data acquisition phase. Previous audit evidence emphasizes the risks associated with this design choice. Dodge et al. [11] report that C4 has much older web data and problems like benchmark contamination and leakage of personally identifiable or copyrighted information. They argue that web-scale corpora should report source-domain metadata since datasets extracted from large web crawls may not be representative of the larger linguistic community and also show uneven domain distributions and content from different time periods. Kreutzer et al. [12] report that quality audits of multilingual web corpora have revealed major problems such as incorrectly identified languages, misclassified information, boilerplate, and non-linguistic elements, which impact lower-resource languages significantly.

By shifting control earlier in the pipeline, focused crawling improves domain-specific corpus construction. For example, Remus and Biemann [13] show that targeted collection better captures particular knowledge domains compared to broad web crawling. These limitations are particularly significant for Arabic information retrieval due to its relatively lower representation compared to English. In addition, Arabic presents challenges such as rich morphology, dialectal variation, and inconsistent web coverage. This situation has prompted the development of Arabic-specific pretraining and retrieval resources, such as Arabic PLMs, dense passage retrieval models, and embedding benchmarks. Recent studies demonstrate that downstream performance can be improved beyond the capabilities of generic multilingual datasets by employing well-curated and highly targeted Arabic data [14,15,16].

This distinction is particularly important in the context of Arabic educational content. Although the rapid expansion of Arabic web content has increased opportunities for accessing educational information, systematically collecting this material remains difficult because it is spread across heterogeneous websites, affected by linguistic noise, and presented in a wide range of structural formats. In works such as FineWeb-Edu [8] and ArabicWeb-Edu [9], educational quality is usually determined after collecting a large, pre-existing web corpus. It is worth recognizing that the majority of corpora, which are in the billions of words, are constructed by downloading texts from the web without regard for their content or type. Thus, the content of such corpora cannot be ascertained prior to their creation; therefore, subsequent filtering, cleaning, and evaluation are essential.

In contrast, we propose ArabicEduCrawler, a framework designed to shape data quality earlier in the acquisition pipeline through focused source selection, Arabic language detection during crawling, extraction validation, and deduplication of URLs. Whereas most Common Crawl-derived pipelines rely on post hoc filtering over large web archives, ArabicEduCrawler embeds part of this quality control directly into the crawling and extraction stages.

This paper addresses the following research question:

RQ: How can an AI-assisted web crawling and scraping framework be developed to accurately navigate, identify, retrieve, and extract unstructured Arabic educational content from heterogeneous websites while addressing language, formatting, and corpus-structuring challenges?

To address this question, this paper presents the following contributions:

• We develop ArabicEduCrawler, a novel framework that integrates domain-aware source selection, FastText-based Arabic language identification, and LLM-assisted XPath extraction for early-stage quality control of Arabic educational web content.
• We employ Scrapy-Playwright to crawl both static and dynamic web pages, focusing on relevant educational domains while reducing unnecessary noise.
• We develop a dual-threshold Arabic language filtering method based on FastText to maintain high corpus purity and effectively remove irrelevant, non-Arabic, and mixed-language pages during data collection.
• We propose a sentence-aware chunking method for Arabic educational text that preserves sentence boundaries while dividing documents into token-bounded segments suitable for transformer-based models. In contrast to fixed-window segmentation, this method maintains sentence coherence while keeping chunk sizes appropriate for downstream indexing, retrieval, and annotation tasks.
• We create an automatic linguistic annotation pipeline using GateNLP and Stanza, which includes tokenization, lemmatization, part-of-speech tagging, and named entity recognition.
• We empirically validate the proposed approach through the construction of an Arabic Educational Web Corpus (AraEdu-WC), which contains 101,770 documents and 286,025 chunks, with a harvest ratio of 95.25%.
• We evaluate sentence-aware chunking against Character Text Splitter and Recursive Character Text Splitter using a FAISS index and ranx metrics in a retrieval experiment. The experiment uses 7505 Arabic query–document pairs derived from 500 source documents randomly selected from AraEdu-WC, where the queries were generated using Llama. The proposed sentence-aware chunking strategy achieves the highest Hit Rate@1, @5, and @10; Precision@1, @5, and @10; and MRR@10 across all evaluated embedding models.
• AraEdu-WC is released as an open, reusable resource to advance research in Arabic NLP and educational technology.

There are three reasons that make this design important. First, the distribution of Arabic educational content on the web is not uniform but is concentrated in specific websites, repositories, and institutional resources, which are the result of targeted acquisition rather than broad web sampling. Second, addressing language identification and normalization challenges early is crucial, as it prevents the accumulation of noisy or mixed-language pages in the retained corpus. Third, educational corpus construction often requires metadata traceability, domain interpretability, and extraction robustness at the document level, rather than only large-scale token counts. This makes our work complementary to Arabic corpus efforts such as ArabicWeb-Edu [9]. ArabicWeb-Edu demonstrates the value of educational-quality filtering for Arabic LLM training. However, neither study is primarily framed as an AI-assisted focused crawling framework for heterogeneous Arabic educational websites with integrated extraction validation, sentence-aware chunking, and automatic linguistic annotation. Our contribution thus resides at the convergence of focused crawling and the development of Arabic corpora for subsequent NLP applications.

The remainder of this paper is structured as follows. Section 2 introduces the background of web crawling and web scraping and examines the challenges related to Arabic NLP. Section 3 surveys relevant literature and identifies the existing research gaps. Section 4 presents the methodology for web crawling and corpus construction, which is organized into two main components: the Focused Web Crawling and Data Acquisition Layer (Section 4.1) and the Arabic Natural Language Processing Layer (Section 4.2). Section 5 describes the experimental setup and procedures, while Section 6 presents and discusses the results. In Section 7, the study outlines limitations and potential directions for future research. Finally, Section 8 concludes the paper.

2. Background

In this section, we describe the background of the major automated techniques to collect and structure large-scale web information. We focus on web crawling and web scraping, which are critical to building large-scale online corpora for various areas such as natural language processing, data mining, and knowledge discovery.

2.1. Web Crawling

Web crawling is the automated technique of navigating the World Wide Web to discover, fetch, and archive web resources by following hyperlinks across pages. This operation is typically performed by software agents known as “crawlers,” “robots,” or “spiders,” and is an essential component of search engines and large-scale data collection systems. The crawling process generally includes downloading HTML pages, building indexes and storing content for future retrieval and analysis. Web crawling has become an important interdisciplinary research field to support tasks such as information retrieval, data extraction, statistics and computational linguistics. Successful crawling requires careful consideration of factors like crawl depth, duplicate detection and politeness mechanisms to avoid overloading target servers [17,18,19].

However, implementing an effective crawler is complex because it requires mechanisms for fault tolerance, server-load management, compliance with the Robots Exclusion Protocol, file handling, and accurate page identification. To support different retrieval objectives, several crawling strategies have been proposed, including breadth-first crawling, parallel crawling, focused crawling, and incremental or repeated crawling. In this study, we use focused crawling, which restricts the crawl to pages relevant to the target topic. This strategy improves the efficient use of hardware and network resources while increasing the likelihood of retrieving useful content. The performance of focused crawlers is commonly evaluated using the harvest rate, defined as the number of relevant pages divided by the total number of retrieved pages, while the loss rate represents its complement [20]. Moreover, effective crawler design requires adherence to key crawling policies, including selection policies for prioritizing pages, politeness policies for avoiding server overload, policies for observing robots.txt instructions, robustness policies for avoiding crawler traps and faulty site structures, and parallelization policies for coordinating distributed crawling [20,21].

2.2. Web Scraping

Web crawling is about discovering and retrieving web pages, while web scraping is about automatically extracting useful information from those pages. It aims to turn scattered, unstructured web content into structured datasets for analysis. Web scraping has been widely utilized in many areas, such as education, academic research, economics, and medicine. Typically, the process starts by sending requests to specific web pages to get their HyperText Markup Language (HTML) content, then the desired information is extracted using scripts, parsing tools, or automation frameworks. As websites become increasingly complex and interactive, data scraping requires technical skill and adaptive strategies, making it not just a technical operation but also a research craft. This automation rapidly transforms scattered and unstructured online data into structured datasets suitable for computational analysis. The term “unstructured data” denotes data that lacks a clear, semantically explicit, and easily interpretable structure for computers [21,22]. On the other hand, recent advances in AI have further strengthened web scraping by allowing systems to adapt to changing page structures, identify patterns, and derive meaningful information from unstructured text [23].

2.3. Challenges in the Arabic NLP

Arabic NLP has increasingly become an area of interest over the past few years, with an increasing number of applications involving human–machine interaction, e.g., search engines, educational platforms, and virtual assistants. Arabic poses inherent computational challenges like root-pattern morphology complexity, orthographic variation, and a general absence of diacritics from the majority of written forms that lead to extremely high ambiguity levels. A single Arabic root, such as ك ت ب (k-t-b, “to write”), can generate multiple inflected forms with different meanings by consisting of different morphemes, as shown in Figure 1. CAMeL Tools [24] decomposes the word into the following components: فَ (coordinating conjunction), َس (future particle), كْتُب (verb stem), ونَ and يَ (third-person masculine plural imperfect verb affixes), and ها (feminine singular object clitic). Frequently, words are formed by combining multiple morphemes, including prefixes, suffixes, and attached pronouns or clitics.

The language includes short vowels, Fatha (بَ, /a/), Dhamma (بُ, /u/), and Kasra (بِ, /i/), which are typically written as diacritics, and long vowels, (ا: ā, long a), (و: ū, long u), and (ي: ī, long i), which are written as letters. The omission of short vowels in most text introduces significant lexical ambiguity. For example, كتب can be read as كَتَبَ (kataba, “he wrote”), كُتُب (kutub, “books”), or كِتَاب (kitāb, “book”). Such ambiguities complicate tokenization and morphological analysis; therefore, comprehensive preprocessing techniques are required to achieve accurate NLP tasks such as Named Entity Recognition (NER). This task involves identifying and classifying key entities in text into predefined categories, such as the names of people, organizations, locations, dates, and more. NER is crucial for various applications, including information retrieval, question answering, and content classification, as it helps in structuring unstructured data by extracting meaningful information [25,26,27,28].

However, the lack of large, annotated datasets for many Arabic languages hampers the training of deep learning models. While Arabic NLP faces significant challenges, ongoing research and innovation hold promise for overcoming these obstacles and enhancing the capabilities of NLP systems in understanding and processing the Arabic language in all its forms.

3. Related Work

Web crawling has been utilized across diverse contexts, complicating its classification due to the methodological variety of current approaches and the heterogeneous nature of web content, especially when considering language and domain-specific limitations. To present a clear review of related work, this section is structured into three main directions: the construction of corpora for low-resource and language-specific contexts via web crawling, domain-targeted crawling and scraping techniques, and the development of web corpora for pre-trained language models. This structure reviews existing studies and highlights the methodological gap that the present work aims to address.

3.1. Corpus Construction for Low-Resource Languages

Researchers have increasingly turned to web-based text corpus creation as an important approach for languages and domains where curated textual resources are limited. In these settings, web crawling is not just a retrieval mechanism but a way to build the baseline datasets for downstream NLP and language modeling applications.

For Arabic content, Hossain et al. [29] crawled Arabic health news using the Scrapy framework and performed content-based quality filtering after crawling, producing a collection of around 152 k web pages and 2 million sentences for text identification. Further research used focused crawling on Punjabi news websites using a Python-based crawler, producing a sizable corpus of 500 k sentences and 134 k pages [30]. In bilingual contexts, WCC-EC 2.0 [31] demonstrated that web crawling can also support specialized corpus construction by collecting about 1.6 M English–Chinese sentence pairs for neural machine translation (NMT). Conversely, increased labor has rendered quality control more explicit. De Jesus and Nunes [32] created the Labadain Crawler, which used an Apache Nutch-based focused crawler to gather general domain Tetun web pages, a custom Tetun language identification model, and filtering during crawling. This resulted in a corpus of about 22 k web pages and 321.7 k sentences. Tahir and Mehmood [33] introduced Corpulyzer, a framework for building low-resource language corpora from the web. Using Urdu as a case study, they generated seed URLs from Common Crawl and crawled the Web to construct UrduWeb20, a corpus of 8.0 million Urdu webpages from 6590 websites. In addition to corpus generation, the framework includes website scoring, content-size filtering, and corpus analysis based on duplicate documents, URL depth, vocabulary distribution, and content richness. Corpulyzer allows for deeper crawling of Urdu websites than Common Crawl alone, showing web-based corpus engineering for low-resource languages. Similarly, Marquard and Suleman [34] demonstrated this direction for isiXhosa, one of South Africa’s LRLs, using a Scrapy-based focused crawler with NLTK for language processing to gather around 202,000 web pages and 4 million sentences from general isiXhosa web content. Their study is important because it treated crawling as corpus engineering rather than simple page discovery. In addition, Humkinar [35] targeted general Urdu web pages through focused crawling, using CLD2 for language identification and URL-based deduplication during crawling, and collected 14 million crawled web pages and 2.2 million scraped Urdu documents to support a search engine. These studies emphasize the value of language-specific preprocessing, relevance assessment, and framework adaptability for constructing quality corpora in low-resource contexts.

3.2. Crawling and Scraping for Research and Educational Content Discovery

In domain-specific crawling and scraping, the focus is not to collect broad language data from across the web but to target acquisition content from a specific subject area or knowledge domain.

In the research domain, Fahrudin et al. [36] introduced an improved scraping approach for scholarly sources with enhanced extraction capabilities, while Mutlu et al. [37] developed a keyword-based intelligent literature search system designed to reduce the effort involved in comprehensive scholarly collection. Both approaches target academic databases to retrieve relevant literature, thereby supporting research discovery and metadata analysis, while also improving the accuracy of retrieving articles related to specified terms. A comparable approach was presented by Ramachandran et al. [38] through the Web Crawler for Pharmacokinetics (WCPK), which uses API-based retrieval, DOI classification, and scheduled full-text harvesting to curate pharmacokinetic literature efficiently. Their system extracted metadata from 74,867 publications and achieved more than 97% accuracy in text extraction.

Educationally oriented work reveals a similar pattern. Barwary et al. [39] combined web crawling, web scraping, MongoDB storage, machine translation, and RDF/LOD-based representation to build a multilingual ontology from university websites and Google Scholar data. Wang [40] employed a Python-based crawling framework using XPath, Beautiful Soup, Selenium, Pandas, and Matplotlib to collect and analyze finance course data from Chinese university MOOC platforms, illustrating how crawled educational data can uncover patterns in course offerings, institutional participation, and resource quality, while also highlighting the lack of use of high-quality digital learning materials. Similarly, Aljemazi and Khder [41] developed Octobot to retrieve Google Scholar data for academic purposes, demonstrating the usefulness of automated relevance-driven collection, although their study offers limited quantitative evidence regarding system performance and accuracy.

Collectively, these studies clearly demonstrate that we can treat extraction quality and operational efficiency as first-class evaluation targets, rather than assuming them as side effects of crawling. These studies are close in spirit to the present work because they emphasize targeted acquisition rather than general web crawling. However, most are designed for the English language and also reveal recurring gaps in multilingual adaptability.

3.3. Web Corpora for Pre-Trained Language Models

Recent work on pretraining corpora has increasingly relied on large-scale web archives, especially Common Crawl, followed by downstream filtering, scoring, and deduplication. Rather than collecting data from narrowly defined sources at crawl time, these approaches typically begin with a broad web snapshot and then apply post hoc quality control to construct corpora suitable for language model training. As shown in

Table 1. Comparison of recent web corpus construction studies relevant to Arabic educational data.

Study	Domain	Lang.	Source	Strategy	QC	LID	Dedup.	Size	Limitation
FineWeb-Edu [8]	Edu.	EN	FineWeb/ Common Crawl	LLM scoring & classifier filtering	Post hoc	FastText	MinHash	1.3 T tokens	Demographic and religious bias skew
ArabicWeb-Edu [9]	Edu.	AR	1 M Arabic Common Crawl documents	Qwen labeling & mGTE filtering	Post hoc	NR	NR	1 M docs	Costly filtering; English-based criteria may not suit Arabic educational contexts.
FineWeb-Edu-Ar [42]	Edu.	AR	Translated FineWeb-Edu	MT construction using NLLB	Post-translation	N/A	Inherited	189.4 M passages; 202.4 B tokens	Translation inaccuracies; English-centric knowledge
101 B Arabic Words Dataset [43]	General	AR	Common Crawl WET	WET extraction, cleaning, normalization	Post hoc	NR	URL & MinHash	101 B words; 89.1 M pages	No downstream evaluation; regional imbalance
ArabicEduCrawler (This study)	Edu.	AR	Targeted Arabic educational websites	AI-assisted crawling, scraping, and processing	During crawling & post-processing	FastText	MD5	101 k docs; 286 k passages; 50 M tokens; 289 k sent.	Smaller scale than broad web archives; depends on source availability

QC: quality control; LID: language identification; Dedup.: deduplication; NR: not reported; EN: English; AR: Arabic; Edu.: Educational; sent.: sentences.

, recent large-scale corpus construction efforts differ in language, source data, and quality-control stage, but most rely on post hoc filtering over broad web archives rather than focused acquisition.

FineWeb-Edu [8] is a representative example of this paradigm. Built from FineWeb, which itself is derived from Common Crawl, it constructs an educational subset through LLM-based scoring and classifier filtering, producing a 1.3 T-token corpus. The study shows that educationally filtered web data can improve downstream benchmark performance, but it also reports demographic and religious bias skews in the resulting corpus.

Another recent effort relevant to Arabic educational corpora is FineWeb-Edu-Ar [42]. FineWeb-Edu-Ar constructs Arabic pretraining data by machine-translating the deduplicated FineWeb-Edu corpus into Arabic. The authors translate approximately 189 million passages using facebook/nllb-200-distilled-600M, comprising about 202 B tokens under an Arabic-trained tokenizer. This work is important because it addresses the scarcity of high-quality Arabic educational data for small language models. However, the resulting corpus is large in scale, but its construction depends on translation quality and inherits the knowledge distribution of the English source data. Therefore, it remains a translation-based adaptation rather than the development of a culturally relevant and contextually Arabic educational corpus. Similarly, the 101 Billion Arabic Words Dataset [43] demonstrates that broad Common Crawl archives can be transformed into a very large Arabic corpus through extensive extraction, cleaning, normalization, and deduplication. However, the authors note limitations such as the absence of downstream evaluation and uneven regional representation.

A similar direction has recently been explored for Arabic in ArabicWeb-Edu [9]. Inspired by FineWeb-Edu, the authors sample 1 million Arabic documents from Common Crawl, assign educational quality labels using Qwen-2.5-72B-Instruct, and train a classifier on top of mGTE-305M to filter large Arabic web corpora. Their experiments indicate that educational quality filtering improves Arabic LLM training outcomes. These studies are highly relevant to the present work because they demonstrate the value of quality-aware educational corpus construction. However, both approaches operate primarily through post hoc filtering over already collected broad web corpora, whereas our work focuses on integrating part of the quality control earlier through focused crawling and source selection.

3.4. Research Gaps

Low-resource corpus studies show how web acquisition can compensate for the scarcity of curated data. Domain-specific crawling studies indicate that source targeting and extraction quality matter in applied settings. What remains underrepresented is an integrated framework that connects acquisition stage decisions to corpus availability in a domain-sensitive setting.

The reviewed literature indicates significant gaps, notably a lack of studies on web crawling systems designed for Arabic content, particularly in the educational sector. While focused crawling has been explored for health, news, and general web domains in languages such as English, Punjabi, Chinese, and Urdu, Arabic remains significantly underrepresented. Within Arabic research, existing works such as AraCovTexFinder apply transformer-based models for specific classification tasks (e.g., COVID-19 text identification), but they do not apply quality filtering into a focused crawling pipeline for structured content acquisition. More recent studies such as FineWeb-Edu and ArabicWeb-Edu show the value of educational-quality filtering, but they remain based on post hoc selection or machine translation from large pre-existing web corpora rather than AI-assisted focused crawling for heterogeneous educational websites. This limitation is especially important because Arabic educational content is often heterogeneous and semi-structured, with metadata represented in diverse formats and vocabulary varying across platforms. At the same time, the scarcity of large-scale Arabic educational corpora continues to constrain the development of high-quality downstream models.

4. Materials and Methods

This section explains the process of designing the architecture of the proposed framework for web crawling and the collection of Arabic educational content, as well as the techniques and algorithms used in the design and evaluation of the approach. Figure 2 shows an overview of the proposed Arabic Educational Crawler (ArabicEduCrawler) architecture presented in this work. The ArabicEduCrawler is designed based on a two-layer architecture to achieve modularity and maintainability. In each layer, we utilize advanced Arabic NLP techniques and web crawling to achieve a specific functionality. The following sections describe in detail each process within the framework.

4.1. Focused Web Crawling and Data Acquisition Layer

In this section, we describe the data sources, selection criteria, crawling architecture, handling of dynamic content, language identification, metadata extraction, and raw data preparation. The extracted data produced by these steps serves as the foundation for the subsequent NLP processing layer.

4.1.1. Data Sources and Domain Scope

The framework is designed to collect Arabic educational content from a diverse array of sources, encompassing Islamic, literary, cultural, scientific, and encyclopedic websites. The purpose of using a limited and diverse seed set was not to maximize web-scale coverage but to evaluate whether the framework could robustly navigate, discover, and structure Arabic educational content across differing site designs and content organizations. This design is the outcome of a targeted selection of sources that are both educationally relevant and structurally diverse enough to validate the extraction and processing pipeline. It also increases the variety of content and semantic depth. Thus, this diversity enriches the content and improves semantic depth, which in turn enhances model comprehension and mitigates bias.

This study draws on the following primary resources:

• Alukah Network [44] is a network composed of thirteen websites, showcasing contributions from a notable assembly of writers and intellectuals. It hosts thousands of published articles across Islamic studies, literature, science, and cultural topics. The platform provides regularly updated content. The network supports both Arabic and English and includes a wide variety of content in various world languages.
• Adab [45] is a large digital repository of Arabic literary works, containing more than 90,000 poems and prose texts by 7316 classical and modern authors. It also provides biographical records and well-organized literary collections. With approximately 1.4 billion total views, the platform represents a linguistically rich and semantically diverse source of Arabic text.
• Shamela Library [46] is one of the largest digitized Arabic text repositories, with over 8000 books and about 7 million pages written by more than 3000 authors. It covers disciplines such as Islamic studies, history, Arabic linguistics, and literature. Its large volume makes it a significant resource for extracting Arabic texts.
• Arabic Wikipedia [47] contains over 1.3 million articles across a wide range of domains, making it a substantial encyclopedic resource. It has approximately 2 million registered users. This encyclopedia is collaboratively curated and offers structured, domain-diverse, and continuously updated Arabic content that contributes modern terminology and cross-disciplinary coverage.

The websites were selected based on a set of specific defined criteria, including domain relevance, presence of educational text, a preference for text-centric content to facilitate processing, and accessibility and licensing compliance. Thus, it is possible to collect reliable, diverse, and valuable Arabic educational material. The chosen resources vary in terms of topic coverage, content type, format, and accessibility, as summarized in

Table 2. Selected Arabic Educational Websites.

Domain	Description	Topics	Type	Format	Access
Alukah Network [44]	Islamic and cultural network presenting articles on science, literature, and contemporary issues from an Islamic perspective.	Islamic studies, literature, culture	Digital Portal	Text, PDF	Open
Adab [45]	Repository of Arabic literature featuring a rich collection of poetry and prose by renowned Arab writers.	Arabic literature, poetry	Encyclopedia	Text	Open
Shamela Library [46]	Digital Arabic e-book library offering a wide range of Islamic and literary works across multiple disciplines.	Islamic studies, history, literature, linguistics	Digital Library	Text, PDF	Open
Arabic Wikipedia [47]	Arabic edition of Wikipedia providing comprehensive information on various scientific disciplines.	Physics, chemistry, biology, mathematics	Encyclopedia	Text	Open

.

4.1.2. Improving XPath Extraction Using LLMs

XML Path Language (XPath) is a query language designed for extracting data from XML documents, which are structured as hierarchical tree representations. It allows for the selection of parts of an XML document, including HTML pages, based on criteria such as the name, position, or content of an element. Moreover, this language enables precise selection of nodes and supports operations on strings, numbers, and Boolean values. The World Wide Web Consortium (W3C) established XPath as a standard, and it is widely used in conjunction with the Document Object Model (DOM) to retrieve structured information from web pages [48,49]. Traditionally, creating XPath selectors manually for each website is both time-consuming and prone to error, particularly when dealing with large and diverse data sources.

A recent development in web data extraction is the use of large language models (LLMs) to automatically generate and refine XPath expressions [50]. Following the approach suggested by Li et al. [51], the system first generates potential XPath expressions using the LLM. Then, these expressions are evaluated and refined to maintain their effectiveness, not only with minor changes in the DOM but also with structural variations across different websites or within the same website. This method enhances the adaptability and accuracy of data extraction and reduces the need for manual intervention.

As shown in Figure 3, our workflow consists of four stages: (1) fetching raw HTML from given URLs, (2) generating XPath expressions using ChatGPT, (3) validating the generated XPaths with existing scraping tools, and (4) automating the extraction and storage of large-scale data.

We first render dynamic web pages using Scrapy-Playwright, which allows the crawler to extract fully loaded HTML content, including elements dynamically generated by JavaScript. We also use lightweight HTTP-based retrieval techniques, such as Python’s requests library in combination with BeautifulSoup parsing, for static or semi-dynamic pages. After obtaining the HTML source, we provide relevant snippets to ChatGPT, a large language model developed by OpenAI. This enables us to automatically produce candidate XPath expressions. The prompt used to generate XPath expressions with ChatGPT is detailed in Appendix A (

Table A1. Prompt for generating XPath expressions.

Component	Prompt
Role	You are a professional software engineer specialized in web scraping and HTML parsing.
Task	Your task is to analyze the HTML code provided by the user and generate robust XPath expressions that extract the requested target values.
Instructions	1. Carefully read the provided HTML snippet. 2. Identify structural patterns surrounding the target value. 3. Generate XPath expressions that reliably locate the target element. 4. Avoid hardcoding exact text values. 5. Use contains(., ’value’) instead of text()=’value’. 6. Prefer contains(@class, ’…’) and contains(@id, ’…’) over strict equality. 7. Ensure the XPath is robust against: - Additional classes being added - Minor ID changes - Extra wrapper elements
Output Format	Output only the XPath expressions, clearly labeled by target field.
Input	HTML snippet provided by the user.

).

The XPath selectors were generated using ChatGPT-4o through the ChatGPT web interface. A uniform prompt template was used across domains to ensure consistency. The prompt requested XPath expressions for a predefined set of article-level fields, such as title, main content, canonical URL, author, publication date, and other available metadata. The same prompt template was applied across all tested domains; however, the provided HTML snippet and the available target fields differed by domain. Consequently, the generated XPath selectors were site-specific, while the prompting procedure remained consistent.

After the LLM generates XPath expressions using methods such as contains() or structural anchoring instead of fragile absolute paths, these expressions are programmatically validated for correctness and structural resilience. This validation process uses Scrapy selectors or lxml parsers to ensure that the XPath correctly extracts the desired content from the document’s architecture. Once validated, the selectors are incorporated into automated crawling pipelines, which include JavaScript-enabled environments such as Scrapy-Playwright. Despite the web’s dynamic content, this integrated methodology makes it possible to collect flexible and reproducible data across different domains.

4.1.3. Focused Web Crawling and Scraping Architecture

The system employs an advanced crawling architecture to enable the thorough acquisition of Arabic educational materials. Scrapy constitutes the core of this architecture, supplemented by Playwright and large language models (LLMs) to refine and optimize the data extraction processes. Scrapy, an open-source Python framework, is specifically designed for crawling structured data from web sources. Originally designed for extracting data from the web, Scrapy is built on seven key components, including the Scrapy engine, spiders, downloader, and scheduler. In addition, it allows users to control the speed and frequency of their requests, helping them adhere to a website’s privacy and usage policies. Beyond that, the Scrapy framework includes several key features. Middleware and pipelines can be configured to implement filters and store data in various file formats. Scrapy’s modular and asynchronous design allows for the efficient management of a high volume of concurrent requests. Concurrently, Playwright facilitates headless browser functionality, thereby enabling the rendering and extraction of content generated by dynamic JavaScript. Consequently, this integrated approach ensures the retrieval of both comprehensive and current data from contemporary web sources. Figure 4 illustrates a comprehensive overview of the Scrapy architecture, comprising seven components [52].

The Scrapy Engine is the heart of the Scrapy framework; the engine manages the flow of data and controls how all other components interact. It is responsible for starting and stopping crawls, as well as triggering events based on system actions. (1) The Engine opens a domain and asks the Spider for the initial URLs to crawl. (2) The Engine receives requests from the Spider and sends them to the Scheduler. The Scheduler queues these requests to be processed later. (3) The Engine requests the next URLs to crawl from the Scheduler. (4) The Scheduler returns the next requests to the Engine. (5) The Engine sends these requests to the Downloader through the Downloader Middlewares. (6) The Downloader fetches the web pages and returns the responses to the Engine. (7) The Engine sends the responses to the Spider through the Spider Middlewares, where the Spider parses the pages, extracts items, and identifies additional URLs for further exploration. (8) The extracted items are then sent through the Item Pipeline, where the data can be cleaned, validated, transformed, and finally stored in a database or output file. Downloader Middlewares serve as hooks, enabling developers to inject custom processing logic into requests and responses as they pass between the engine and the downloader. These middlewares are useful for tasks like altering headers or managing proxies. Spider Middlewares, much like downloader middlewares, offer hooks for processing data as it travels between the engine and the spiders.

4.1.4. FastText for Arabic Language Identification

To collect large volumes of Arabic web content, we integrated FastText’s pre-trained language identification model directly into the crawling process. FastText, developed by Joulin et al. [53], operates at the subword level and uses a linear classifier to identify the language of text samples. Trained on extensive multilingual datasets such as Wikipedia, Tatoeba, and SETimes, FastText can recognize 176 languages, including those with limited resources. The model is highly efficient, processing approximately one thousand documents per second on a single CPU core [54,55].

For instance, Wenzek et al. [5] utilize FastText for language identification and duplicate removal to extract high-quality monolingual data from Common Crawl. Similarly, RefinedWeb [56] and Dolma [57] employ FastText to filter web data, applying strict confidence thresholds to ensure language purity. Penedo et al. [56] set a threshold of 0.65 to guarantee higher quality in their English corpus, while Soldaini et al. [57] exclude documents with an English probability below 0.5, resulting in the removal of about 61.7% of candidates. These thresholds highlight FastText’s effectiveness in producing high-quality monolingual datasets that preserve linguistic coherence.

Building on these results, we present a FastText-based algorithm employing a dual-threshold strategy to efficiently filter and identify Arabic web content, ensuring robust classification. The algorithm accepts as input a text snippet, a pre-trained FastText model, a minimum character length, two confidence thresholds (primary and secondary), and the number of top predictions to consider.

The primary and secondary thresholds were determined through a combination of prior web corpus construction practices and empirical calibration on our Arabic educational dataset. Previous studies commonly use FastText confidence thresholds to enhance language purity and reduce contamination from non-target languages, with typical values such as 0.65 and 0.5 for English corpus filtering. However, in our preliminary inspection of Arabic educational web pages, using a single lower threshold retained a relatively high number of mixed or noisy pages, especially those containing navigation text, metadata, citations, URLs, embedded foreign terms, or boilerplate material.

Based on this analysis, the primary threshold was set at 0.8 to prioritize precision. A document is accepted only when Arabic is the top-ranked FastText prediction with a confidence score equal to or exceeding this threshold. This strict criterion helps maintain the purity of Arabic content and reduces the risk of including non-Arabic or weakly Arabic pages. The secondary threshold, set at 0.5, enhances recall by retaining documents where Arabic appears among the top-k predictions with sufficient confidence, even if it is not the top-ranked label.

The dual-threshold mechanism operates as follows: if the top-ranked prediction is an Arabic label variant with a confidence score above the primary threshold, the text is classified as Arabic. Alternatively, if an Arabic label appears within the top-k predictions (usually the top three) with a confidence above the secondary threshold, the text is also classified as Arabic. Pages failing to meet either condition are excluded, reducing dataset contamination and maintaining efficient web crawling.

For transparency, the algorithm logs the rationale behind each classification decision. Every processed text is accompanied by a label indicating acceptance by the primary threshold, acceptance by the secondary threshold, or rejection (with reasons such as insufficient length, low confidence, or model unavailability). This structured logging facilitates auditing, threshold calibration, and future adaptation of threshold values for other domains or languages. The procedure is formally summarized in Algorithm 1.

Algorithm 1 Arabic Language Identification Using FastText

Input: Text $text$; FastText model $fastTextModel$; $min\text{\_}chars$; Primary Threshold $T_{\mathrm{p}}$; Secondary Threshold $T_{\mathrm{s}}$; Top-k Predictions k. Output: $is\text{\_}arabic$, $reason$, $labels$, $confidences$.   1: Initialize $is\text{\_}arabic\leftarrow \mathrm{False}$   2: Define $arabic\text{\_}labels=\{\text{\_\_}label\text{\_\_}ar,\text{\_\_}label\text{\_\_}ara,\dots \}$ as the set of Arabic label variants   3: if model $fastTextModel$ is not loaded then   4: return False, “Model not loaded”   5: end if   6: if $text$ is empty or $\left|text\right|<min\text{\_}chars$ then   7: return False, “Text too short”   8: end if   9: Normalize $text$ (removing diacritics and standardizing Arabic characters) 10: Predict $(labels,confidences)$ using top-k$\leftarrow fastTextModel.predict(text,k)$ 11: Convert confidence scores to float values 12: Primary threshold: Arabic as the top-1 prediction 13: if the top-1 predicted label $\in arabic\text{\_}labels$ and its confidence $\ge T_p$ then 14: return True, “Primary threshold”, $labels$, $confidences$ 15: end if 16: Secondary threshold: Arabic present in top-k predictions 17: for each $(label,conf)$ in top-k predictions do 18:     if $label\in arabic\text{\_}labels$ and $conf\ge T_s$ then return True, “Secondary threshold”, $labels$, $confidences$ 19:     end if 20: end for 21: If no condition is satisfied, return False, “Low confidence”, $labels$, $confidences$ 22: If an error occurs during prediction, return False, “Prediction error”, $labels$, $confidences$

4.1.5. Metadata Extraction and Storage

The pipeline stores metadata at both the crawl level and the resource level for each retained page. This metadata includes the URL, source domain, document title, publication information (if available), and crawl timestamps. Moreover, technical information such as HTTP status codes and content length is collected. The preservation of metadata is crucial for three reasons. First, it enables the corpus to be traceable and reproducible. Second, it simplifies domain-level statistics and source analysis. Third, it ensures that the corpus remains useful for downstream search, indexing, and interpretability. In the current system, both content and metadata are treated as equal products of the acquisition process. All data and metadata collected are stored in MongoDB, providing efficient management, traceability, and easy integration with subsequent processes. The metadata schema for the collected sources is summarized in

Table 3. Source Domains with Resource and Crawling Metadata.

Source Domain	Resource Metadata	Crawling Metadata *
alukah.net [44]	Title, Author, Publication Date, View Count.	Crawl Timestamp, HTTP Status, Response Headers, Encoding, Crawl Depth, Referer URL, Source URL, Content Length, HTML Length, MD5 Hash.
adab.com [45]	Title, Author, Profile URL, Post Date, Page Number, Poem Length, Excerpt.
shamela.ws [46]	Title, Author, Publisher, Edition, Book ID, Total Pages, Page Number.
ar.wikipedia.org [47]	Title, Author, Subcategory, Last Modified Date.

* Crawling metadata is shared across all source domains.

.

4.2. Arabic Natural Language Processing Layer

The second layer of the proposed system establishes a linguistic foundation for the dataset through an Arabic NLP pipeline. It is designed to address the morphological complexity and nuances inherent in the Arabic language. This phase serves several purposes: it preprocesses and normalizes the data, transforming raw text into unified, linguistically annotated units. Additionally, this layer introduces a text chunking strategy that goes beyond standard fixed-length segmentation to implement sentence-aware chunking, leveraging AraBERT model tokenization.

4.2.1. Sampling Strategy Using Yamane’s Formula

A sampling approach based on statistical grounds is used to ensure representative quality checks, following methods shown in large Arabic corpus construction efforts [43]. We use Yamane’s formula to draw a subset of documents for manual review so we can identify and address data quality concerns. The sample size n for a population of size N documents can be calculated as:

$$n={\displaystyle \frac{N}{1+N{e}^2}}$$

(1)

where e is the desired margin of error. For our dataset with N = 104,421 documents and a margin of error $e=0.01$ (1%), the formula yields a sample size of approximately $n\approx 9126$ documents. A manual inspection of this sample shows a number of problems, including empty lines, special Unicode characters, and residual HTML noise, which inform subsequent normalization, cleaning, and annotation processes.

4.2.2. Data Preprocessing

The preprocessing pipeline begins with data ingestion and inspection of raw text using Yamane’s sample (see Section 4.2.1). The collected corpus from various web pages often contains irrelevant elements such as HTML residues, JavaScript code, boilerplate navigation text, zero-width markers, control characters, incorrectly formed language segments, and other noise.

Each document is checked for UTF-8 encoding to prevent character corruption. The Normalization Form Compatibility Composition (NFKC) method is applied to enhance consistency in tokenization and embedding. Emojis, decorative symbols, random Unicode characters, and formatting controls are removed to reduce unnecessary information. Nonlinguistic artifacts are removed, retaining only essential punctuation marks that separate sentences. Repeated punctuation marks are reduced, like “!!!!!” would be changed to “!”. We eliminate diacritics and elongation, substituting references and citations with <REF>. All Latin script, URLs, and inline hyperlinks are eliminated, as they do not contribute semantic meaning and can degrade embedding quality.

4.2.3. Text Chunking

Large documents can be segmented into meaningful units called chunks, which are important for tasks such as retrieval. Recent research examined various approaches to enhance text segmentation, with Al-Subaie et al. [58] specifically assessing four strategies in Arabic RAG systems. They discovered that sentence-aware chunking achieves the highest performance score (74.78), surpassing fixed-size, recursive, and semantic chunking techniques. On the other hand, semantic chunking, as described in [59], utilizes document summaries to facilitate semantic similarity-based chunking. Although this approach improved retrieval metrics, Qu et al. [60] discovered that semantic chunking did not consistently justify the additional computational cost in comparison to fixed-size chunking. Merola et al. [61] noted that larger chunks reduce retrieval speed and relevance, while smaller chunks risk losing context.

These considerations are also relevant to Arabic embedding and retrieval evaluation ecosystems such as ArabicMTEB [16], where retrieval performance depends on producing coherent, semantically meaningful Arabic passages that can be effectively encoded by dense embedding models. Therefore, sentence-aware chunking is adopted in this work because it preserves sentence boundaries while maintaining transformer-compatible token limits, making the resulting chunks suitable for Arabic dense retrieval. Notably, inadequate segmentation reduces semantic relevance, while overly complex chunking increases computational costs without corresponding benefits. The optimal chunking strategy depends on the context and specific task requirements.

Based on these findings, we implemented a sentence-aware chunking strategy, dividing text at sentence boundaries and using the AraBERT tokenizer to ensure chunk size limits: a maximum of 450 tokens and a minimum of 20 tokens. The upper limit of 450 tokens was chosen to ensure compatibility with transformer-based embedding models while still providing enough educational context. The lower limit of 20 tokens was established after manually reviewing a representative sample determined using Yamane’s formula; chunks falling below this threshold often consisted of brief references or incomplete text segments that had limited value for independent retrieval. This range is specific to the dataset and may be modified for other fields or retrieval purposes.

Due to the morphological complexity of Arabic, a single word can be split into multiple subword tokens by the tokenizer. For example, “فسيكتبونها” (and they will write it) is tokenized as [’##ونها‘,’##كتب‘,’فسي‘] (3 tokens), and “مدرسة” (school) as [’##ة‘,’مدرس‘] (2 tokens), even though each is a single word [14]. Therefore, chunk size is determined by token count, not word count, to ensure compatibility with transformer model input requirements. This approach supports transformer models and retrieval-based NLP systems, prioritizing simplicity and scalability over computationally expensive semantic chunking. A document D is defined as an ordered sequence of sentences:

$$D=\{S_1,S_2,\dots ,S_n\}.$$

(2)

Each chunk $C_k$ consists of a set of consecutive sentences:

$$C_k=\{S_i,S_{i+1},\dots ,S_j\},$$

(3)

such that the number of tokens produced by the tokenizer satisfies

$$T_{\min }\le \left|\mathrm{Tokenizer}\left(C_k\right)\right|\le T_{\max },$$

(4)

where $T_{\max }=450$ and $T_{\min }=20$.

The proposed chunking algorithm operates in three sequential stages:

• Primary Sentence Segmentation: The document is first segmented using strong punctuation marks, including the period, question mark, exclamation mark, and their Arabic equivalents.
• Secondary Sub-Sentence Segmentation: If a primary sentence exceeds the maximum token limit ($T_{\max }$), it is further divided using weaker punctuation such as the Arabic comma and semicolon.
• Word-Level Fallback: In case no suitable punctuation is found, and the sentence surpasses the token limit, the algorithm will divide the text at the word level to comply with the token constraints.

Ultimately, the acquired segments are combined into groups that are contextually relevant and fall within the specified token limit. This approach ensures that the resulting chunks are suitable for document indexing and for NLP tasks, such as semantic retrieval.

4.2.4. From Raw Text to Annotations

In this research, we used Python GateNLP, a modern library for the General Architecture for Text Engineering (GATE) framework, and the Stanza Annotator, which is designed for Arabic text annotation. GATE is an open source platform created in 1995 at the University of Sheffield, which allows a range of natural language processing applications. Its modular architecture facilitates efficient text processing, annotation, and information extraction. GateNLP extends this framework to annotate text, manage corpora, and integrate with popular NLP libraries such as spaCy and NLTK, facilitating the development of complex processing pipelines. One advantage is that the annotations and features can be visualized interactively in HTML format, which can be embedded in Jupyter or Colab notebooks [62,63]. Figure 5 shows an example of an interactive visualization of annotations with Stanza NER outputs over an Arabic text from our dataset. The sample is a Wikipedia paragraph about Martin Seligman, used to demonstrate Arabic named entity annotation; for example, terms such as person names and locations are identified and labeled.

Our pipeline comprises the following core NLP tasks: tokenization, part-of-speech (POS) tagging, lemmatization, and named entity recognition (NER). These tasks are essential for understanding the grammatical and semantic dimensions of the text. Furthermore, this framework not only annotates datasets but also assists in content discovery from web documents. In this study, POS, lemmatization, and NER annotations are automatically generated using GateNLP and Stanza. The resulting annotations provide an enriched linguistic layer that supports corpus exploration, indexing, retrieval, and descriptive statistics. The cleaned textual corpus remains available independently, allowing researchers to apply alternative annotation tools or annotation schemes according to their research objectives. The annotated data is stored in PostgreSQL along with the cleaned text, facilitating efficient retrieval and subsequent analysis. Thus, this approach ensures precise processing of Arabic data while simultaneously improving the scalability of NLP research endeavors.

Figure 6 summarizes the workflow and presents an overview of the main steps in our data preprocessing and annotation pipeline. The crawling and filtering pipeline initially processed 109,633 documents. After applying URL deduplication and FastText-based Arabic language filtering, 104,421 documents were retained as the raw accepted corpus, while 5212 were excluded. The sample size for quality inspection was determined using Yamane’s formula, based on this raw corpus before any downstream cleaning.

Following preprocessing, normalization, and removal of residual noise, the corpus was reduced to a final cleaned set of 101,770 documents. These documents were then segmented into 286,025 retrieval chunks using a sentence-aware chunking strategy, with chunk size constraints defined by the AraBERT tokenizer.

Linguistic annotation and sentence-level analysis were subsequently performed with Stanza, resulting in 289,778 sentence units across the chunked corpus. Therefore, statistics based on 104,421 documents refer to the raw, accepted crawl output, while those based on 101,770 documents correspond to the final cleaned corpus used for chunking, annotation, and analysis.

5. Experimental Setup

The experimental workflow was designed to be modular and completely scripted to ensure it can be replicated. Furthermore, we used independently scalable and adaptable Docker-based services. Code and prompts are publicly available on GitHub (https://github.com/afyaa-alkhamisi/ArabicEduCrawler-code, accessed on 5 June 2026).

5.1. Hardware and Software Environment

This study used multiple computing environments across the experimental pipeline to ensure reproducibility, dependency isolation, and sufficient computational resources. The main hardware and software configurations are summarized in

Table 4. Experiment environment details for reproducibility.

Configuration	Crawling	GateNLP	Chunking Experiment
Platform	macOS		Windows 11
Hardware	Apple Silicon, 8 CPU cores, 16 GB memory		i7-13620H, 24 GB RAM, RTX 5060 8 GB
Runtime	Docker		Python 3.12, PyTorch, CUDA 13.1
Image	python:3.11-slim	jupyter/base-notebook:python-3.11.5	Not applicable

. The focused web crawling and GateNLP stages were run on a macOS 15.0 computer with Apple Silicon, 8 CPU cores, and 16 GB of memory. For reproducibility and environment isolation, the crawling stage was run inside a Docker Desktop 4.48.0 container derived from python:3.11-slim, whereas the GateNLP stage was conducted in a separate Docker container derived from jupyter/base-notebook:python-3.11.5.

The chunking experiment was run on a Windows 11 computer equipped with an Intel Core i7-13620H processor, 24 GB of memory, and an RTX 5060 Laptop GPU with 8 GB of VRAM, using Python 3.12, PyTorch (version 2.10.0+cu130), and CUDA 13.1. These environments supported the collection of Arabic web content, its subsequent linguistic processing, and the chunking retrieval experiment under controlled execution conditions.

5.2. Evaluation Protocol

The evaluation of the proposed framework adheres to established methodologies in both focused crawling research and corpus-construction studies. The framework was assessed along four primary dimensions: crawl efficiency, filter effectiveness, corpus yield, and chunking quality.

Crawl efficiency was quantified using acquisition and coverage statistics, including the number of URLs, the number of domains covered, crawl speed, and crawl duration. Within this context, the harvest ratio constitutes a key performance indicator, as it captures the proportion of relevant and retained webpages relative to the total volume of crawled content. A high harvest ratio is indicative of effective exclusion of irrelevant material, which is critical for preserving the topical focus of the crawler.

Filter effectiveness was examined through in-crawl quality-control signals, particularly the FastText top-3 predictions and their associated confidence values for both accepted and rejected documents, in order to better understand the linguistic profile of the collected corpus and the behavior of the language filtering stage.

Corpus yield was assessed through corpus structuring statistics, including the number of documents, chunks, tokens, sentences, lemmas, and named entities. Overall, this evaluation design is grounded in prior works on focused web crawling, webpage language identification, deduplication and filtering pipelines, and Arabic web corpus construction while adapting those ideas to a setting in which quality control is integrated into crawling itself rather than applied only after large-scale Common Crawl harvesting.

5.3. Focused Arabic Crawling Experiment

In this study, a focused Arabic web crawling procedure was employed to construct the corpus. The proposed methodology harvested data directly from contemporary Arabic web sources that were selected on the basis of their educational relevance. In particular, four domains and 11 seed URLs related to predefined topics were crawled, as shown in

Table 5. Seed URLs and Their Corresponding Main Topic Areas in the Selected Websites (all URLs accessed on 8 June 2026).

Domain	Seed URLs	Main Topic Area
ar.wikipedia.org	https://ar.wikipedia.org/wiki/تصنيف:كيمياء	Chemistry
	https://ar.wikipedia.org/wiki/تصنيف:علوم_فيزيائية	Physics
	https://ar.wikipedia.org/wiki/تصنيف:علم_الأحياء	Biology
	https://ar.wikipedia.org/wiki/تصنيف:رياضيات	Mathematics
alukah.net	https://www.alukah.net/sharia/0/	Sharia
	https://www.alukah.net/literature_language/0/	Literature & Language
	https://www.alukah.net/culture/0/	Culture
	https://www.alukah.net/library/0/	Library
	https://www.alukah.net/social/0/	Social Sciences
adab.com	https://www.adab.com	Arabic Literature and Poetry
shamela.ws	https://shamela.ws	Islamic Studies, Literature, Linguistics, History

. The crawler was implemented using a multi-spider architecture, wherein each spider was customized to the structural characteristics of its target website and extracted content via pagination links, category pages, article pages, and AJAX responses, using site-specific, automatically generated XPath expressions.

To ensure stable and ethically compliant crawling behavior, the framework relied on controlled configuration settings regulating request delays, concurrency levels, retry policies, timeout management, and adaptive traffic regulation. These parameters were tuned to achieve an effective trade-off between crawl coverage, politeness, and robustness when interacting with heterogeneous Arabic web sources. To further improve corpus quality, the pipeline integrated the facebook/fasttext-language-identification (https://huggingface.co/facebook/fasttext-language-identification, accessed on 8 June 2026) model [54], as detailed in Section 4.1.4. This model was used to filter out pages that did not contain sufficiently reliable Arabic content, based on two predefined thresholds. Pages that satisfied these criteria were subsequently stored in MongoDB, using two separate collections: one for raw textual data and another for structured metadata. Each stored record included the page URL, title, crawl-related information, and associated metadata, as specified in Section 4.1.5. Additional implementation details, including the primary crawling parameters, are provided in Appendix B (

Table A2. Main Parameters and Configuration of the Scrapy-Playwright Crawling Pipeline.

Parameter	Value	Purpose
Browser Type	Chromium (headless)	Dynamic page rendering
ROBOTSTXT_OBEY	True	Crawl policy compliance
DOWNLOAD_DELAY	2.0 s	Request delay rate
RANDOMIZE_DOWNLOAD_DELAY	True	Reduced timing regularity
CONCURRENT_REQUESTS	16	Total request limit
CONCURRENT_REQUESTS_PER_DOMAIN	8	Per-domain request limit
CONCURRENT_REQUESTS_PER_IP	8	Per-IP request limit
AUTOTHROTTLE_ENABLED	True	Automatically throttling crawling speed based on server response
AUTOTHROTTLE_TARGET_CONCURRENCY	2.0	Limits server load
RETRY_ENABLED	True	Failed request recovery
RETRY_TIMES	3	Retry limit per request
DOWNLOAD_TIMEOUT	30 s	Response wait limit
HTTPCACHE_ENABLED	True	Response caching
HTTPCACHE_EXPIRATION_SECS	86,400 s	Cache duration period (24 h)

).

5.4. Sentence-Aware Chunking Evaluation

We conducted a downstream retrieval experiment using a randomly sampled query–document subset from AraEdu-WC. This subset comprised 7505 query–document pairs associated with 500 unique source documents. The queries were generated using llama3.1:8b (https://ollama.com/library/llama3.1:8b, accessed on 8 June 2026) [64]. All generated queries were subject to systematic cleaning and preprocessing to ensure consistency, remove generation artifacts, and standardize formatting prior to evaluation. The exact prompt used for generating synthetic Arabic user queries is provided in Appendix C (

Table A3. Prompt for generating Arabic user questions from educational documents.

Component	Prompt
Role	You are an AI assistant tasked with generating exactly 6 realistic Arabic user questions based on educational documents.
Task	Generate questions strictly based on the provided document excerpt.
Instructions	1. Questions must be answerable using ONLY the information in the excerpt. 2. Do not assume or use external knowledge. 3. Do not ask about information not explicitly stated in the document. 4. Generate at least 2 questions in Modern Standard Arabic (dialect: “msa”). 5. Generate at least 2 questions in Saudi dialect (dialect: “saudi”). 6. Include the exact snippet in “location” that answers each question. 7. Focus on concrete facts, not interpretation or opinion.
Output Format	Output must be ONLY a single valid JSON object with the following structure: { “questions”: [ { “question_id”: 1, “question”: “question text”, “location”: “Quote from excerpt that contains the answer”, “dialect”: “msa | saudi” } ] }
Input	Document excerpt provided by the user.

).

For query generation, the model parameters were configured as follows: the temperature was fixed at 0.7 to balance lexical diversity and semantic coherence; top_p was set to 0.9 to restrict token sampling to high-probability candidates; num_predict was set to 500 to cap the maximum output length; and stream was set to False to return each generated response in a single batch. The resulting query set comprises both Modern Standard Arabic (MSA) and Saudi dialect variants, with 4350 MSA queries and 3155 Saudi dialect queries.

The documents were sampled from the four source collections used in AraEdu-WC: Alukah, Shamela, Arabic Wikipedia, and Adab. Appendix D presents

Table A4. Summary of the query-document subset used in the chunking retrieval experiment.

Statistic	Value
Query-document pairs	7505
Unique source documents	500
Query generation model	llama3.1:8b
MSA queries	4350
Saudi dialect queries	3155

and

Table A5. Source and dialect distribution of the query-document subset.

Source	Total Pairs	MSA	Saudi Dialect
Alukah	3731	2215	1516
Shamela	1236	694	542
Arabic Wikipedia	1535	892	643
Adab	1003	549	454
Total	7505	4350	3155

, which summarize and detail the distribution of the dataset employed in the chunking-based retrieval experiment. Furthermore,

Table A6. Examples of synthetic Arabic queries (MSA and Saudi dialect), retrieved answers generated by llama3.1:8b, and supporting document snippets from the AraEdu-WC dataset.

Ex.	Dialect	Query	Retrieved Answer	Document Snippet
E1	Saudi	ما عدد العينات التي تم جمعها من القمر خلال برنامج أبوللو؟ How many samples were collected from the moon during the Apollo program?	تم تجميع ٣٤٨ كجم من العينات من القمر A total of 348 kg of samples were collected from the moon.	وفي فترة برنامج الفضاء أبوللو تم تجميع ٣٤٨ كجم من العينات من القمر During the Apollo space program, 348 kg of samples were collected from the moon.
	Saudi	ما هو اسم البرنامج الذي تم خلاله تجميع العينات من القمر؟ What is the name of the program during which the moon samples were collected?	في فترة برنامج الفضاء أبوللو تم تجميع ٣٤٨ كجم من العينات من القمر During the Apollo space program, 348 kg of samples were collected from the moon.	وفي فترة برنامج الفضاء أبوللو تم تجميع ٣٤٨ كجم من العينات من القمر During the Apollo space program, 348 kg of samples were collected from the moon.
	MSA	من هو عالم الفلك الذي اكتشف الأقمار الأربعة في المشتري؟ Who is the astronomer who discovered Jupiter’s four moons?	وفي هذا الإتجاه نجد أن عالم الفلك الكوكبي الأول هو جاليليو جاليلي In this context, the first planetary astronomer is Galileo Galilei.	عالم الفلك الكوكبي الأول هو جاليليو جاليلي الذي اكتشف الأقمار الأربعة في المشترى The first planetary astronomer is Galileo Galilei, who discovered Jupiter’s four moons.
	MSA	كيف يتم حل شفرة تاريخ سطح الكوكب؟ How is the chronology of a planet’s surface decoded?	يتم حل شفرة تاريخ سطح الكوكب عن طريق وضع خرائط جيولوجية للمعالم الموجودة عليه من القمة إلى القاع طبقا لتتابع الطبقات وترتيبها The chronology of a planet’s surface is decoded by creating geological maps of its features from top to bottom, according to the order and succession of layers.	يتم حل شفرة تاريخ سطح الكوكب عن طريق وضع خرائط جيولوجية للمعالم الموجودة عليه من القمة إلى القاع طبقا لتتابع الطبقات وترتيبها The chronology of a planet’s surface is decoded by creating geological maps of its features from top to bottom, according to the order and succession of layers.
E2	Saudi	متى طور برنامج فيورستين التعليمي الإغنائي؟ When was the Feuerstein Instrumental Enrichment program developed?	برنامج فيورستين التعليمي الإغنائي، ١٩٨٠ The Feuerstein Instrumental Enrichment program, 1980.	وبرنامج فيورستين التعليمي الإغنائي، ١٩٨٠ And the Feuerstein Instrumental Enrichment program, 1980.
	Saudi	مين اللي بنزل برنامج الفلسفة للأطفال؟ Who introduced the Philosophy for Children program?	برنامج الفلسفة للأطفال Philosophy for Children program.	برنامج الفلسفة للأطفال للبمان Lipman’s Philosophy for Children program.
	MSA	كم عدد وحدات برنامج كورت؟ How many units are in the CoRT program?	يتكون برنامج كورت من ست وحدات تعليمية The CoRT program consists of six educational units.	يتكون برنامج كورت من ست وحدات تعليمية تعطي جوانب عديدة للتفكير The CoRT program consists of six educational units that address various aspects of thinking.
	MSA	من هو مؤلف برنامج البناء العقلي لجيلفورد؟ Who is the author of Guilford’s Structure of Intellect program?	برنامج البناء العقلي لجيلفورد، الذي طورته الباحثة ميكر، ١٩٦٩ Guilford’s Structure of Intellect program, which was developed by researcher Meeker in 1969.	ومن بين البرامج المعروفة التي تمثل اتجاه العمليات المعرفية برنامج البناء العقلي لجيلفورد، الذي طورته الباحثة ميكر، ١٩٦٩ Among the well-known programs representing the cognitive operations approach is Guilford’s Structure of Intellect program, which was developed by researcher Meeker in 1969.

provides examples of synthetic Arabic queries, their corresponding retrieved answers generated by llama3.1:8b, and supporting evidence snippets extracted from the original source documents. These examples encompass both MSA and Saudi dialect queries, thereby illustrating the range of linguistic variation represented in the generated query set. Collectively, they demonstrate the diversity of question formulations and the relevance of the retrieved answers and passages, highlighting the practical capabilities of the proposed retrieval framework in Arabic educational settings.

The proposed sentence-aware strategy was empirically compared against two baseline methods implemented in LangChain. The CharacterTextSplitter was used as a fixed-size, character-level baseline, configured with a chunk size of 1500 characters and zero overlap. The RecursiveCharacterTextSplitter served as a stronger, separator-aware baseline, instantiated with the AraBERT tokenizer, a chunk size of 450 tokens, and no overlap.

We evaluated all chunking strategies using four embedding models: multilingual-E5-large (https://huggingface.co/intfloat/multilingual-e5-large, accessed on 8 June 2026) [65], paraphrase-multilingual-MiniLM-L12-v2 (https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, accessed on 8 June 2026) [66], Arabic-Triplet-Matryoshka-V2 (https://huggingface.co/Omartificial-Intelligence-Space/Arabic-Triplet-Matryoshka-V2, accessed on 8 June 2026) [67], and Arabic-mpnet-base-all-nli-triplet (https://huggingface.co/Omartificial-Intelligence-Space/Arabic-mpnet-base-all-nli-triplet, accessed on 8 June 2026) [68]. The resulting chunk representations were indexed using FAISS [69], and retrieval performance was measured with ranx [70]. We report standard information retrieval metrics, including Hit Rate@1, @5, and @10; Precision@1, @5, and @10; Recall@1, @5, and @10; as well as MRR@10, MAP@10, and nDCG@10.

6. Results and Discussion

6.1. Crawling Performance

The crawling performance of the four spiders was measured using four metrics: crawl speed, crawl duration, total accepted documents scraped, and average extracted content length.

As shown in Figure 7a, shamela achieved the highest crawl speed at 20.8 items per minute, followed by alukah at 19.6 and arabic_wiki at 17.1. Meanwhile, the adab spider recorded the lowest speed at 13.5 items per minute, which is perhaps explained by the AJAX structure of the target website. These environments use dynamic loading of content which leads to higher latency due to rendering, asynchronous requests and interaction overhead. The extraction throughput is therefore lower than static sources.

Figure 7b shows that alukah, adab, and shamela each crawled for approximately 26.5 h, whereas arabic_wiki completed in 18.3 h. This indicates that the crawl space is more bounded or smaller. The most accepted pages were documented for shamela at 33,019, succeeded by alukah at 31,138, adab at 21,475, and arabic_wiki at 18,789, as shown in Figure 7c. However, volume alone does not capture corpus richness.

Figure 7d shows that alukah produced the longest documents on average at 11,626 bytes, substantially exceeding arabic_wiki at 5295 bytes, shamela at 2877 bytes, and adab at 1618 bytes. To summarize, these results reveal complementary strengths across the spiders: shamela was the most efficient in terms of speed and accepted document yield, whereas alukah contributed richer content per document. The arabic_wiki spider showed balanced performance with a shorter runtime, while adab appears to have been constrained by the more demanding structure of its source.

These findings indicate that crawler performance cannot be reduced to a single measure of efficiency. A spider that maximizes item count may not maximize document richness. Conversely, a spider that runs more slowly may still be valuable if it accesses otherwise difficult but important content. Therefore, crawling and scraping performance is better understood as a multidimensional trade-off among extraction speed, operational cost, corpus yield, and textual quality.

Additional examination details on document rejection by source are provided in Appendix E (

Table A7. Counts of dropped documents by spider and filtering reason.

Source	Too Short	Duplicate URL	Non-Arabic Detected 1	Non-Arabic Detected 2	Total Dropped
Alukah	4478	0	360	18	4695
Arabic Wiki	1	104	153	0	231
Shamela	1	0	47	35	83
Adab	0	0	14	1	15
Total	4480	104	574	54	5212

¹ Arabic was not confidently detected. ² The text was too short for reliable language identification.

), which summarizes the main reasons for document rejection across spiders and clarifies how source-specific characteristics interacted with the filtering pipeline. A total of 109,633 documents were processed during the crawl. Among them, 104,421 documents were included in the raw data, and 5212 were excluded, resulting in a harvest rate of 95.25% and a drop rate of 4.75%. The crawl-stage harvest ratio was calculated before preprocessing. The overall crawl throughput was 65.7 accepted documents per minute.

6.2. Analysis of FastText Predictions

FastText served as a lightweight quality control layer before more computationally intensive stages such as chunking. Its purpose was not to act as a standalone language-classification benchmark but to reduce unnecessary processing of low-value, irrelevant, or mixed-language pages. This approach was important because the pipeline’s goal was not merely to maximize crawl volume but to improve the quality and usability of the retained Arabic corpus.

6.2.1. Distribution of FastText Top-3 Predictions

As shown in Figure 8a, the accepted documents are predominantly classified as Modern Standard Arabic (MSA) with 104,421 documents assigned this top-1 label. The second- and third-ranked predictions often include other Arabic-script languages such as Persian, Sindhi, Urdu, and South Azerbaijani. This pattern is expected since these languages share script characteristics and partial lexical or orthographic overlap with Arabic. Accordingly, the top-k output should not be interpreted purely as misclassifications but also as evidence that the model distinguishes among closely related script communities while maintaining Arabic as the primary accepted signal.

In contrast, the rejected documents display a markedly different distributional pattern. Figure 8b reveals a substantially higher prevalence of English, Korean, Cantonese, Hindi, and other clearly non-target language labels among the top-ranked predictions. Even when MSA is present, it typically does not constitute the dominant label to a degree sufficient to warrant retention. This pattern suggests that the filter was not merely discarding marginally Arabic content but was also actively removing linguistically inappropriate and mixed-language material. Such findings corroborate the effectiveness of language validation during the crawling process. From a corpus construction standpoint, this behavior is precisely desired: the filtering stage enhances corpus purity by preventing noisy or linguistically ambiguous pages from propagating to subsequent stages of the processing pipeline.

6.2.2. Confidence Score Analysis

The confidence analysis corroborates the interpretation presented above. Figure 9a indicates that the accepted set exhibits extremely high classification confidence for MSA, with a mean top-1 confidence score of 0.999. By contrast, lower-ranked alternative labels within the accepted set generally receive near-zero confidence scores, even when they correspond to closely related Arabic-script languages. This pattern indicates that pages were accepted not simply because Arabic appeared among the top-k predictions, but because it was the dominant label and was assigned consistently high confidence.

Conversely, Figure 9b shows that the discarded subset exhibits a markedly dispersed confidence distribution. In this case, high confidence values are frequently assigned to non-target languages—such as English, Hindi, and Urdu—at the top rank, while the separation between confidence scores at lower ranks is comparatively attenuated. This pattern reflects greater linguistic ambiguity and provides further evidence that the rejected subset contains a substantially higher proportion of non-target language content.

6.2.3. FastText and Rule-Based Filtering Results

To further evaluate the FastText-based language identification component, we conducted an analysis of its outputs generated throughout the crawling and filtering pipelines. In addition, representative samples from the crawl results were subjected to manual inspection in order to assess the adequacy of the selected primary and secondary probability thresholds for the accurate detection of Arabic educational web pages.

Table 6. Examples of filtered crawling results produced by the FastText dual-threshold language identification and filtering rules.

Case	Source	Sample Text	FastText Labels	Scores	Filtering Threshold	Filtering Justification
S1	Arabic Wikipedia	…حساب التفاضل والتكامل من الاختلافات العثور على القيم القصوى للعمليات مشابه …لإيجاد القيم العظمى والصغرى للمعادلات	__label__arb_Arab __label__yue_Hant __label__azb_Arab	0.896 0.046 0.015	Accepted by primary	Arabic is the top prediction and its confidence is above the primary threshold of 0.8.
S2	Arabic Wikipedia	…طرائق ح لانهاية في نظرية التحكم هي أحد طرائق بناء المتحكمات والتي …يمكن تطبيقها على الأنظمة الخطية واللاخطية	__label__arb_Arab __label__yue_Hant __label__azb_Arab	0.883 0.069 0.011	Accepted by primary	Arabic is the top prediction and its confidence is above the primary threshold of 0.8.
S3	Alukah Network	…المهارات والقدرات المستهدفة بتبني نهج التقصي نهج التقصي هو منهجية تعليمية تشجع المتعلمين …من خلال طرح الأسئلة وإجراء البحوث والتحريات	__label__arb_Arab __label__pes_Arab __label__azb_Arab	0.999 0.00004 0.00004	Accepted by primary	The page is clearly Arabic educational content, with very high Arabic confidence.
S4	Arabic Wikipedia	فلوريد الفضة وهو مركب كيميائي …مكون من عنصري الفضة والفلور .mw-parser-output .dmbox{display:flex;…}	__label__arb_Arab __label__yue_Hant __label__azb_Arab	0.537 0.146 0.107	Accepted by secondary	Arabic is still the top prediction, but CSS noise reduces confidence below 0.8. Since the score remains above 0.5, the page is retained.
S5	Arabic Wikipedia	…المُلْتَقَى أو المَقْرَن مكان التقاء مسطحين مائيين @font-face{font-family: “TemplateStyles-Calibri-Quran”; …} ٩٤… (وَتَرَى الْمُجْرِمِينَ يَوْمَئِذٍ مُقَرَّنِينَ فِي الْأَصْفَادِ)	__label__arb_Arab __label__azb_Arab __label__yue_Hant	0.756 0.060 0.045	Accepted by secondary	Arabic is still the top prediction, but CSS templates and citation noise reduce confidence below 0.8. Since the score remains above 0.5, the page is retained.
S6	Adab	لا أيها الزمن، لن تستطيع المباهاة بأن التغير …يطرأ عليّ إن أهراماتك التي شُيِّدَتْ بعزيمة جديدة No, Time, thou shalt not boast that I do change: Thy pyramids built up with newer might…	__label__arb_Arab __label__eng_Latn __label__azb_Arab	0.649 0.332 0.009	Accepted by secondary	Arabic is still the top prediction, but the presence of both Arabic and English translations lowers confidence below 0.8. Since the score remains above 0.5, the page is retained.
S7	Arabic Wikipedia	(Homochiral) …التجانس اليدواني …أو تجانس عدم التناظر المرآتي)	–	–	Rejected	Rejected as duplicate content. The page had already been identified during crawling and was not retained as a unique Arabic page.
S8	Alukah Network	“Kəlimə-i səva (Haqq Söz)” Mesajı Sevimli Peyğəmbər Məhəmməd (s.ə.s.) sünnəsinin əhlindən şiə əhlinə	__label__azj_Latn __label__bak_Cyrl __label__knc_Latn	1.000 0.00001 0.00001	Rejected	Rejected because Arabic was not confidently detected. FastText predicted Azerbaijani Latin with 1.000 confidence, and Arabic was absent from the top three labels. This case was therefore excluded by the language filter.
S9	Shamela Library	قصة الفيل	–	–	Rejected	Rejected because the text was too short for reliable language detection, even though the sample itself contains Arabic words.
S10	Alukah Network	Empty extracted text	–	–	Rejected	Rejected as too short because the extraction produced no usable text for language identification.

presents three filtering categories: documents accepted by the primary threshold, documents retained by the secondary threshold, and documents rejected by the filtering rules. The primary threshold prioritizes precision by directly accepting pages where Arabic is the top FastText prediction with a confidence score of at least 0.8. As shown in examples S1–S3, these cases generally correspond to clean Arabic educational content with minimal structural or linguistic noise.

In contrast, the secondary threshold improves recall by retaining valid Arabic pages whose confidence scores may be lowered by web-specific noise, such as HTML or CSS fragments, metadata, citations, embedded English text, or limited code-switching, as illustrated in examples S4–S6. This secondary category facilitates manual inspection by distinguishing critical cases and potentially valid cases from clearly accepted pages. This differentiation enables us to concentrate on pages requiring further verification rather than treating all accepted pages uniformly.

Finally, the rejected examples (S7–S10) demonstrate that the filtering rules exclude pages for several reasons, including cases where Arabic is not detected among the top predictions, the extracted text is too short for reliable language identification, or the content has already been identified as duplicate.

The use of scraped web text for language identification involves an important trade-off between quality and coverage. The accuracy of language identification can be significantly improved by the implementation of more robust preprocessing and quality control methods, such as the removal of boilerplate, normalization, de-duplication, and the filtering of short or noisy pages. This is because the model receives clearer linguistic signals instead of HTML fragments, repeated menus, symbols, or mixed-content noise.

However, aggressive filtering can reduce data diversity and coverage by discarding short but valid documents, code-switched text, multilingual pages, dialectal content, or rare linguistic forms that may still be valuable for training. To address these issues, the dual-threshold strategy offers a practical compromise.

The primary threshold ensures high-precision acceptance of clearly Arabic pages, while the secondary threshold preserves recall for valid Arabic educational documents affected by noise or limited multilingual content. Thus, assembling a corpus requires balancing quality with quantity. Increasing data integrity improves classification reliability downstream, but overly rigorous cleaning risks losing valuable instances and skewing the corpus toward more standard, well-structured text.

6.3. Analysis of the Web Corpus

6.3.1. Corpus Statistics

Table 7. Statistics of the Arabic educational corpus by processing layer.

Phase/Layer	Statistic	Value
Crawling	Number of crawled pages	109,633
In-crawl filtering	Number of accepted documents	104,421
Preprocessing	Number of retained documents	101,770
Sentence-aware chunking	Number of chunks	286,025
GateNLP	Number of tokens	50,366,982
GateNLP	Number of sentences	289,778
GateNLP	Vocabulary size	684,221
GateNLP	Number of unique lemmas	603,535
GateNLP	Number of named entities	3,429,315

summarizes the scale of the Arabic educational corpus across the main processing layers, including crawling, in-crawl filtering, preprocessing, sentence-aware chunking, and automatic GateNLP-based linguistic annotation. The crawling layer collected 109,633 pages, of which 104,421 documents were accepted during in-crawl filtering. After preprocessing, the final corpus contained 101,770 documents, which were segmented into 286,025 chunks. The GateNLP layer generated linguistic statistics, including 50,366,982 tokens, 289,778 sentences, a vocabulary of 684,221 surface forms, 603,535 unique lemmas, and 3,429,315 named entities. These results demonstrate that the proposed system produced a substantial corpus in terms of volume while also creating a linguistically rich resource for Arabic semantic retrieval and downstream educational search applications.

6.3.2. Sentence and Token Analysis Across Chunks

To assess the structural behavior of the chunking stage, we analyzed both the number of sentences and tokens per chunk. Figure 10a presents the sentence-count distribution, revealing that the vast majority of chunks contain a single sentence (282,459 cases), while 3398 contain two sentences, 151 have three, 15 have four, and only 2 contain five sentences. This pattern indicates that final chunks were typically aligned with single-sentence units, with minimal merging across chunk boundaries.

Figure 10b shows the token-count distribution per chunk. Most chunks fall within a controlled range, with the highest concentration around 200–240 tokens. There is also a lower-frequency spread among shorter chunks and a sharp decline after the main concentration. This suggests that the chunking strategy successfully maintained bounded segment sizes within the specified token constraints. The resulting token-count distribution further indicates that chunk lengths are suitable for downstream transformer-based processing.

6.3.3. Automatic Linguistic Annotation Analysis

The automatic linguistic annotation results further illustrate the internal composition of the corpus. Figure 11a shows the distribution of the top named entity categories. Person-related entities dominate the corpus, with B-PER accounting for 23.60%, E-PER for 21.41%, and I-PER for 20.51% of the top-10 named entity distribution. These are followed by S-PER at 11.87% and location-related entities, such as S-LOC, at 7.11%. This pattern suggests that the corpus contains a substantial amount of person-centered and narrative content, which is consistent with its educational, historical, literary, and religious source material.

Figure 11b shows the distribution of the top 10 POS tags. Nouns are the most frequent category, representing 27.79% of the top-10 POS tags, followed by adpositions at 13.91%, coordinating conjunctions at 11.64%, and verbs at 11.13%. Pronouns, punctuation, particles, and adjectives also appear at notable, even lower, proportions. This distribution is consistent with explanatory and informational texts, where nominal expressions and connective structures are frequently used to organize concepts and discourse. The prominence of nouns, together with the large number of named entities, suggests that the resulting corpus preserves semantically meaningful and structurally rich Arabic content suitable for downstream linguistic analysis.

A notable observation is that 7.79% of tokens were tagged as X, indicating forms that were difficult for the automatic tagger to assign to a more specific universal POS category. These tokens may include foreign proper names, borrowed terms, specialized scientific terminology, or token fragments produced by Arabic tokenization conventions, including the separation of bound morphemes. This result highlights the limitations of fully automatic annotation for heterogeneous Arabic educational web content. Appendix F provides additional material illustrating annotation behavior and lexical diversity in the corpus.

Table A8. Qualitative examples of POS and NER annotation challenges in the linguistic annotation pipeline.

No.	Title	Text Sample	Token	UPOS	Lemma	NER	Annotation Observation
1	الأدب بين نفس المروءة ولهاث الإثارة	لم يكن الأدب العربي بشقيه المنظوم والمنثور إلا مرآة لروح صاحبه، وصورة حية لخلجات نفسه… كما تساءل ابن قتيبة في أدب الكاتب	لم	PART	لَم	O	POS ambiguity in proper names. NER correctly detects ابن قتيبة as PER, but UPOS assigns X to both name tokens. This suggests difficulty with historical Arabic names in literary texts.
			يكن	VERB	كَان	O
			الأدب	NOUN	أَدَب	O
			العربي	ADJ	عَرَبِيّ	O
			ابن	X	ابن	B-PER
			قتيبة	X	قتيبة	E-PER
2	منهج القرآن الكريم في تنمية التفكير التأملي	الخاتمة: يبرهن القرآن الكريم على أن التفكير التأملي ليس ترفا ذهنيا، بل ضرورة تربوية لصناعة إنسان مسؤول وقادر على اتخاذ قرارات مستنيرة	الخاتمة	NOUN	خَاتِمَة	O	The religious phrase is treated as lexical material, not a named entity. This is not necessarily incorrect, but scripture and religious references may need clearer NER guidelines.
			يبرهن	VERB	أَبرَه	O
			القرآن	NOUN	قُرآن	O
			الكريم	ADJ	كَرِيم	O
			التفكير	NOUN	تَفكِير	O
3	فروع الكيمياء	ويدرس بنية وخواص وتفاعلات المركبات والمواد العضوية التي تحتوي على عنصر الكربون ووضع ميخائيل لومونوسوف كتابا في الكيمياء الفيزيائية	الكربون	NOUN	كَربُون	S-MISC	POS ambiguity in foreign names. ميخائيل لومونوسوف is detected as PER, but assigned X in UPOS. This suggests difficulty tagging foreign proper names in scientific text.
			ميخائيل	X	مِيخَائِيل	B-PER
			لومونوسوف	X	لومونوسوف	E-PER
			الكيمياء	NOUN	كِيمِيَاء	B-MISC
			الفيزيائية	ADJ	فِيزِيَائِيّ	E-MISC
4	فورميمينو-٥ ثلاثي هيدروفولات	فورميمينو ثلاثي هيدروفولات هو مركب وسطي ينتج من عملية الهدم للحمض الأميني هستيدين بوساطة الغلوتاميت فورماميدويلترانسفيريز	فورميمينو	X	فورميمينو	S-MISC	Biomedical terms are mostly tagged as X in UPOS and MISC in NER. Main issue: specialized chemical and biochemical vocabulary lacks fine-grained entity categories.
			هيدروفولات	X	هيدروفولات	S-MISC
			هستيدين	X	هستيدين	S-MISC
			الغلوتاميت	X	الغلوتاميت	S-MISC
			سايكلوديامينيز	X	سايكلوديامينيز	S-MISC
5	أثر التناقض اللفظي في المعنى	هو أسلوب أدبي يجمع بين كلمتين متناقضتين ظاهريا لتشكيل مفهوم مثير للتفكير كلمة أوكسيمورون مشتقة من أوكسيس وموروس	هو	PRON	هُوَ	O	POS variation in borrowed terms. Terms such as أوكسيمورون, أوكسيس, and موروس are labeled as MISC, while UPOS varies between X and NOUN. This suggests ambiguity in assigning POS tags to foreign-origin or borrowed technical terms in Arabic text.
			أسلوب	NOUN	أُسلُوب	O
			أدبي	ADJ	دِبِّيّ	O
			يجمع	VERB	جَمَع	O
			أوكسيمورون	X	أوكسيمورون	S-MISC
			أوكسيس	X	أوكسيس	S-MISC
			موروس	NOUN	مَورُوس	S-MISC

Note. UPOS refers to the universal part-of-speech tag; NER refers to the named entity recognition label. B, I, E, S, and O denote beginning, inside, end, single-token entity, and outside entity, respectively. PER, LOC, and MISC refer to person, location, and miscellaneous entity types. The tag X indicates that the token was not assigned to a more specific universal POS category. These examples summarize common annotation challenges, including morphological ambiguity, tokenization issues, borrowed terminology, foreign proper names, and domain-specific vocabulary.

presents qualitative examples of POS and NER annotation challenges, including X-tag assignments, foreign proper names, borrowed terminology, and domain-specific vocabulary. Figure A1 further illustrates lexical diversity through a TF-IDF word cloud visualization.

6.3.4. Results of the Sentence-Aware Chunking Strategy

To further evaluate the proposed sentence-aware chunking strategy, we assessed the semantic consistency of adjacent chunks using Arabic sentence embeddings. Specifically, chunk embeddings were generated with the GATE-AraBert-v1 model [67], and cosine similarity was calculated for consecutive chunks within the same document.

Figure 12 illustrates the distribution of consecutive-chunk cosine similarity scores. Most scores fall within the moderate to high range, peaking between 0.65 and 0.75. This indicates that consecutive chunks generally maintain meaningful semantic continuity rather than being arbitrarily segmented. Simultaneously, the distribution is not overly concentrated near 1.0, suggesting that consecutive chunks remain distinct textual units rather than redundant overlaps.

Figure 13 presents representative annotated similarity heatmaps for selected documents containing varying numbers of chunks. In these examples, adjacent or nearby chunks generally display higher similarity compared to more distant chunks, while still preserving meaningful variation within each document. These results provide further evidence supporting the validity of the proposed chunking method. The sentence-count and token-count distributions have already demonstrated that the chunking process yields bounded retrieval units, and the embedding-based similarity analysis further indicates that these segmented chunks preserve local semantic continuity across documents.

Table 8. Chunk Statistics for Selected Strategies After 20–450 AraBERT Token Filtering.

Chunking Method	Chunks	Docs	Qrels	Avg. Tokens	Median	Min	Max
CharacterTextSplitter	1301	494	7476	265.33	301	22	442
RecursiveCharacterTextSplitter	1044	494	7476	333.91	449	22	450
Sentence-Aware (Proposed)	1291	494	7476	263.50	290	21	450

Note. Qrels denotes the number of query-relevance judgments used for retrieval evaluation.

presents the chunk distributions obtained after applying the 20–450 AraBERT token filter, which are essential for interpreting retrieval results, as different chunking methods yield varying numbers and lengths of chunks. CharacterTextSplitter and the proposed sentence-aware method produced nearly identical chunk distributions: CharacterTextSplitter generated 1301 chunks with an average length of 265.33 AraBERT tokens, while the sentence-aware method produced 1291 chunks averaging 263.50 tokens each. This close alignment facilitates direct comparison between the fixed-size baseline and the proposed method.

In contrast, RecursiveCharacterTextSplitter generated fewer chunks with longer passages, with a median length of 449 tokens, likely improving recall metrics by encompassing a broader context in each retrieved chunk.

Retrieval effectiveness was evaluated using the ranx library based on 7476 query-relevance judgments for each chunking strategy.

Table 9. Comparison of chunking approaches in retrieval tasks using four embedding models.

Embedding Model	Dim.	Chunking Method	Hit Rate			Precision			Recall			MRR@10	MAP@10	nDCG@10
			@1	@5	@10	@1	@5	@10	@1	@5	@10
E5-large	1024	Character	0.5091	0.6499	0.7041	0.5091	0.2694	0.1820	0.1974	0.3611	0.4282	0.5699	0.3470	0.4354
		Recursive	0.4831	0.6311	0.6851	0.4831	0.2415	0.1578	0.2153	0.3901	0.4566	0.5463	0.3642	0.4379
		Sentence-aware	0.5185	0.6540	0.7081	0.5185	0.2725	0.1859	0.1938	0.3534	0.4252	0.5771	0.3442	0.4358
MiniLM	384	Character	0.0736	0.1471	0.1907	0.0736	0.0368	0.0271	0.0420	0.0757	0.0947	0.1058	0.0606	0.0816
		Recursive	0.0602	0.1170	0.1552	0.0602	0.0274	0.0201	0.0423	0.0757	0.0940	0.0850	0.0594	0.0752
		Sentence-aware	0.0940	0.1663	0.2063	0.0940	0.0490	0.0360	0.0438	0.0820	0.1049	0.1253	0.0674	0.0946
Ara- Matryoshka	768	Character	0.4176	0.5700	0.6328	0.4176	0.2375	0.1672	0.1624	0.3188	0.3920	0.4825	0.3036	0.3829
		Recursive	0.4073	0.5564	0.6264	0.4073	0.2209	0.1499	0.1811	0.3508	0.4276	0.4719	0.3270	0.3953
		Sentence-aware	0.4347	0.5766	0.6422	0.4347	0.2434	0.1746	0.1620	0.3132	0.3942	0.4969	0.3050	0.3898
Ara-MPNet	768	Character	0.1807	0.3253	0.4012	0.1807	0.0891	0.0632	0.0808	0.1489	0.1872	0.2420	0.1212	0.1693
		Recursive	0.1497	0.2925	0.3731	0.1497	0.0755	0.0535	0.0795	0.1531	0.1956	0.2115	0.1206	0.1628
		Sentence-aware	0.1958	0.3456	0.4255	0.1958	0.0976	0.0691	0.0830	0.1540	0.1935	0.2608	0.1263	0.1787

Note. Best results for each embedding model are highlighted in boldface. Dim. indicates the embedding dimension. Model abbreviations: E5-large: multilingual-E5-large; MiniLM: paraphrase-multilingual-MiniLM-L12-v2; Ara-Matryoshka: Arabic-Triplet-Matryoshka-V2; Ara-MPNet: Arabic-mpnet-base-all-nli-triplet. Chunking abbreviations: Character: CharacterTextSplitter; Recursive: RecursiveCharacterTextSplitter.

presents the retrieval performance of the three chunking strategies across four embedding models. The comparative results reveal notable differences in retrieval effectiveness, both across chunking strategies and among embedding models. The sentence-aware chunking method provides the most consistent improvements in top-k retrieval performance. In particular, it achieves the highest Hit Rate@1, Hit Rate@5, Hit Rate@10, and MRR@10 across all four embedding models. For multilingual-E5-large, the sentence-aware method achieves a Hit Rate@10 of 0.708 and an MRR@10 of 0.577, outperforming both CharacterTextSplitter and RecursiveCharacterTextSplitter.

Despite these strengths, RecursiveCharacterTextSplitter achieves the highest recall scores and, in some cases, slightly higher MAP@10 and nDCG@10, particularly with multilingual-E5-large and Arabic-Triplet-Matryoshka-V2. This advantage stems from its tendency to produce longer segments, as shown in Table 8. Longer chunks frequently capture more document context, increasing the likelihood of retrieving a relevant passage, especially at higher ranks. However, this approach incurs a trade-off in chunk specificity and precision at the top ranks.

A detailed comparison of the embedding models provides further insight. Multilingual-E5-large achieves the best overall performance, likely due to its larger model capacity and multilingual pretraining. Arabic-Triplet-Matryoshka-V2 also performs competitively, highlighting the effectiveness of Arabic-specific triplet-based contrastive training. In contrast, paraphrase-multilingual-MiniLM-L12-v2, which uses smaller 384-dimensional embeddings, and arabic-mpnet-base-all-nli-triplet, despite using 768-dimensional embeddings, achieve lower absolute retrieval scores than the two leading models. Nevertheless, the sentence-aware strategy consistently improves ranking performance over the fixed-size baseline across all evaluated embedding models, indicating that preserving sentence boundaries benefits retrieval regardless of the underlying encoder.

Importantly, the sentence-aware method achieves chunk lengths and document coverage comparable to the fixed-size CharacterTextSplitter, enabling fair comparison and practical deployment. By preserving sentence boundaries, the method increases the likelihood of retrieving relevant passages near the top of the ranked list. This is particularly important for RAG and QA applications, where the quality of top-ranked evidence strongly affects downstream performance.

In summary, the experiments demonstrate that sentence-aware chunking consistently enhances retrieval quality across embedding models and outperforms the fixed-size baseline. While recursive chunking can improve recall by retrieving longer passages, the sentence-aware approach provides a more effective balance of precision, coherence, and downstream utility for Arabic educational retrieval tasks.

7. Limitations and Future Work

7.1. Limitations

Several limitations should be acknowledged. First, the corpus was constructed from a focused set of domains and thus does not offer exhaustive coverage of the Arabic educational web. The framework prioritizes targeted quality and source relevance over unrestricted web-scale breadth. Although AraEdu-WC draws from educationally relevant and structurally diverse Arabic sources, the current version remains limited to a curated set of websites. As a result, the corpus may reflect the topical focus, writing style, and source distribution of these selected platforms.

Second, extraction robustness may vary across websites with unstable layouts or dynamically rendered content, even with AI-assisted XPath support. While the framework improves adaptability, it does not eliminate maintenance requirements when source structures evolve. Although LLM-assisted XPath generation was successfully applied in practice to support multi-site extraction, it was not instrumented as a standalone experimental variable during corpus construction.

Third, linguistic annotation remains automatic and inherits model uncertainty, as indicated by the skewed proportion of tokens assigned to the X tag. A further limitation is the absence of manually verified gold-standard annotations in the current version of AraEdu-WC. The POS tags, lemmas, and named entities in the corpus were generated automatically using state-of-art NLP tools. Although Appendix F provides qualitative examples of annotation challenges, manually verified gold-standard annotations are not included. These automatic annotations, while useful for exploratory analysis, may vary in accuracy across domains and text types.

Fourth, we evaluated the proposed sentence-aware strategy against two baselines using four different embedding models. The results emphasize the necessity of careful dataset-specific calibration. The selected 20–450 AraBERT-token range was determined by manual inspection of AraEdu-WC chunks, not as a generalizable standard. This finding aligns with prior work that emphasizes that dataset characteristics significantly influence retrieval behavior and chunk-size effectiveness [60,71]. For other domains, such as legal, biomedical, or conversational educational data, the minimum and maximum token thresholds may require adjustment.

Additionally, the evaluation is limited by the absence of a comprehensive Arabic long-document chunking benchmark. Bhat et al. [71] similarly highlight that existing datasets may not fully capture real-world retrieval scenarios and that some benchmarks lack detailed relevance annotations or rely on synthetic structures. In this study, available Arabic datasets did not include long educational documents paired with human-written queries, graded relevance judgments, and human-labeled evidence sentences. As a result, we employed a controlled query-document subset from AraEdu-WC with generated queries. While this approach enables a practical and reproducible evaluation, future work should focus on developing Arabic long-document benchmarks featuring human-authored queries and evidence annotations to more directly assess chunk quality.

Finally, the experiments were designed to evaluate the framework as a corpus construction pipeline rather than as a benchmark crawler. Consequently, the reported results should be interpreted primarily in terms of practical corpus engineering value, encompassing acquisition quality, structuring, and downstream usability.

7.2. Future Work

Several directions can be pursued to extend this work. First, AraEdu-WC could be expanded by incorporating additional Arabic educational sources, such as university repositories, governmental educational platforms, MOOCs, and interactive learning environments. This expansion would enhance topic coverage, mitigate source distribution bias, and further assess the adaptability of ArabicEduCrawler across diverse website structures.

Second, future work should create a manually verified subset of AraEdu-WC to evaluate the quality of annotations, including POS tagging, lemmatization, and named entity recognition. Such a resource would enable more systematic error analysis and serve as a valuable benchmark for Arabic educational NLP.

Third, retrieval evaluation can be advanced by constructing an Arabic long-document educational benchmark that includes human-written queries, graded relevance judgments, and evidence annotations.

Finally, the resulting corpus facilitates a range of downstream applications, including Arabic semantic retrieval, retrieval-augmented generation (RAG), and question-answering (QA) systems. In this regard, ArabicEduCrawler provides both a practical resource and a methodological foundation for future research in Arabic NLP and information access.

8. Conclusions

The limitations of Arabic educational search extend beyond retrieval effectiveness to include the scarcity of domain-specific corpora constructed through reproducible, NLP-oriented collection pipelines. While much previous Arabic NLP research relies on pre-existing datasets, less attention has been paid to systems that acquire domain-specific web content and transform heterogeneous online materials into structured resources. This paper addresses that gap by presenting ArabicEduCrawler, an integrated framework for extracting, processing, and constructing Arabic educational content.

We introduced ArabicEduCrawler, an AI-assisted focused crawling framework for constructing Arabic educational web corpora from heterogeneous online sources. The framework moves beyond simple page collection by integrating focused crawling, Arabic-aware language filtering, LLM-assisted XPath generation and validation, metadata preservation, text cleaning, sentence-aware chunking, and automatic linguistic enrichment within a unified corpus-engineering workflow. By incorporating GateNLP and Stanza for automatic annotation, the system produces a structured and reusable corpus that supports downstream Arabic NLP and retrieval applications. In this work, corpus acquisition is treated as a central methodological contribution rather than as a preliminary preprocessing step.

In our experiments, the framework collected content from four source domains using 11 seed URLs, resulting in a final retained corpus of 101,770 source documents. After preprocessing and segmentation, this corpus yielded 286,025 chunks, comprising 50,366,982 tokens and 289,778 sentences. The resulting dataset contains 684,221 vocabulary terms, 603,535 distinct lemmas, and 3,429,315 named entities. Additionally, the chunking evaluation demonstrated that sentence-aware segmentation preserved local semantic continuity across adjacent chunks.

The downstream chunking evaluation further demonstrates the effectiveness of the proposed sentence-aware strategy for Arabic educational retrieval. Compared to the fixed-size Character Text Splitter and Recursive Character Text Splitter baselines, the sentence-aware method achieved the highest Hit Rate@1, @5, @10; Precision@1, @5, @10; and MRR@10 across four embedding models. These results indicate that preserving sentence boundaries enables the retrieval of relevant evidence closer to the top of the ranked list. At the same time, the comparison reveals that longer recursive chunks can improve recall-oriented metrics, highlighting a trade-off between compact, top-ranked evidence and broader contextual coverage. Overall, sentence-aware chunking offers a practical balance among coherence, compactness, and retrieval effectiveness for Arabic educational corpora.

The evaluation also demonstrates the framework’s value across three practical dimensions. First, regarding crawl efficiency, a small but targeted seed configuration proved sufficient to generate a substantial educational corpus. Second, in terms of filtering effectiveness, Arabic-aware filtering and acquisition-stage quality control successfully reduced untargeted and low-value content prior to subsequent processing. Third, with respect to corpus development, the system went beyond mere page collection by transforming content into a structured resource for retrieval and analysis, preserving metadata, producing cleaned text, and enabling chunk-level representation.

Overall, the experimental results demonstrate that the proposed approach can transform heterogeneous live web sources into a structured, reusable, and practical Arabic educational corpus for downstream NLP and retrieval tasks.