Collecting and Analyzing IBD Clinical Data for Machine-Learning: Insights from an Italian Cohort

Abstract

Research of Inflammatory Bowel Disease (IBD) involves integrating diverse and heterogeneous data sources, from clinical records to imaging and laboratory results, which presents significant challenges in data harmonization and exploration. These challenges are also reflected in the development of machine-learning applications, where inconsistencies in data quality, missing information, and variability in data formats can adversely affect the performance and generalizability of models. In this study, we describe the collection and curation of a comprehensive dataset focused on IBD. In addition, we present a dedicated research platform. We focus on ethical standards, data protection, and seamless integration of different data types. We also discuss the challenges encountered, as well as the insights gained during its implementation.

1. Introduction

Inflammatory Bowel Disease (IBD) research spans a wide range of disciplines, encompassing epidemiology, basic research, molecular bioscience, translational research, and the analysis of patient data to evaluate and compare therapeutic interventions. IBD care similarly integrates multiple fields, including ’omics’ sciences like genomics, to support comprehensive diagnostic and therapeutic approaches [1,2]. Within this realm, the structured collection and management of IBD data are crucial for advancing both research efforts and the development of machine-learning applications [2,3,4,5]. A well-organized repository of IBD data allows for training machine-learning algorithms, enabling the creation of sophisticated diagnostic tools and predictive models [6]. Moreover, curated datasets form the foundation for rigorous research studies, providing insights into disease mechanisms, treatment efficacy, and patient outcomes [2]. However, in a fully operational hospital setting, IBD data originate from diverse and often heterogeneous sources, creating challenges in harmonizing and standardizing this extensive information for research purposes. IBD patient records include a variety of data types—clinical records, imaging, pathology reports, and laboratory results—each with unique formats and coding standards. Integrating these disparate sources into a unified repository requires advanced data management strategies and interoperability solutions [3]. Additionally, IBD data frequently reside in silos across various departments and hospital systems, further complicating data organization efforts. Researchers often face fragmented data landscapes, making it challenging to access relevant information and slowing the pace of research. Establishing strong data governance frameworks and interoperability standards is essential to streamline data integration and promote collaborative research [7]. Despite these complexities, the benefits of a centralized IBD data repository are substantial. A standardized dataset improves data accessibility, promotes collaboration, and accelerates scientific discovery and clinical innovation. Organized IBD data enable researchers to uncover new insights into disease pathology, assess treatment outcomes, and understand patient responses, ultimately enhancing the quality of care of individuals affected by IBD.

This paper introduces the Humanendo-IBD dataset, a structured collection of heterogeneous data sources focused on IBD. The data set includes diverse types of information, from clinical records, imaging, and pathology reports to laboratory results, organized to ensure consistency, precision, and reliability. The dataset is designed to support both clinical and research applications, promote interoperability, and enable cross-institutional studies in IBD research. It can support several research questions, including: (i) which biomarkers predict response to biological therapy over time? (ii) Can baseline clinical or histological features identify patient subgroups with differing prognoses? (iii) How do clinical and laboratory variables evolve at follow-up in responders versus non-responders? (iv) Can histopathology images be used to discover features associated with inflammation severity or therapeutic resistance? In addition, we discuss the challenges encountered and the lessons learned throughout the data collection and curation process, addressing practical considerations for building a robust and privacy-compliant IBD dataset. While the Humanendo-IBD dataset is not currently publicly available due to patient privacy concerns and institutional restrictions, access can be granted for collaborative research upon reasonable request and following ethical approval. We acknowledge that open access would enhance reproducibility and are exploring GDPR-compliant solutions such as data enclaves or federated analysis platforms.

The paper is organized as follows: in Section 2, we describe the types of data collected and the technical details of the platform implementation with a focus on the user experience and privacy and security issues. In Section 3, we describe our dataset using basic statistics. Finally, in Section 4, we discuss our results, and we draw our conclusions in Section 5.

2. Materials and Methods

IBD activities generate a large set of heterogeneous data from different sources. Organizing and linking such data efficiently is critical for enabling research activities. Our dataset currently aggregates the following data types:

• Clinical and demographic data sourced from clinical care systems, including endoscopic and treatment information
• Digital pathology data, including histopathology images acquired at baseline

• It is worth noticing that data are collected over time to allow the track of the evolution of the clinical situation. Specifically, the dataset comprises clinical and biological information from patients diagnosed with IBD, specifically Crohn’s disease (CD) and ulcerative colitis (UC). Data are organized into two main components: baseline demographic and clinical data, as well as treatment information, with a follow-up recorded one year after baseline (±3 months).

  Table 1. Overview of the Humanendo-IBD dataset structure.

  Data Component	Variable	Description
  Clinical Data	gender	Patient gender
  	patient_weight	Patient weight at baseline (kg)
  	patient_height	Patient height (cm)
  	date_of_birth	Date of birth
  	baseline_diagnosis	Diagnosis (CD or UC)
  	baseline_date_of_diagnosis	Date of initial diagnosis
  	baseline_age_at_diagnosis	Age at initial diagnosis (years)
  	baseline_history_of_autoimmune	History of autoimmune diseases (Yes/No)
  	baseline_family_history	Family history of IBD (Yes/No)
  	smoking_status	Smoking status (Yes/No)
  	number_of_stools_per_day	Number of stools per day
  	abdominal_pain	Presence of abdominal pain (Yes/No)
  	abdominal_mass	Presence of abdominal mass (Yes/No)
  	rectal_bleeding	Presence of rectal bleeding (Yes/No)
  	arthralgia	Presence of joint pain (Yes/No)
  	ocular_involvement	Ocular involvement (Yes/No)
  	pyodema_gangrenosum	Presence of pyoderma gangrenosum (Yes/No)
  	erythema_nodosum	Presence of erythema nodosum (Yes/No)
  	new_fistula	Presence of a new fistula (Yes/No)
  	harvey_bradshaw_index	Harvey–Bradshaw Index for CD patients
  	mayo_score	Mayo score for UC patients
  Laboratory Data	c_reactive_protein_crp	CRP level (mg/dL)
  	fecal_calprotectin_fc	Fecal calprotectin level (µg/g of stool)
  	biology_hemoglobin	Hemoglobin level (g/dL)
  	biology_platelet_count	Platelet count ($\times 109$/L)
  	biology_serum_creatinine_value	Serum creatinine value (mg/dL)
  	biology_ferritin_level_value	Ferritin level (ng/mL)
  	ast_aspartate_amino_value	Aspartate Aminotransferase level (U/L)
  	alt_alanine_trans_value	Alanine Aminotransferase level (U/L)
  	ggt_gamma_glutamyl_transf_value	Gamma-Glutamyl Transferase level (U/L)
  	alp_alkaline_phosphatase_value	Alkaline Phosphatase level (U/L)
  	biology_albumin_value	Albumin level (g/dL)
  	biology_vitamin_b12_value	Vitamin B12 level (pg/mL)
  	biology_vitamin_d_value	Vitamin D level (ng/mL)
  Endoscopy Data	endoscopy_type	Type of endoscopy performed
  	visit_date_endoscopy	Date of endoscopy
  	aphthous	Presence of aphthous ulcers (Yes/No)
  	ulcerations	Presence of ulcerations (Yes/No)
  	stenosis_level	Level of stenosis if present
  	stenosis_class	Classification of stenosis (e.g., inflammatory, fibrotic)
  	uc_extension	Disease extension in UC patients
  Virology Data	viral_sierology_hbv	Hepatitis B virus serology (Positive/Negative)
  	viral_sierology_hcv	Hepatitis C virus serology (Positive/Negative)
  	viral_sierology_ebv	Epstein-Barr virus serology (Positive/Negative)
  Pathology and Imaging Data	pathology_digitised_he	Digitized H&E stained histopathological images
  	biopsy_location	Biopsy location
  Treatment Data	aminosalicylates_admin	Aminosalicylate administration (Yes/No)
  	aminosalicylates_type	Type of aminosalicylate administered
  	aminosalicylates_start_date	Start date of aminosalicylate treatment
  	aminosalicylates_end_date	End date of aminosalicylate treatment
  	aminosalicylates_dose	Dosage of aminosalicylate
  	aminosalicylates_interruption	Treatment interruption (Yes/No)
  	antibiotics_admin	Antibiotic administration (Yes/No)
  	antibiotics_start_date	Start date of antibiotic treatment
  	antibiotics_end_date	End date of antibiotic treatment
  	antibiotics_dose	Dosage of antibiotics
  	steroids_admin	Steroid administration (Yes/No)
  	steroids_type	Type of steroid administered
  	steroids_start_date	Start date of steroid treatment
  	steroids_end_date	End date of steroid treatment
  	steroids_dose	Dosage of steroids
  	steroids_interruption	Steroid treatment interruption (Yes/No)
  	immunomodulator_admin	Immunomodulator administration (Yes/No)
  	immunomodulator_type	Type of immunomodulator used
  	immunomodulator_start_date	Start date of immunomodulator treatment
  	immunomodulator_end_date	End date of immunomodulator treatment
  	anti_tnf_admin	Anti-TNF therapy administration (Yes/No)
  	anti_tnf_type	Type of anti-TNF therapy
  	anti_tnf_secondary_loss	Secondary loss of response (Yes/No)

  provides a summary of the data structure. Adult patients (>18 years) with active luminal UC or CD were eligible. Inclusion required treatment with anti-TNF-$\alpha$ agents (biosimilars included) or Vedolizumab as first-line therapy, or Ustekinumab/Tofacitinib (UC) or Ustekinumab/Vedolizumab (CD) as second-line therapy. Patients must have undergone baseline endoscopy/biopsies under biologics, regardless of treatment start date. Exclusion criteria included lack of follow-up and prior colectomy for UC.

The baseline data include demographic variables such as gender, date of birth, and smoking status, alongside the initial diagnosis and diagnosis date. Family history of autoimmune conditions and specific clinical metrics are recorded, including weight, height, and disease-specific scores like the Mayo score for UC and the Harvey–Bradshaw Index for CD. Symptom data are also available, detailing indicators such as abdominal pain, rectal bleeding, and extra-intestinal manifestations (e.g., ocular involvement and skin conditions). The dataset further incorporates detailed laboratory and endoscopic data at baseline, including C-reactive protein (CRP) levels, fecal calprotectin, hemoglobin, serum creatinine, and liver enzymes. Endoscopic findings specify lesion characteristics such as the presence of ulcers and stenosis level. Additionally, histopathological data includes digitized hematoxylin and eosin (H&E) stained images from biopsy samples collected at baseline, supporting the examination of tissue morphology and potential biomarker studies. The treatment-data component provides comprehensive information on the administration of medications commonly prescribed for IBD management. This includes records on the use of aminosalicylates, antibiotics, corticosteroids, immunomodulators, and biologics (e.g., anti-TNF, anti-IL, and JAK inhibitors). For each therapeutic category, details such as the specific drug type, start and end dates, dosages, and any treatment interruptions or secondary loss of response are documented. These entries facilitate the longitudinal assessment of therapeutic strategies, patient adherence, and medication efficacy. At the follow-up, the dataset tracks clinical outcomes and updates on patient health status, including changes in clinical scores, symptom progression, and updated laboratory values. Each patient had one piece of complete follow-up data at approximately one year (±3 months). By capturing data at both baseline and follow-up, the dataset enables a robust evaluation of disease progression, therapeutic impact, and the identification of predictive biomarkers.

2.1. Data Source Harmonization

In pursuing data harmonization, the initial phase involves a systematic examination and delineation of existing data reservoirs. Each group undertakes a methodical exploration of the retrospective datasets, accurately documenting their respective attributes. This documentation includes categorical data composition, variable parameters, patient cohort size, and temporal extent. Next, a comprehensive data dictionary is compiled, with the goal of delineating a core of shared information across units. Another important step is to ensure the congruence of data sources, which demands a substantial technical investment, as it requires an evaluation of the various clinical data collection tools and institutional databases. Critical to this effort is the creation of a universal patient identifier, which serves as a pivot for integration efforts. By leveraging this identifier, diverse data streams are seamlessly unified. Missing data were not explicitly handled as we retained all entries and tracked missingness patterns across variables. It is worth noticing that such patterns are mainly related to systematic gaps due to free-text entries and non-standard documentation of clinical history, particularly for smoking status, symptom reports and some laboratory values not relevant for the diagnosis. The visual summary of the steps of the pipeline is illustrated in Figure 1.

2.2. Software Components and Technical Details

The platform uses Microsoft Azure as the core infrastructure, leveraging its Data Lake capabilities as the DBMS for scalable and secure data storage. Azure services enable efficient handling of large, heterogeneous datasets. Python (https://python.org) was chosen for its rich ecosystem and strong compatibility with Azure tools for data processing and automation. These technologies provided an optimal balance between interoperability, scalability, and ease of integration with AI workflows. Extraction/Transform/Loading (ETL) pipeline is set up for the different data sources. Oracle SQL serves as the primary querying language for accessing and retrieving data from the hospital database, ensuring efficient and reliable data retrieval. Concerning digital pathology, the Philips Digital Pathology suite (https://www.philips.it/healthcare/solutions/pathology, accessed on 18 June 2025) is used for the management and analysis of digital pathology data, leveraging its advanced features for handling high-resolution images and extracting valuable insights from pathological specimens.

The Oracle SQL database was used as it is the standard DBMS integrated with our hospital’s electronic systems, ensuring secure, stable, and performant querying for large-scale clinical data.

2.3. Technical Details and Privacy

The platform runs on a set of secure Linux-based cloud virtual machines: one for the database and one for the platform back-end. The machines are equipped with 8 cores and 16 GB RAM. Data retrieval and harmonization procedures are implemented on premise. Pseudo-anonymization is performed on the harmonized data before being uploaded on the platform database according to the European Data Protection Board guidelines [8]: for each patient, all the directly identifying data are removed. Only month and year are left in for dates, shifted by a random number of days between −365 and 365. De-identification information is never shared outside the hospital’s servers.

2.4. Ethics, Legal, and Data-Property Issues

The study was conducted in accordance with the Declaration of Helsinki. Ethical approval was obtained from the ethics committee at IRCCS Humanitas Research Hospital. Patients participating in the proof of concept study were provided with informed consent and were informed of their right to withdraw at any point. Platform users, including clinicians and researchers, are required to accept a user agreement outlining basic ethical principles. It is important to note that data access is restricted to users who have been authorized by the hospital.

3. Results

In this section, we present a comprehensive summary of the dataset, showcasing its basic statistics.

Table 2. Ulcerative colitis (UC) patient summary of baseline and follow-up data.

Category	Female (F)-Baseline	Male (M)-Baseline	Female (F)-Follow-Up	Male (M)-Follow-Up
Gender Distribution	35 (36%)	63 (64%)
Average Age	35.92 years	30.83 years
Age Groups	18–30 (21), 31–50 (42), 51–70 (21), 71+ (14)
Height Groups	<150 cm (2), 150–170 cm (46), >170 cm (50)
Patient Weight (kg)	Mean: 59.37, Range: 43.0–88.0	Mean: 73.35, Range: 49.0–99.0	Mean: 59.37, Range: 43.0–88.0	Mean: 73.60, Range: 49.0–99.0
Patient Height (cm)	Mean: 163.47, Range: 148.0–182.0	Mean: 174.75, Range: 160.0–192.0
Age at Diagnosis (years)	Median: 35.92, Range: 11–74	Median: 30.83, Range: 0–77
Mayo Score	Median: 3.0, Max: 3.0, Min: 2.0	Median: 3.0, Max: 3.0, Min: 2.0	Median: 1.0, Max: 3.0, Min: 0.0	Median: 1.5, Max: 3.0, Min: 0.0

and

Table 3. Crohn’s disease (CD) patient summary of baseline and follow-up data.

Category	Female (F)-Baseline	Male (M)-Baseline	Female (F)-Follow-Up	Male (M)-Follow-Up
Gender Distribution	41 (36%)	61 (62%)
Average Age	32.17 years	33.38 years
Age Groups	18–30 (21), 31–50 (42), 51–70 (21), 71+ (14)
Height Groups	<150 cm (2), 150–170 cm (46), >170 cm (50)
Patient Weight (kg)	Mean: 63.61, Range: 41.0–104.0	Mean: 74.38, Range: 45.0–105.0	Mean: 59.37, Range: 43.0–88.0	Mean: 73.60, Range: 49.0–99.0
Patient Height (cm)	Mean: 162.78, Range: 150.0–180.0	Mean: 175.52, Range: 160.0–198.0
Age at Diagnosis (years)	Median: 32.0, Range: 11–54	Median: 33.0, Range: 11–64
Harvey–Bradshaw Index	Median: 5.5, Max: 17.0, Min: 0.0	Median: 3.0, Max: 13.0, Min: 0.0	Median: 3.0, Max: 14.0, Min: 0.0	Median: 2.0, Max: 12.0, Min: 0.0

report demographic distributions along with Mayo score (for UC) and Harvey–Bradshaw Index (for CD) at baseline and follow-up.

Table 4. Ulcerative colitis (% UC) status summary (baseline and follow-up).

Category	Status at Baseline	Status at Follow-Up
Smoking Status	Non-Smoker (46), Former Smoker (34), Active Smoker (16), Unknown (3)	Non-Smoker (43), Former Smoker (1), Active Smoker (14), Unknown (4)
Number of Stools per Day (Median)	3.0	2.0
Number of Stools per Day (Min–Max)	2–7	1–10
Family History	13
History of Autoimmune Disease	19
Pyodema Gangrenosum	0	0
Erythema Nodosum	0	0
New Fistula	0	0
Patients Suffering Aphthous	2	0
Patients Suffering Ulcerations	1	99
Patients Suffering Abdominal Pain	66	37
Patients Suffering Rectal Bleeding	64	26
Patients Suffering Arthralgia	57	47
Patients Suffering Ocular Involvement	0	0
Patients Suffering Pyodema Gangrenosum	0	0
Patients Suffering Erythema Nodosum	0	0
Patients Suffering New Fistula	99	0

and

Table 5. Crohn’s disease (CD) status summary (baseline and follow-up).

Category	Status at Baseline	Status at Follow-Up
Smoking Status	Non-Smoker (38), Former Smoker (34), Active Smoker (31), Unknown (3)	Non-Smoker (38), Former Smoker (30), Active Smoker (34), Unknown (3)
Number of Stools per Day (Median)	2.0	2.0
Number of Stools per Day (Min–Max)	1–20	1–10
Family History	15
History of Autoimmune Disease	11
Pyodema Gangrenosum	0	0
Erythema Nodosum	0	0
New Fistula	0	0
Patients Suffering Aphthous	0	0
Patients Suffering Ulcerations	0	0
Patients Suffering Abdominal Pain	66	37
Patients Suffering Rectal Bleeding	64	26
Patients Suffering Arthralgia	57	47
Patients Suffering Ocular Involvement	0	0
Patients Suffering Pyodema Gangrenosum	0	0
Patients Suffering Erythema Nodosum	0	0
Patients Suffering New Fistula	0	0

capture the overall patient status, including smoking status, stool frequency, family history, autoimmune associations, as well as the overall presence of IBD sympthoms at baseline and follow-up. Finally,

Table 6. Summary of biological and treatment information for ulcerative colitis (UC) patients, including baseline and follow-up measurements of various biomarkers and treatment types.

Category	Patients	Mean (Std)	Range
TREATMENT INFO
Aminosalicylates	19	3331.0 (2799.21)	59.0–8135.0
Antibiotics	1	304.0 (0)	304.0–304.0
Steroids	72	500.75 (1220.79)	2.0–6962.0
Immunomodulator	49	837.14 (1352.2)	1.0–6848.0
Anti-TNF	30	259.97 (304.8)	18.0–1569.0
Anti-IL	1	166.0 (0)	166.0–166.0
Anti-Integrin	8	321.88 (212.34)	80.0–762.0
JAK Inhibitor	0
BIOLOGICAL STATUS
CRP Level at Baseline	65	2.17 (4.1)	0.02–21.79
CRP Level at Follow-up	90	0.7 (1.4)	0.006–7.67
Fecal Calprotectin at Baseline	51	700.4 (1106.44)	5.0–5583.0
Fecal Calprotectin at Follow-up	87	422.02 (776.86)	5.0–6000.0
Hemoglobin at Baseline	84	13.62 (2.12)	7.6–18.1
Hemoglobin at Follow-up	96	14.24 (1.43)	10.1–17.0
Platelet Count at Baseline	79	307.57 (96.89)	150.0–651.0
Platelet Count at Follow-up	94	267.6 (71.5)	141.0–490.0
Serum Creatinine at Baseline	46	1.0 (0.43)	0.2–2.58
Serum Creatinine at Follow-up	69	0.98 (0.25)	0.54–2.38
Ferritin Level at Baseline	43	111.19 (166.65)	2.61–1027.0
Ferritin Level at Follow-up	77	97.88 (80.0)	4.1–430.0
Aspartate Aminotransferase (AST) Baseline	31	22.84 (9.22)	12.0–44.0
Aspartate Aminotransferase (AST) Follow-up	37	24.41 (13.43)	11.0–76.0
Alanine Transaminase (ALT) Baseline	32	33.91 (19.87)	12.0–92.0
Alanine Transaminase (ALT) Follow-up	42	31.1 (19.81)	6.0–110.0
Gamma-Glutamyl Transferase (GGT) Baseline	29	43.81 (48.47)	10.0–208.0
Gamma-Glutamyl Transferase (GGT) Follow-up	45	31.04 (32.5)	6.0–183.0
Alkaline Phosphatase (ALP) Baseline	25	88.94 (33.64)	52.0–183.0
Alkaline Phosphatase (ALP) Follow-up	38	85.6 (29.63)	38.0–209.0
Albumin Level at Baseline	12	23.05 (18.27)	0.599–44.5
Albumin Level at Follow-up	5	3.58 (1.73)	0.509–4.56
Vitamin B12 Level at Baseline	24	455.24 (165.51)	168.0–771.0
Vitamin B12 Level at Follow-up	48	492.88 (128.48)	113.0–694.0
Vitamin D Level at Baseline	17	30.54 (16.72)	13.0–65.0
Vitamin D Level at Follow-up	53	33.95 (17.99)	8.8–65.9

and

Table 7. Summary of biological and treatment information for Crohn’s disease (CD) patients, including baseline and follow-up measurements of various biomarkers and treatment types.

Category	Patients	Mean (Std)	Range
TREATMENT INFO
Aminosalicylates	51	2073.61 (2597.28)	4.0–8716.0
Antibiotics	6	35.17 (28.96)	6.0–89.0
Steroids	64	526.77 (1611.94)	1.0–8491.0
Immunomodulator	27	2558.7 (7858.55)	29.0–41,086.0
Anti-TNF	42	1024.57 (955.05)	79.0–3956.0
Anti-IL	11	323.91 (230.16)	45.0–936.0
Anti-Integrin	5	768.0 (1085.09)	54.0–2588.0
JAK Inhibitor	1	220.0	220.0–220.0
BIOLOGICAL STATUS
CRP Level at Baseline	76	2.35 (3.81)	0.0–19.7
CRP Level at Follow-up	103	1.43 (3.11)	0.0–21.4
Fecal Calprotectin at Baseline	67	494.33 (902.57)	12.5–5893.0
Fecal Calprotectin at Follow-up	89	198.73 (343.96)	1.9–2000.0
Hemoglobin at Baseline	81	15.22 (14.69)	10.2–145.2
Hemoglobin at Follow-up	100	13.99 (1.38)	8.9–16.7
Platelet Count at Baseline	75	2176.65 (16,246.77)	188.0–141,000.0
Platelet Count at Follow-up	98	292.56 (292.16)	135.0–3058.0
Serum Creatinine at Baseline	46	0.89 (0.22)	0.063–1.54
Serum Creatinine at Follow-up	72	0.96 (0.14)	0.68–1.31
Ferritin Level at Baseline	49	84.57 (81.69)	7.0–414.0
Ferritin Level at Follow-up	68	90.22 (72.95)	9.0–424.0
Aspartate Aminotransferase (AST) Baseline	20	22.1 (8.97)	11.0–52.0
Aspartate Aminotransferase (AST) Follow-up	41	25.45 (19.75)	14.0–135.0
Alanine Transaminase (ALT) Baseline	23	29.3 (13.77)	12.0–65.0
Alanine Transaminase (ALT) Follow-up	44	34.26 (21.04)	13.0–105.0
Gamma-Glutamyl Transferase (GGT) Baseline	30	34.52 (46.73)	9.0–247.0
Gamma-Glutamyl Transferase (GGT) Follow-up	45	37.94 (77.59)	12.0–531.0
Alkaline Phosphatase (ALP) Baseline	18	85.06 (29.35)	41.0–184.0
Alkaline Phosphatase (ALP) Follow-up	40	77.92 (10.02)	54.0–108.0
Albumin Level at Baseline	8	26.14 (23.25)	0.584–58.0
Albumin Level at Follow-up	6	81.91 (153.03)	4.0–354.0
Vitamin B12 Level at Baseline	24	381.25 (190.03)	125.0–781.0
Vitamin B12 Level at Follow-up	55	410.27 (167.8)	86.0–550.0
Vitamin D Level at Baseline	19	35.11 (20.66)	8.0–65.0
Vitamin D Level at Follow-up	56	30.92 (18.3)	8.0–65.0

provide insights into biological and treatment characteristics of the UC and CD cohort at baseline and follow-up.

To assess potential redundancy among the clinical and laboratory variables, we performed a correlation analysis across key numeric variables at baseline and follow-up. Pearson correlation was used for continuous variables and Cramer’s V for categorical variables. Preliminary results show that most biomarkers are not strongly collinear, suggesting that the dataset contains a sufficient range of statistically independent variables. Full correlation matrices are provided in Supplementary Materials.

4. Discussion

The collection of comprehensive clinical data for both ulcerative colitis and Crohn’s disease, including histopathology images, may provide valuable insights into disease progression and treatment outcomes. However, the process of collecting such data poses several technological and infrastructural challenges. This study highlights key obstacles faced during data collection and offers practical insights that could guide future research endeavors and help healthcare institutions optimize their data collection strategies.

4.1. Contribution

IBD research has significantly benefited from datasets focusing on various modalities such as clinical records, genomics, and imaging. For instance, the TriNetX IBD dataset [9] provides longitudinal EHR data filtered for IBD-relevant covariates, while the Gut Reaction project [10] integrates large-scale genomic and demographic data. Image-centric datasets like KVASIR [6] focus on gastrointestinal endoscopy, and the CHOC dataset offers histopathology slides for pediatric IBD [11]. However, these datasets are modality-specific, population-limited, or lack longitudinal biomarker tracking, restricting their potential for comprehensive disease modeling. This paper introduces a novel dataset that combines multimodal data, including clinical variables, longitudinal biomarker measurements, and histopathology images from IBD patients. Unlike existing resources, this dataset bridges the gap between imaging and clinical domains, enabling a holistic approach to understanding disease progression and treatment response. Compared to TriNetX, which focuses on EHR data, and Gut Reaction, which emphasizes genomic and demographic data, the Humanendo-IBD dataset integrates clinical, imaging, laboratory, and pathology information with temporal follow-up. Unlike KVASIR, which is limited to endoscopic image data, our dataset offers a multimodal perspective that supports comprehensive modeling of disease trajectories and treatment outcomes. It is worth noticing that the limited sample size represents an important limitation of this study. This constraint arises from the fact that the cohort is highly uniform, includes a follow-up period, and specifically consists of individuals undergoing a biological treatment, which significantly reduced the initial IBD population available in our center. Furthermore, the dataset is derived from a single-center Italian cohort and may not capture the full demographic or geographic variability of the broader IBD population in Italy. Therefore, we acknowledge that some categories of patients may be underrepresented. This limits the external generalizability of findings derived from the dataset. However, the platform we have developed may help overcome this limitation in the long term by enabling continuous updates and gradual enlargement of the sample size, thereby allowing for more comprehensive and generalizable insights in the future. Indeed, the modular architecture of our platform supports the addition of new patients and data types, such as multi-omics data or new imaging modalities. Using our ETL pipelines and standardized data models, the platform is designed to scale across institutions, paving the way for multicenter data federation. Finally, although the dataset itself is not openly released, the innovation of this work lies in demonstrating the feasibility of harmonizing multimodal, longitudinal data from a real-world hospital setting. The architecture, data schema, and integration strategies shared in this manuscript serve as a template for other institutions seeking to build similar datasets, offering direct utility to the research community even in the absence of full data release. Nevertheless, data are available upon specific request.

4.2. Technological and Infrastructural Challenges

The process of our data collection presents several technological and infrastructural challenges. One of the first challenges encountered was the technological infrastructure required to manage and store vast amounts of clinical, laboratory, and imaging data. It is important to note that the final population was the result of an extensive screening process that demanded considerable efforts in resource organization and management. Hospitals and research institutions often face limitations with existing data management systems [12], which may not be equipped to handle the diverse data types involved in IBD research. In this study, data from multiple sources—including clinical records, laboratory test results, treatment regimens, and histopathological images—needed to be integrated into a single research database. The lack of seamless interoperability between different healthcare systems was a major barrier. Electronic Health Record (EHR) systems often lack the flexibility required for easy data extraction and integration with other platforms, necessitating significant manual work in harmonizing the data. Moving forward, hospitals should adopt integrated systems that support cross-platform data exchange, which would reduce the time spent on data preparation and facilitate a smoother research process [13]. To this aim, collaborative data collection tools, such as cloud-based platforms, are suggested for facilitating coordination across different institutions and research teams. These platforms offer substantial advantages in terms of data accessibility and sharing. However, they also introduce challenges related to patient privacy and security [14,15,16]. Dedicated agreements with cloud providers are required to ensure secure communication and to allow the safe use of AI-based tools integrated within these platforms. Furthermore, the effective use of these tools presents its own set of challenges. Research teams—including clinical staff, pathologists, and data scientists—must be adequately trained in how to use these platforms, which can be resource-intensive. Additionally, a standardized approach to data entry is essential for maintaining consistency across all contributors. During data collection, inconsistencies in how data were recorded, particularly with free-text fields, required multiple revision steps to standardize the dataset. For example, smoking status and patient histories were inconsistently documented across different operators, leading to missing or incomplete data. To mitigate such issues, future research should implement standardized data entry templates and ensure comprehensive training for all research staff [17,18]. Large language models could serve this purpose. Automated data validation checks should also be integrated to flag inconsistencies or errors during the data entry process, thus reducing the need for manual revisions. In retrospect, employing digital pathology tools with standardized annotation features and AI-assisted data harmonization (such as locally deployed large language models [19]) could have streamlined this process, improving data quality and reducing human error. Finally, managing high-resolution whole-slide images presented storage and processing challenges. We addressed this by keeping pathology images on our internal storage systems, linking them with their corresponding patient using structured metadata. We utilized efficient image formats and pyramidal tiling to reduce memory usage and allow for multi-resolution access. However, more advanced technical solutions could be explored, such as the integration of image compression algorithms based on deep learning to optimize storage without compromising diagnostic fidelity.

4.3. Data Governance and Quality Control

Beyond technological improvements, several insights can guide hospitals and research teams in collecting similar data in the future. Establishing a comprehensive data governance framework is essential to ensure data quality and consistency [12,20]. This framework should define protocols for data collection, standardization, and validation while assigning clear roles and responsibilities for data management. It is also crucial to establish a feedback loop between clinical and research teams to address issues that arise during data collection promptly. Hospitals should also prioritize the creation of systems to facilitate long-term data collection. The value of longitudinal data, particularly in chronic conditions like UC and CD, cannot be overstated. By implementing automated follow-up reminders and patient tracking systems, institutions can reduce the loss of follow-up and ensure consistent collection of valuable longitudinal data. Additionally, protocols should be developed to document the reasons for missing or incomplete data, thus allowing for more robust and actionable results. Finally, fostering greater collaboration and communication between researchers, clinicians, and administrative staff is essential to streamline data collection and maintain efficiency.

4.4. Informed Consent Process

A critical insight from this study is the importance of preparing hospitals for the informed consent process well in advance of data collection. Obtaining informed consent is a crucial step in any clinical research study, especially when handling sensitive health data. However, challenges were encountered during patient recruitment, as the informed consent process was occasionally delayed or inconsistent. These delays hindered the timely initiation of data collection and added unnecessary complexity to the workflow. To avoid such issues in future studies, hospitals should ensure that the informed consent process is streamlined and organized before data collection begins. A clear, standardized consent form covering all aspects of the study—such as data usage, patient privacy, and the inclusion of biological samples—will help prevent delays and ensure that patients are well-informed about their participation from the outset.

4.5. Biological Sample Handling and Digital Pathology

Finally, issues related to the storage and handling of biological samples, specifically histopathology specimens, presented additional challenges. While histopathology images were not analyzed in this study, difficulties arose in the storage process of biological samples. Ensuring the integrity and reliability of histopathology data depends on proper preservation of samples. In this case, inconsistencies in sample storage—such as temperature control and processing delays—led to complications in ensuring that all samples were suitable for future analysis. To address this issue, hospitals should digitalize histopathology images within the clinical workflow. Implementing digital pathology systems that capture high-resolution images at the point of care will not only streamline the data collection process but also reduce the risk of sample degradation or mishandling. This shift would help ensure that high-quality, analyzable images are consistently available for future research, reducing potential errors in sample management.

5. Conclusions

In this paper, a novel dataset for machine-learning-ready IBD research was presented. We integrated diverse and heterogeneous data sources, ranging from clinical records to imaging and laboratory results. The dataset is structured for machine-learning applications, including supervised classification of treatment response, regression models to predict biomarker levels, time-series modeling of disease evolution, and multimodal fusion approaches combining clinical, lab, and image features. Clinically, this dataset can inform the development of AI tools that predict early response to therapy, identify high-risk patients, and tailor treatment plans. It supports personalized medicine approaches by enabling patient stratification based on objective biomarkers, endoscopic features, and clinical profiles. We believe that the structure and depth of this dataset can serve as a model for similar data collection efforts in other centers, promoting multicenter collaborations and larger-scale studies. Such efforts could accelerate the validation and generalizability of predictive models, ultimately contributing to more precise and effective patient care across diverse clinical settings.