Machine Learning and Knowledge Extraction

We present a model of classical binary time series derived from a matrix product state (MPS) Ansatz widely used in one-dimensional quantum systems. We discuss how this quantum Ansatz allows us to generate classical time series in a sequential manner. Our time series are built in two steps: First, a lower-level series (the driving noise or the increments) is created directly from the MPS representation, which is then integrated to create our ultimate higher-level series. The lower- and higher-level series have clear interpretations in the quantum context, and we elaborate on this correspondence with specific examples such as the spin-1/2 Ising model in a transverse field (ITF model), where spin configurations correspond to the increments of discrete-time, discrete-level stochastic processes with finite or infinite autocorrelation lengths, Gaussian or non-Gaussian limit distributions, nontrivial Hurst exponents, multifractality, asymptotic self-similarity, etc. Our time series model is a parametric model, and we investigate how flexible the model is in some synthetic and real-life calibration problems. Full article

Deep learning models have an intrinsic privacy issue as they memorize parts of their training data, creating a privacy leakage. Membership inference attacks (MIAs) exploit this to obtain confidential information about the data used for training, aiming to steal information. They can be repurposed as a measurement of data integrity by inferring whether the data were used to train a machine learning model. While state-of-the-art attacks achieve significant privacy leakage, their requirements render them infeasible, hindering their use as practical tools to assess the magnitude of the privacy risk. Moreover, the most appropriate evaluation metric of MIA, the true positive rate at a low false positive rate, lacks interpretability. We claim that the incorporation of few-shot learning techniques into the MIA field and a suitable qualitative and quantitative privacy evaluation measure should resolve these issues. In this context, our proposal is twofold. We propose a few-shot learning-based MIA, termed the FeS-MIA model, which eases the evaluation of the privacy breach of a deep learning model by significantly reducing the number of resources required for this purpose. Furthermore, we propose an interpretable quantitative and qualitative measure of privacy, referred to as the Log-MIA measure. Jointly, these proposals provide new tools to assess privacy leakages and to ease the evaluation of the training data integrity of deep learning models, i.e., to analyze the privacy breach of a deep learning model. Experiments carried out with MIA over image classification and language modeling tasks, and a comparison to the state of the art, show that our proposals excel in identifying privacy leakages in a deep learning model with little extra information. Full article

With the evolution of data collection technologies, sensor-generated data have become the norm. However, decades of manually recorded maintenance data still hold untapped value. Natural Language Processing (NLP) offers new ways to extract insights from these historical records, especially from short, unstructured maintenance texts often accompanying structured database fields. While NLP has shown promise in this area, technical texts pose unique challenges, particularly in preprocessing and manual annotation. This study proposes a novel methodology combining Failure Mode and Effect Analysis (FMEA), a reliability engineering tool, into the NLP pipeline to enhance Named Entity Recognition (NER) in maintenance records. By leveraging the structured and domain-specific knowledge encapsulated in FMEAs, the annotation process becomes more systematic, reducing the need for exhaustive manual effort. A case study using real-world data from a major electrical utility demonstrates the effectiveness of this approach. The custom NER model, trained using FMEA-informed annotations, achieves high precision, recall, and F1 scores, successfully identifying key reliability elements in maintenance text. The integration of FMEA not only improves data quality but also supports more informed asset management decisions. This research introduces a novel cross-disciplinary framework combining reliability engineering and NLP. It highlights how domain expertise can be used to streamline annotation, improve model accuracy, and unlock actionable insights from legacy maintenance data. Full article

As programming education becomes increasingly complex, grading student code has become a challenging task. Traditional methods, such as dynamic and static analysis, offer foundational approaches but often fail to provide granular insights, leading to inconsistencies in grading and feedback. This study addresses the limitations of these methods by integrating contrastive learning with explainable AI techniques to assess SQL code submissions. We employed contrastive learning to differentiate between student and correct SQL solutions, projecting them into a high-dimensional latent space, and used the Frobenius norm to measure the distance between these representations. This distance was used to predict the percentage of points deducted from each student’s solution. To enhance interpretability, we implemented feature ablation and integrated gradients, which provide insights into the specific tokens in student code that impact the grading outcomes. Our findings indicate that this approach improves the accuracy, consistency, and transparency of automated grading, aligning more closely with human grading standards. The results suggest that this framework could be a valuable tool for automated programming assessment systems, offering clear, actionable feedback and making machine learning models in educational contexts more interpretable and effective. Full article

Knee osteoarthritis (KOA) is a highly prevalent muscoloskeletal joint disorder affecting a significant portion of the population worldwide. Accurate predictions of KOA progression can assist clinicians in drawing preventive strategies for patients. In this paper, we present an integrated approach based on hypergraph convolutional networks (HGCNs) for longitudinal predictions of KOA grades and progressions from MRI images. We propose two novel models, namely, the C_Shape.Net and the predictor network. The C_Shape.Net operates on a hypergraph of volumetric nodes, especially designed to represent the surface and volumetric features of the cartilage. It encompasses deep HGCN convolutions, graph pooling, and readout operations in a hierarchy of layers, providing, at the output, expressive 3D shape descriptors of the cartilage volume. The predictor is a spatio-temporal HGCN network (ST_HGCN), following the sequence-to-sequence learning scheme. Concretely, it transforms sequences of knee representations at the historical stage into sequences of KOA predictions at the prediction stage. The predictor includes spatial HGCN convolutions, attention-based temporal fusion of feature embeddings at multiple layers, and a transformer module that generates longitudinal predictions at follow-up times. We present comprehensive experiments on the Osteoarthritis Initiative (OAI) cohort to evaluate the performance of our methodology for various tasks, including node classification, longitudinal KL grading, and progression. The basic finding of the experiments is that the larger the depth of the historical stage, the higher the accuracy of the obtained predictions in all tasks. For the maximum historic depth of four years, our method yielded an average balanced accuracy (BA) of 85.94% in KOA grading, and accuracies of 91.89% (+1), 88.11% (+2), 84.35% (+3), and 79.41% (+4) for the four consecutive follow-up visits. Under the same setting, we also achieved an average value of Area Under Curve (AUC) of 0.94 for the prediction of progression incidence, and follow-up AUC values of 0.81 (+1), 0.77 (+2), 0.73 (+3), and 0.68 (+4), respectively. Full article

This paper introduces a novel framework enabling natural language question answering on Piping and Instrumentation Diagrams (P&IDs), addressing a critical gap between engineering design documentation and intuitive information retrieval. Our approach transforms static P&IDs into queryable knowledge bases through a three-stage pipeline. First, we recognize entities in a P&ID image and organize their relationships to form a base entity graph. Second, this entity graph is converted into a Labeled Property Graph (LPG), enriched with semantic attributes for nodes and edges. Third, a Large Language Model (LLM)-based information retrieval system translates a user query into a graph query language (Cypher) and retrieves the answer by executing it on LPG. For our experiments, we augmented a publicly available P&ID image dataset with our novel PIDQA dataset, which comprises 64,000 question–answer pairs spanning four categories: (I) simple counting, (II) spatial counting, (III) spatial connections, and (IV) value-based questions. Our experiments (using gpt-3.5-turbo) demonstrate that grounding the LLM with dynamic few-shot sampling robustly elevates accuracy by 10.6–43.5% over schema contextualization alone, even under high lexical diversity conditions (e.g., paraphrasing, ambiguity). By reducing barriers in retrieving P&ID data, this work advances human–AI collaboration for industrial workflows in design validation and safety audits. Full article

The reciprocal relationship between Large Language Models (LLMs) and Knowledge Graphs (KGs) highlights their synergistic potential in enhancing artificial intelligence (AI) applications. LLMs, with their natural language understanding and generative capabilities, support the automation of KG construction through entity recognition, relation extraction, and schema generation. Conversely, KGs serve as structured and interpretable data sources that improve the transparency, factual consistency and reliability of LLM-based applications, mitigating challenges such as hallucinations and lack of explainability. This study conducts a systematic literature review of 77 studies to examine AI methodologies supporting LLM–KG integration, including symbolic AI, machine learning, and hybrid approaches. The research explores diverse applications spanning healthcare, finance, justice, and industrial automation, revealing the transformative potential of this synergy. Through in-depth analysis, this study identifies key limitations in current approaches, including challenges in scalability with maintaining dynamic and real-time Knowledge Graphs, difficulty in adapting general-purpose LLMs to specialized domains, limited explainability in tracing model outputs to interpretable reasoning, and ethical concerns surrounding bias, fairness, and transparency. In response, the study highlights potential strategies to optimize LLM–KG synergy. The findings from this study provide actionable insights for researchers and practitioners aiming for robust, transparent, and adaptive AI systems to enhance knowledge-driven AI applications through LLM–KG integration, further advancing generative AI and explainable AI (XAI) applications. Full article

Particle tracking velocimetry (PTV) forms the basis for many fluid dynamic experiments, in which individual particles are tracked across multiple successive images. However, when the experimental setup involves high-speed, high-density particles that are indistinguishable and follow complex or unknown flow fields, matching particles between images becomes significantly more challenging. Reliable PTV algorithms are crucial in such scenarios. Previous work has demonstrated that the Self-Organizing Map (SOM) machine learning approach offers superior outcomes on complex-plasma data compared with traditional methods, though its performance is sensitive to hyperparameter calibration, which requires optimization for specific flow scenarios. In this article, we describe how the dependence of the various hyperparameters on different flow scenarios was studied and the optimal settings for diverse flow conditions were identified. Based on these results, automatic hyperparameter calibration was implemented in the PTV framework. Furthermore, the SOM’s performance was directly compared with that of the preceding conventional PTV method, Trackpy, for complex plasmas using synthetic data. Finally, as a new approach to identifying incorrectly matched particle traces, a Long Short-Term Memory (LSTM) neural network was developed to sort out all inaccuracies to further improve the outcome. Combined with automatic hyperparameter calibration, outlier detection and additional computational speed optimization, this work delivers a robust, versatile and efficient framework for PTV analysis. Full article

We used machine learning (ML) to characterize 894,154 medical records of outpatient visits from the Veterans Administration Central Data Warehouse (VA CDW) by the likelihood of assignment of 200 International Classification of Diseases (ICD) code blocks. Using four different predictive models, we found the ML-derived predictions for the code blocks were consistently more effective in predicting death or 90-day rehospitalization than the assigned code block in the record. We reviewed records of ICD chapter assignments. The review revealed that the ML-predicted chapter assignments were consistently better than those humanly assigned. Impact factor analysis, a method of explanation of AI findings that was developed in our group, demonstrated little effect on any one assigned ICD code block but a marked impact on the ML-derived code blocks of kidney disease as well as several other morbidities. In this study, machine learning was much better than human code assignment at predicting the relatively rare outcomes of death or rehospitalization. Future work will address generalizability using other datasets, as well as addressing coding that is more nuanced than that of the categorization provided by code blocks. Full article

Large language models (LLMs) have recently demonstrated notable performance, particularly in addressing the challenge of extensive data requirements when training traditional forecasting models. However, these methods encounter significant challenges when applied to high-dimensional and domain-specific datasets. These challenges primarily arise from inability to effectively model inter-variable dependencies and capture variable-specific characteristics, leading to suboptimal performance in complex forecasting scenarios. To address these limitations, we propose ADTime, an adaptive LLM-based approach for multivariate time series forecasting. ADTime employs advanced preprocessing techniques to identify latent relationships among key variables and temporal features. Additionally, it integrates temporal alignment mechanisms and prompt-based strategies to enhance the semantic understanding of forecasting tasks by LLMs. Experimental results show that ADTime outperforms state-of-the-art methods, reducing MSE by 9.5% and MAE by 6.1% on public datasets, and by 17.1% and 13.5% on domain-specific datasets. Furthermore, zero-shot experiments on real-world refinery datasets demonstrate that ADTime exhibits stronger generalization capabilities across various transfer scenarios. These findings highlight the potential of ADTime in advancing complex, domain-specific time series forecasting tasks. Full article

In electron microscopy, obtaining low-noise images is often difficult, especially when examining biological samples or delicate materials. Therefore, the suppression of noise is essential for the analysis of such noisy images. State-of-the-art image denoising methods are dominated by supervised Convolution neural network (CNN)-based methods. However, supervised CNNs cannot be used if a noise-free ground truth is unavailable. To address this problem, we propose a method that uses re-occurring patterns in images. Our proposed method does not require noise-free images for the denoising task. Instead, it is based on the idea that averaging images with the same signal having independent noise suppresses the overall noise. In order to evaluate the performance of our method, we compare our results with other state-of-the-art denoising methods that do not require a noise-free image. We show that our method is the best for retaining fine image structures. Additionally, we develop a confidence map for evaluating the denoising quality of the proposed method. Furthermore, we analyze the time complexity of the algorithm to ensure scalability and optimize the algorithm to improve the runtime efficiency. Full article

Regression is a fundamental task in machine learning, and neural networks have been successfully employed in many applications to identify underlying regression patterns. However, they are often criticised for their lack of interpretability and commonly referred to as black-box models. Feature selection approaches address this challenge by simplifying datasets through the removal of unimportant features, while improving explainability by revealing feature importance. In this work, we leverage mathematical programming to identify the most important features in a trained deep neural network with a ReLU activation function, providing greater insight into its decision-making process. Unlike traditional feature selection methods, our approach adjusts the weights and biases of the trained neural network via a Mixed-Integer Linear Programming (MILP) model to identify the most important features and thereby uncover underlying relationships. The mathematical formulation is reported, which determines the subset of selected features, and clustering is applied to reduce the complexity of the model. Our results illustrate improved performance in the neural network when feature selection is implemented by the proposed approach, as compared to other feature selection approaches. Finally, analysis of feature selection frequency across each dataset reveals feature contribution in model predictions, thereby addressing the black-box nature of the neural network. Full article

Due to their considerable costs, deep neural networks (DNNs) are valuable assets that need to be protected in terms of intellectual property (IP). From this statement, DNN watermarking gains significant interest since it allows DNN owners to prove their ownership. Various methods that embed ownership information in the model behavior have been proposed. They need to fill several requirements, among them the security, which represents an attacker’s difficulty in breaking the watermarking scheme. There is also the robustness requirement, which quantifies the resistance against watermark removal techniques. The problem is that the proposed methods generally fail to meet these necessary standards. This paper presents RoSe-Mix, a robust and secure deep neural network watermarking technique designed for black-box settings. It addresses limitations in existing DNN watermarking approaches by integrating key features from two established methods: RoSe, which uses cryptographic hashing to ensure security, and Mixer, which employs image Mixup to enhance robustness. Experimental results demonstrate that RoSe-Mix achieves security across various architectures and datasets with a robustness to removal attacks exceeding 99%. Full article

This paper explores the potential use of Large Language Models (LLMs), such as ChatGPT, Google Gemini, and Microsoft Copilot, in threat hunting, specifically focusing on Living off the Land (LotL) techniques. LotL methods allow threat actors to blend into regular network activity, which makes detection by automated security systems challenging. The study seeks to determine whether LLMs can reliably generate effective queries for security tools, enabling organisations with limited budgets and expertise to conduct threat hunting. A testing environment was created to simulate LotL techniques, and LLM-generated queries were used to identify malicious activity. The results demonstrate that LLMs do not consistently produce accurate or reliable queries for detecting these techniques, particularly for users with varying skill levels. However, while LLMs may not be suitable as standalone tools for threat hunting, they can still serve as supportive resources within a broader security strategy. These findings suggest that, although LLMs offer potential, they should not be relied upon for accurate results in threat detection and require further refinement to be effectively integrated into cybersecurity workflows. Full article

In today’s rapidly evolving transportation infrastructure, developing long-lasting, high-performance pavement materials remains a significant priority. Integrating machine learning (ML) techniques provides a transformative approach to optimizing asphalt mix design and performance prediction. This study investigates the use of waste plastics, including Polyethylene Terephthalate (PET), High-Density Polyethylene (HDPE), and Polyvinyl Chloride (PVC), as modifiers in asphalt concrete to enhance durability and mechanical performance. A predictive modeling approach was employed to estimate the bulk-specific gravity (G_mb) of asphalt concrete using various ML techniques, including Artificial Neural Networks (ANNs), Support Vector Machines (SVMs), Gaussian Processes (GPs), and Reduced Error Pruning (REP) Tree. The accuracy of each model was evaluated using statistical performance metrics, including the correlation coefficient (CC), scatter index (SI), mean absolute error (MAE), and root mean square error (RMSE). The results demonstrate that the ANN model outperformed all other ML techniques, achieving the highest correlation (CC = 0.9996 for training, 0.9999 for testing) and the lowest error values (MAE = 0.0004, RMSE = 0.0006, SI = 0.00026). A comparative analysis between actual and predicted G_mb values confirmed the reliability of the proposed ANN model, with minimal error margins and superior accuracy. Additionally, sensitivity analysis identified bitumen content (BC) and volume of bitumen (V_b) as the most influential parameters affecting G_mb, emphasizing the need for precise parameter optimization in asphalt mix design. This study demonstrates the effectiveness of machine learning-driven predictive modeling in optimizing sustainable asphalt mix design, offering a cost-effective, time-efficient, and highly accurate alternative to traditional experimental methods. Full article

Waste heat recovery is a critical strategy for optimizing energy consumption and reducing greenhouse gas emissions. In this context, the circular economy highlights the importance of this practice as a key tool to enhance energy efficiency, minimize waste, and decrease environmental impact. Artificial neural networks are particularly well-suited for managing nonlinearities and complex interactions among multiple variables, making them ideal for controlling a double-stage absorption heat transformer. This study aims to simultaneously optimize both user-defined parameters. Levenberg–Marquardt and scaled conjugated gradient algorithms were compared from five to twenty-five neurons to determine the optimal operating conditions while the coefficient of performance and the gross temperature lift were simultaneously maximized. The methodology includes R2024a MATLAB© programming, real-time data acquisition, visual engineering environment software, and flow control hardware. The results show that applying the Levenberg–Marquardt algorithm resulted in an increase in the correlation coefficient (R) at 20 neurons, improving the thermodynamic performance and enabling greater energy recovery from waste heat. Full article

As climate change transforms our environment and human intrusion into natural ecosystems escalates, there is a growing demand for disease spread models to forecast and plan for the next zoonotic disease outbreak. Accurate parametrization of these models requires data from diverse sources, including the scientific literature. Despite the abundance of scientific publications, the manual extraction of these data via systematic literature reviews remains a significant bottleneck, requiring extensive time and resources, and is susceptible to human error. This study examines the application of a large language model (LLM) as an assessor for screening prioritisation in climate-sensitive zoonotic disease research. By framing the selection criteria of articles as a question–answer task and utilising zero-shot chain-of-thought prompting, the proposed method achieves a saving of at least 70% work effort compared to manual screening at a recall level of 95% (NWSS@95%). This was validated across four datasets containing four distinct zoonotic diseases and a critical climate variable (rainfall). The approach additionally produces explainable AI rationales for each ranked article. The effectiveness of the approach across multiple diseases demonstrates the potential for broad application in systematic literature reviews. The substantial reduction in screening effort, along with the provision of explainable AI rationales, marks an important step toward automated parameter extraction from the scientific literature. Full article

(1) Background: This paper intends to accomplish a comparative study and analysis regarding the multiclass classification of facial thermal images, i.e., in three classes corresponding to predefined emotional states (neutral, happy and sad). By carrying out a comparative analysis, the main goal of the paper consists in identifying a suitable algorithm from machine learning field, which has the highest accuracy (ACC). Two categories of images were used in the process, i.e., images with Gaussian noise and images with “salt and pepper” type noise that come from two built-in special databases. An augmentation process was applied to the initial raw images that led to the development of the two databases with added noise, as well as the subsequent augmentation of all images, i.e., rotation, reflection, translation and scaling. (2) Methods: The multiclass classification process was implemented through two subsets of methods, i.e., machine learning with random forest (RF), support vector machines (SVM) and k-nearest neighbor (KNN) algorithms and deep learning with the convolutional neural network (CNN) algorithm. (3) Results: The results obtained in this paper with the two subsets of methods belonging to the field of artificial intelligence (AI), together with the two categories of facial thermal images with added noise used as input, were very good, showing a classification accuracy of over 99% for the two categories of images, and the three corresponding classes for each. (4) Discussion: The augmented databases and the additional configurations of the implemented algorithms seems to have had a positive effect on the final classification results. Full article

Equatorial plasma bubbles (EPBs) are regions of depleted electron density that form in the Earth’s ionosphere due to Rayleigh–Taylor instability. These bubbles can cause signal scintillation, leading to signal loss and errors in position calculations. EPBs can be detected in images captured by All-Sky Imager (ASI) systems. This study proposes a low-cost automatic detection method for EPBs in ASI data that can be used for both real-time detection and classification purposes. This method utilizes Two-Dimensional Principal Component Analysis (2DPCA) with Recursive Feature Elimination (RFE), in conjunction with a Random Forest machine learning model, to create an Explainable Artificial Intelligence (XAI) model capable of extracting image features to automatically detect EPBs with the lowest possible dimensionality. This led to having a small-sized and extremely fast-trained model that could be used to identify EPBs within the captured ASI images. A set of 2458 images, classified into two categories—Event and Empty—were used to build the database. This database was randomly split into two subsets: a training dataset (80%) and a testing dataset (20%). The produced XAI model demonstrated slightly higher detection accuracy compared to the standard 2DPCA model while being significantly smaller in size. Furthermore, the proposed model’s performance has been evaluated and compared with other deep learning baseline models (ResNet18, Inception-V3, VGG16, and VGG19) in the same environment. Full article