Search Results (97)

Neural Architecture search (NAS) aims to automate the design process of Deep Neural Networks, reducing the Deep Learning (DL) expertise required and avoiding a trial-and-error process. Nonetheless, one of the main drawbacks of NAS is the high consumption of computational resources. Consequently, efficient evaluation methods (EEMs) to assess the quality of candidate architectures are an open research problem. This work tests various EEMs in the Deep Genetic Algorithm (DeepGA), including early stopping, population memory, and training-free proxies. The Fashion MNIST, CIFAR-10, and CIFAR-100 datasets were used for experimentation. The results show that population memory has a valuable impact on avoiding repeated evaluations. Additionally, early stopping achieved competitive performance while significantly reducing the computational cost of the search process. The training-free configurations using the Logsynflow and Linear Regions proxies, as well as a combination of both, were only partially competitive but dramatically reduced the search time. Finally, a comparison of the architectures and hyperparameters obtained with the different algorithm configurations is presented. The training-free search processes resulted in deeper architectures with more fully connected layers and skip connections than the ones obtained with accuracy-guided search configurations. Full article

In the previous article, we introduced a neural network framework based on symmetric differential equations. This novel framework exhibits complete symmetry, endowing it with perfect mathematical properties. While we have examined some of the system’s mathematical characteristics, a detailed discussion of the network training methodology has not yet been presented. Drawing on the principles of the traditional backpropagation algorithm, this study proposes an alternative training approach that utilizes differential equation signal propagation instead of chain rule derivation. This approach not only preserves the effectiveness of training but also offers enhanced biological interpretability. The foundation of this methodology lies in the system’s reversibility, which stems from its inherent symmetry—a key aspect of our research. However, this method alone is insufficient for effective neural network training. To address this, we further introduce a distributed Proportional–Integral–Derivative (PID) control approach, emphasizing its implementation within a closed system. By incorporating this method, we achieved both faster training speeds and improved accuracy. This approach not only offers novel insights into neural network training but also extends the scope of research into control methodologies. To validate its effectiveness, we apply this method to the MNIST (Modified National Institute of Standards and Technology database) and Fashion-MNIST, demonstrating its practical utility. Full article

In this study, we present a comparative analysis of gradient descent-based optimizers frequently used in Convolutional Neural Networks (CNNs), including SGD, mSGD, RMSprop, Adadelta, Nadam, Adamax, Adam, and the recent EVE optimizer. To explore the interaction between optimization strategies and activation functions, we systematically evaluate all combinations of these optimizers with four activation functions—ReLU, LeakyReLU, Tanh, and GELU—across three benchmark image classification datasets: CIFAR-10, Fashion-MNIST (F-MNIST), and Labeled Faces in the Wild (LFW). Each configuration was assessed using multiple evaluation metrics, including accuracy, precision, recall, F1-score, mean absolute error (MAE), and mean squared error (MSE). All experiments were performed using k-fold cross-validation to ensure statistical robustness. Additionally, two-way ANOVA was employed to validate the significance of differences across optimizer–activation combinations. This study aims to highlight the importance of jointly selecting optimizers and activation functions to enhance training dynamics and generalization in CNNs. We also consider the role of critical hyperparameters, such as learning rate and regularization methods, in influencing optimization stability. This work provides valuable insights into the optimizer–activation interplay and offers practical guidance for improving architectural and hyperparameter configurations in CNN-based deep learning models. Full article

Background/Objectives: Convolutional Neural Networks (CNNs), while effective in tasks such as image classification and language processing, often experience overfitting and inefficient training due to static, structure-agnostic regularization techniques like traditional dropout. This study aims to address these limitations by proposing a more dynamic and context-sensitive dropout strategy. Methods: We introduce Probabilistic Feature Importance Dropout (PFID), a novel regularization method that assigns dropout rates based on the probabilistic significance of individual features. PFID is integrated with adaptive, structured, and contextual dropout strategies, forming a unified framework for intelligent regularization. Results: Experimental evaluation on standard benchmark datasets including CIFAR-10, MNIST, and Fashion MNIST demonstrated that PFID significantly improves performance metrics such as classification accuracy, training loss, and computational efficiency compared to conventional dropout methods. Conclusions: PFID offers a practical and scalable solution for enhancing CNN generalization and training efficiency. Its dynamic nature and feature-aware design provide a strong foundation for future advancements in adaptive regularization for deep learning models. Full article

Inkjet-printed circuits on flexible substrates are rapidly emerging as a key technology in flexible electronics, driven by their minimal fabrication process, cost-effectiveness, and environmental sustainability. Recent advancements in inkjet-printed devices and circuits have broadened their applications in both sensing and computing. Building on this progress, this work has developed a nonlinear computational element coined as mTanh to serve as an activation function in neural networks. Activation functions are essential in neural networks as they introduce nonlinearity, enabling machine learning models to capture complex patterns. However, widely used functions such as Tanh and sigmoid often suffer from the vanishing gradient problem, limiting the depth of neural networks. To address this, alternative functions like ReLU and Leaky ReLU have been explored, yet these also introduce challenges such as the dying ReLU issue, bias shifting, and noise sensitivity. The proposed mTanh activation function effectively mitigates the vanishing gradient problem, allowing for the development of deeper neural network architectures without compromising training efficiency. This study demonstrates the feasibility of mTanh as an activation function by integrating it into an Echo State Network to predict the Mackey–Glass time series signal. The results show that mTanh performs comparably to Tanh, ReLU, and Leaky ReLU in this task. Additionally, the vanishing gradient resistance of the mTanh function was evaluated by implementing it in a deep multi-layer perceptron model for Fashion MNIST image classification. The study indicates that mTanh enables the addition of 3–5 extra layers compared to Tanh and sigmoid, while exhibiting vanishing gradient resistance similar to ReLU. These results highlight the potential of mTanh as a promising activation function for deep learning models, particularly in flexible electronics applications. Full article

Federated learning represents a newly emerging methodology in the field of machine learning that enables distributed agents to collaboratively learn a centralized model without sharing their raw data. Some scholars have already proposed many first-order algorithms and second-order algorithms for federated learning to reduce communication costs and speed up convergence. However, these algorithms generally rely on gradient or Hessian information, and we find it difficult to solve such federated optimization problems when the analytical expression of the loss function is not available, that is, when gradient information is not available. Therefore, we employed derivative-free federated zero-order optimization in this paper, which does not rely on specific gradient information, but instead utilizes the changes in function values or model outputs to estimate the optimization direction. Furthermore, to enhance the performance of derivative-free zero-order optimization, we propose an effective adaptive algorithm that can dynamically adjust the learning rate and other hyperparameters based on the performance during the optimization process, aiming to accelerate convergence. We rigorously analyze the convergence of our approach, and the experimental findings demonstrate our method can indeed achieve faster convergence speed on the MNIST, CIFAR-10 and Fashion-MNIST datasets in cases where gradient information is not available. Full article

Spike neural networks (SNNs) perform excellently in various domains. However, SNNs based on differential privacy (DP) protocols introduce uniform noise to the gradient parameters, which may affect the trade-off between model efficiency and personal privacy. Therefore, the adaptive differential private SNN (ADPSNN) is proposed in this work. It dynamically adjusts the privacy budget based on the correlations between the output spikes and labels. In addition, the noise is added to the gradient parameters according to the privacy budget. The ADPSNN is tested on four datasets with different spiking neurons including leaky integrated-and-firing (LIF) and integrate-and-fire (IF) models. Experimental results show that the LIF neuron model provides superior utility on the MNIST (accuracy 99.56%) and Fashion-MNIST (accuracy 92.26%) datasets, while the IF neuron model performs well on the CIFAR10 (accuracy 90.67%) and CIFAR100 (accuracy 66.10%) datasets. Compared to existing methods, the accuracy of ADPSNN is improved by 0.09% to 3.1%. The ADPSNN has many potential applications, such as image classification, health care, and intelligent driving. Full article

Optical neural networks are hardware neural networks implemented based on physical optics, and they have demonstrated advantages of high speed, low energy consumption, and resistance to electromagnetic interference in the field of image processing. However, most previous optical neural networks were designed for coherent light inputs, which required the introduction of an electro-optical conversion module before the optical computing device. This significantly hindered the inherent speed and energy efficiency advantages of optical computing. In this paper, we propose a diffraction algorithm for incoherent light based on mutual intensity propagation, and on this basis, we established a model of an incoherent optical neural network. This model is completely passive and directly performs inference calculations on natural light, with the detector directly outputting the results, achieving target classification in an all-optical environment. The proposed model was tested on the MNIST, Fashion-MNIST, and ISDD datasets, achieving classification accuracies of 82.32%, 72.48%, and 93.05%, respectively, with experimental verification showing an accuracy error of less than 5%. This neural network can achieve passive and delay-free inference in a natural light environment, completing target classification and showing good application prospects in the field of remote sensing. Full article

Federated learning (FL) is a distributed learning technique that ensures data privacy and has shown significant potential in cross-institutional image analysis. However, existing methods struggle with the inherent dynamic heterogeneity of real-world data, such as changes in cellular differentiation during disease progression or feature distribution shifts due to different imaging devices. This dynamic heterogeneity can cause catastrophic forgetting, leading to reduced performance in medical predictions across stages. Unlike previous federated learning studies that paid insufficient attention to dynamic heterogeneity, this paper proposes the FedDyH framework to address this challenge. Inspired by the adaptive regulation mechanisms of biological systems, this framework incorporates several core modules to tackle the issues arising from dynamic heterogeneity. First, the framework simulates intercellular information transfer through cross-client knowledge distillation, preserving local features while mitigating knowledge forgetting. Additionally, a dynamic regularization term is designed in which the strength can be adaptively adjusted based on real-world conditions. This mechanism resembles the role of regulatory T cells in the immune system, balancing global model convergence with local specificity adjustments to enhance the robustness of the global model while preventing interference from diverse client features. Finally, the framework introduces a genetic algorithm (GA) to simulate biological evolution, leveraging mechanisms such as gene selection, crossover, and mutation to optimize hyperparameter configurations. This enables the model to adaptively find the optimal hyperparameters in an ever-changing environment, thereby improving both adaptability and performance. Prior to this work, few studies have explored the use of optimization algorithms for hyperparameter tuning in federated learning. Experimental results demonstrate that the FedDyH framework improves accuracy compared to the SOTA baseline FedDecorr by 2.59%, 0.55%, and 5.79% on the MNIST, Fashion-MNIST, and CIFAR-10 benchmark datasets, respectively. This framework effectively addresses data heterogeneity issues in dynamic heterogeneous environments, providing an innovative solution for achieving more stable and accurate distributed federated learning. Full article

This work presents a mathematical solution to data privacy and integrity issues in Split Learning which uses Homomorphic Encryption (HE) and Zero-Knowledge Proofs (ZKP). It allows calculations to be conducted on encrypted data, keeping the data private, while ZKP ensures the correctness of these calculations without revealing the underlying data. Our proposed system, HavenSL, combines HE and ZKP to provide strong protection against attacks. It uses Discrete Cosine Transform (DCT) to analyze model updates in the frequency domain to detect unusual changes in parameters. HavenSL also has a rollback feature that brings the system back to a verified state if harmful changes are detected. Experiments on CIFAR-10, MNIST, and Fashion-MNIST datasets show that using Homomorphic Encryption and Zero-Knowledge Proofs during training is feasible and accuracy is maintained. This mathematical-based approach shows how crypto-graphic can protect decentralized learning systems. It also proves the practical use of HE and ZKP in secure, privacy-aware collaborative AI. Full article

This paper presents a second examination of trigonometric step sizes and their impact on Warm Restart Stochastic Gradient Descent (SGD), an essential optimization technique in deep learning. Building on prior work with cosine-based step sizes, this study introduces three novel trigonometric step sizes aimed at enhancing warm restart methods. These step sizes are formulated to address the challenges posed by non-smooth and non-convex objective functions, ensuring that the algorithm can converge effectively toward the global minimum. Through rigorous theoretical analysis, we demonstrate that the proposed approach achieves an $O\left({\displaystyle \frac{1}{\sqrt{T}}}\right)$ convergence rate for smooth non-convex functions and extend the analysis to non-smooth and non-convex scenarios. Experimental evaluations on FashionMNIST, CIFAR10, and CIFAR100 datasets reveal significant improvements in test accuracy, including a notable $2.14\%$ increase on CIFAR100 compared to existing warm restart strategies. These results underscore the effectiveness of trigonometric step sizes in enhancing optimization performance for deep learning models. Full article

As an important branch of machine learning, federated learning still suffers from statistical heterogeneity. Therefore, personalized federated learning (PFL) is proposed to deal with this obstacle. However, the privacy of local and global gradients is still under threat in the scope of PFL. Additionally, the correctness of the aggregated result is unable to be identified. Therefore, we propose a secure and verifiable personalized federated learning protocol that could protect privacy using homomorphic encryption and verify the aggregated result using Lagrange interpolation and commitment. Furthermore, it could resist the collusion attacks performed by servers and clients who try to pass verification. Comprehensive theoretical analysis is provided to verify our protocol’s security. Extensive experiments on MNIST, Fashion-MNIST and CIFAR-10 are carried out to demonstrate the effectiveness of our protocol. Our model achieved accuracies of 88.25% in CIFAR-10, 99.01% in MNIST and 96.29% in Fashion-MNIST. The results show that our protocol could improve security while maintaining the classification accuracy of the training model. Full article

How can human feedback be effectively integrated into generative models? This study addresses this question by proposing a method to enhance image denoising and achieve domain adaptation using human feedback. Deep generative models, while achieving remarkable performance in image denoising within training domains, often fail to generalize to unseen domains. To overcome this limitation, we introduce a novel approach that fine-tunes a denoising model using human feedback without requiring labeled target data. Our experiments demonstrate a significant improvement in denoising performance. For example, on the Fashion-MNIST test set, the peak signal-to-noise ratio (PSNR) increased by 94%, with an average improvement of 1.61 ± 2.78 dB and a maximum increase of 18.21 dB. Additionally, the proposed method effectively prevents catastrophic forgetting, as evidenced by the consistent performance on the original MNIST domain. By leveraging a reward model trained on human preferences, we show that the quality of denoised images can be significantly improved, even when applied to unseen target data. This work highlights the potential of human feedback for efficient domain adaptation in generative models, presenting a scalable and data-efficient solution for enhancing performance in diverse domains. Full article

Recently, convolutional neural networks (CNNs) have received a massive amount of interest due to their ability to achieve high accuracy in various artificial intelligence tasks. With the development of complex CNN models, a significant drawback is their high computational burden and memory requirements. The performance of a typical CNN model can be enhanced by the improvement of hardware accelerators. Practical implementations on field-programmable gate arrays (FPGA) have the potential to reduce resource utilization while maintaining low power consumption. Nevertheless, when implementing complex CNN models on FPGAs, these may may require further computational and memory capacities, exceeding the available capacity provided by many current FPGAs. An effective solution to this issue is to use quantized neural network (QNN) models to remove the burden of full-precision weights and activations. This article proposes an accelerator design framework for FPGAs, called FPGA-QNN, with a particular value in reducing high computational burden and memory requirements when implementing CNNs. To approach this goal, FPGA-QNN exploits the basics of quantized neural network (QNN) models by converting the high burden of full-precision weights and activations into integer operations. The FPGA-QNN framework comes up with 12 accelerators based on multi-layer perceptron (MLP) and LeNet CNN models, each of which is associated with a specific combination of quantization and folding. The outputs from the performance evaluations on Xilinx PYNQ Z1 development board proved the superiority of FPGA-QNN in terms of resource utilization and energy efficiency in comparison to several recent approaches. The proposed MLP model classified the FashionMNIST dataset at a speed of 953 kFPS with 1019 GOPs while consuming 2.05 W. Full article

Stereo-orientation selectivity is a fundamental neural mechanism in the brain that plays a crucial role in perception. However, due to the recognition process of high-dimensional spatial information commonly occurring in high-order cortex, we still know little about the mechanisms underlying stereo-orientation selectivity and lack a modeling strategy. A classical explanation for the mechanism of two-dimensional orientation selectivity within the primary visual cortex is based on the Hubel-Wiesel model, a cascading neural connection structure. The local-to-global information aggregation thought within the Hubel-Wiesel model not only contributed to neurophysiology but also inspired the development of computer vision fields. In this paper, we provide a clear and efficient conceptual understanding of stereo-orientation selectivity and propose a quantitative explanation for its generation based on the thought of local-to-global information aggregation within the Hubel-Wiesel model and develop an artificial visual system (AVS) for stereo-orientation recognition. Our approach involves modeling depth selective cells to receive depth information, simple stereo-orientation selective cells for combining distinct depth information inputs to generate various local stereo-orientation selectivity, and complex stereo-orientation selective cells responsible for integrating the same local information to generate global stereo-orientation selectivity. Simulation results demonstrate that our AVS is effective in stereo-orientation recognition and robust against spatial noise jitters. AVS achieved an overall over 90% accuracy on noise data in orientation recognition tasks, significantly outperforming deep models. In addition, the AVS contributes to enhancing deep models’ performance, robustness, and stability in 3D object recognition tasks. Notably, AVS enhanced the TransNeXt model in improving its overall performance from 73.1% to 97.2% on the 3D-MNIST dataset and from 56.1% to 86.4% on the 3D-Fashion-MNIST dataset. Our explanation for the generation of stereo-orientation selectivity offers a reliable, explainable, and robust approach for extracting spatial features and provides a straightforward modeling method for neural computation research. Full article