Skip to Content

875 Results Found

  • Article
  • Open Access
11 Citations
3,913 Views
12 Pages

Adaptive Gradient Estimation Stochastic Parallel Gradient Descent Algorithm for Laser Beam Cleanup

  • Shiqing Ma,
  • Ping Yang,
  • Boheng Lai,
  • Chunxuan Su and ,
  • Wang Zhao,
  • Kangjian Yang,
  • Ruiyan Jin,
  • Tao Cheng and
  • Bing Xu
  • + 4 authors+ 0 author

For a high-power slab solid-state laser, obtaining high output power and high output beam quality are the most important indicators. Adaptive optics systems can significantly improve beam qualities by compensating for the phase distortions of the las...

(This article belongs to the Special Issue Semiconductor Lasers)
  • Article
  • Open Access
3 Citations
2,509 Views
12 Pages

Pipelined Stochastic Gradient Descent with Taylor Expansion

  • Bongwon Jang,
  • Inchul Yoo and
  • Dongsuk Yook

26 October 2023

Stochastic gradient descent (SGD) is an optimization method typically used in deep learning to train deep neural network (DNN) models. In recent studies for DNN training, pipeline parallelism, a type of model parallelism, is proposed to accelerate SG...

(This article belongs to the Special Issue Application, Optimization and Architecture of Deep Learning Neural Network)
  • Article
  • Open Access
7 Citations
3,793 Views
22 Pages

21 May 2019

In the existing stochastic gradient matching pursuit algorithm, the preliminary atomic set includes atoms that do not fully match the original signal. This weakens the reconstruction capability and increases the computational complexity. To solve the...

(This article belongs to the Special Issue Sensor Signal and Information Processing II)
  • Article
  • Open Access
5 Citations
6,682 Views
7 Pages

15 January 2020

This paper is a step towards developing a geometric understanding of a popular algorithm for training deep neural networks named stochastic gradient descent (SGD). We built upon a recent result which observed that the noise in SGD while training typi...

(This article belongs to the Section Information Theory, Probability and Statistics)
  • Article
  • Open Access
11 Citations
4,053 Views
16 Pages

This paper mainly proposes some improved stochastic gradient descent (SGD) algorithms with a fractional order gradient for the online optimization problem. For three scenarios, including standard learning rate, adaptive gradient learning rate, and mo...

(This article belongs to the Section Engineering)
  • Article
  • Open Access
3 Citations
5,847 Views
15 Pages

A Natural Gradient Algorithm for Stochastic Distribution Systems

  • Zhenning Zhang,
  • Huafei Sun,
  • Linyu Peng and
  • Lin Jiu

4 August 2014

In this paper, we propose a steepest descent algorithm based on the natural gradient to design the controller of an open-loop stochastic distribution control system (SDCS) of multi-input and single output with a stochastic noise. Since the control in...

  • Article
  • Open Access
10 Citations
5,150 Views
16 Pages

Stochastic gradient descent is the method of choice for solving large-scale optimization problems in machine learning. However, the question of how to effectively select the step-sizes in stochastic gradient descent methods is challenging, and can gr...

(This article belongs to the Special Issue Recent Advances in Fractional Differential Equations and Their Applications)
  • Article
  • Open Access
7 Citations
3,160 Views
37 Pages

A Family of Hybrid Stochastic Conjugate Gradient Algorithms for Local and Global Minimization Problems

  • Khalid Abdulaziz Alnowibet,
  • Salem Mahdi,
  • Ahmad M. Alshamrani,
  • Karam M. Sallam and
  • Ali Wagdy Mohamed

1 October 2022

This paper contains two main parts, Part I and Part II, which discuss the local and global minimization problems, respectively. In Part I, a fresh conjugate gradient (CG) technique is suggested and then combined with a line-search technique to obtain...

(This article belongs to the Special Issue Development and Optimization of Mathematical Models for Operations Research)
  • Feature Paper
  • Article
  • Open Access
10 Citations
5,819 Views
24 Pages

11 February 2018

The development of accurate and efficient image reconstruction algorithms is a central aspect of quantitative photoacoustic tomography (QPAT). In this paper, we address this issues for multi-source QPAT using the radiative transfer equation (RTE) as...

(This article belongs to the Special Issue Probabilistic Methods for Inverse Problems)
  • Article
  • Open Access
6 Citations
4,350 Views
16 Pages

28 October 2019

Gradient descent method is an essential algorithm for learning of neural networks. Among diverse variations of gradient descent method that have been developed for accelerating learning speed, the natural gradient learning is based on the theory of i...

(This article belongs to the Special Issue Applications of AI for 5G and Beyond Communications: Network Management, Operation, and Automation)
  • Article
  • Open Access
12 Citations
3,533 Views
12 Pages

Damped Newton Stochastic Gradient Descent Method for Neural Networks Training

  • Jingcheng Zhou,
  • Wei Wei,
  • Ruizhi Zhang and
  • Zhiming Zheng

29 June 2021

First-order methods such as stochastic gradient descent (SGD) have recently become popular optimization methods to train deep neural networks (DNNs) for good generalization; however, they need a long training time. Second-order methods which can lowe...

(This article belongs to the Special Issue Statistical Data Modeling and Machine Learning with Applications)
  • Feature Paper
  • Article
  • Open Access
246 Views
16 Pages

Mini-Batch Pruning Techniques in Stochastic Gradient Descent Methods

  • Roberto Cavoretto,
  • Alessandra De Rossi and
  • Enrico Scibilia

1 October 2026

Mini-batch Stochastic Gradient Descent (SGD) is a standard optimization method for training neural networks and other large-scale parametric models. We study three loss-based mini-batch pruning strategies—Perceptron-Inspired SGD (PISGD), Prunin...

(This article belongs to the Section E: Applied Mathematics)
  • Article
  • Open Access
3 Citations
1,685 Views
15 Pages

16 August 2022

This paper researches the identification problem for the unknown parameters of the multivariate equation-error autoregressive systems. Firstly, the original identification model is decomposed into several sub-identification models according to the nu...

(This article belongs to the Section E2: Control Theory and Mechanics)
  • Review
  • Open Access
311 Citations
30,438 Views
23 Pages

29 January 2023

In the age of artificial intelligence, the best approach to handling huge amounts of data is a tremendously motivating and hard problem. Among machine learning models, stochastic gradient descent (SGD) is not only simple but also very effective. This...

(This article belongs to the Section E: Applied Mathematics)
  • Article
  • Open Access
1 Citations
3,995 Views
18 Pages

12 November 2019

We consider estimating the marginal likelihood in settings with independent and identically distributed (i.i.d.) data. We propose estimating the predictive distributions in a sequential factorization of the marginal likelihood in such settings by usi...

(This article belongs to the Special Issue MaxEnt 2019—The 39th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering)
  • Article
  • Open Access
8 Citations
2,308 Views
17 Pages

23 February 2023

Stochastic configuration network (SCN) is a mathematical model of incremental generation under a supervision mechanism, which has universal approximation property and advantages in data modeling. However, the efficiency of SCN is affected by some net...

(This article belongs to the Special Issue Adaptive Dynamic Programming and Control Application in Intelligent Systems)
  • Article
  • Open Access
3,280 Views
19 Pages

13 February 2023

A multilevel Monte Carlo (MLMC) method is applied to simulate a stochastic optimal problem based on the gradient projection method. In the numerical simulation of the stochastic optimal control problem, the approximation of expected value is involved...

  • Article
  • Open Access
13 Citations
5,458 Views
15 Pages

12 September 2022

The Deep Operator Network (DeepONet) framework is a different class of neural network architecture that one trains to learn nonlinear operators, i.e., mappings between infinite-dimensional spaces. Traditionally, DeepONets are trained using a centrali...

(This article belongs to the Special Issue Gradient Methods for Optimization)
  • Article
  • Open Access
4 Citations
3,978 Views
21 Pages

The stochastic gradient matching pursuit algorithm requires the sparsity of the signal as prior information. However, this prior information is unknown in practical applications, which restricts the practical applications of the algorithm to some ext...

(This article belongs to the Special Issue Signal Processing and Analysis of Electrical Circuit)
  • Article
  • Open Access
6 Citations
2,499 Views
16 Pages

A Novel Sine Step Size for Warm-Restart Stochastic Gradient Descent

  • Mahsa Soheil Shamaee and
  • Sajad Fathi Hafshejani

6 December 2024

This paper proposes a novel sine step size for warm-restart stochastic gradient descent (SGD). For the SGD based on the new proposed step size, we establish convergence rates for smooth non-convex functions with and without the Polyak–Łoja...

(This article belongs to the Section Mathematical Analysis)
  • Article
  • Open Access
4 Citations
2,953 Views
18 Pages

A Scalable Bayesian Sampling Method Based on Stochastic Gradient Descent Isotropization

  • Giulio Franzese,
  • Dimitrios Milios,
  • Maurizio Filippone and
  • Pietro Michiardi

28 October 2021

Stochastic gradient sg-based algorithms for Markov chain Monte Carlo sampling (sgmcmc) tackle large-scale Bayesian modeling problems by operating on mini-batches and injecting noise on sgsteps. The sampling properties of these algorithms are determin...

(This article belongs to the Special Issue Approximate Bayesian Inference)
  • Feature Paper
  • Article
  • Open Access
37 Citations
4,847 Views
21 Pages

20 August 2022

This paper presents an adaptive PID using stochastic gradient descent with momentum (SGDM) for a proton exchange membrane fuel cell (PEMFC) power system. PEMFC is a nonlinear system that encounters external disturbances such as inlet gas pressures an...

(This article belongs to the Special Issue Microgrids Integrating Renewable Energy Sources, Fuel Cells and Plug-In Hybrid Electric Vehicles)
  • Article
  • Open Access
18 Citations
5,225 Views
21 Pages

30 September 2025

Wavefront aberrations caused by thermal flows or arising from the quality of optical components can significantly impair wireless communication links. Such aberrations may result in an increased error rate in the received signal, leading to data loss...

(This article belongs to the Section Information Systems)
  • Article
  • Open Access
4 Citations
1,930 Views
11 Pages

Stochastic Gradient Descent for Kernel-Based Maximum Correntropy Criterion

  • Tiankai Li,
  • Baobin Wang,
  • Chaoquan Peng and
  • Hong Yin

17 December 2024

Maximum correntropy criterion (MCC) has been an important method in machine learning and signal processing communities since it was successfully applied in various non-Gaussian noise scenarios. In comparison with the classical least squares method (L...

(This article belongs to the Section Information Theory, Probability and Statistics)
  • Article
  • Open Access
4 Citations
1,747 Views
25 Pages

27 April 2024

The uncertainty in the new power system has increased, leading to limitations in traditional stability analysis methods. Therefore, considering the perspective of the three-dimensional static security region (SSR), we propose a novel approach for sys...

(This article belongs to the Section Electrical, Electronics and Communications Engineering)
  • Article
  • Open Access
6 Citations
2,412 Views
21 Pages

In this work, we explored, for the first time, to the best of our knowledge, the potential of stochastic gradient descent (SGD) to optimize random phase functions for application in non-iterative phase-only hologram generation. We defined and evaluat...

  • Article
  • Open Access
11 Citations
3,412 Views
15 Pages

A Novel Sophia-SPGD (Stochastic Parallel Gradient Descent) Optimization Method for Wavefront Correction in WFS-Less AO (Wavefront Sensorless Adaptive Optics) Systems

  • Peng Chen,
  • Wenjie Yang,
  • Yongqi Ge,
  • Zhiguang Zhang and ,
  • Xianshuo Li,
  • Zhengqing Qi and
  • Huizhen Yang
  • + 2 authors+ 0 author

In wavefront sensorless adaptive optics (WFS-less AO) systems, stochastic parallel gradient descent (SPGD) is the primary optimization method for correcting wavefront distortions. However, as the intensity of atmospheric turbulence interference incre...

  • Article
  • Open Access
12 Citations
6,737 Views
14 Pages

16 March 2022

Supply chain management (SCM) is considered at the forefront of many organizations in the delivery of their products. Various optimization methods are applied in the SCM to improve the efficiency of the process. In this research, the projected stocha...

(This article belongs to the Section Sustainable Engineering and Science)
  • Article
  • Open Access
12 Citations
5,924 Views
15 Pages

17 May 2020

This paper demonstrates a novel approach to training deep neural networks using a Mutual Information (MI)-driven, decaying Learning Rate (LR), Stochastic Gradient Descent (SGD) algorithm. MI between the output of the neural network and true outcomes...

(This article belongs to the Section Information Theory, Probability and Statistics)
  • Feature Paper
  • Article
  • Open Access
3 Citations
3,308 Views
9 Pages

This paper proposes a novel mathematical theory of adaptation to convexity of loss functions based on the definition of the condense-discrete convexity (CDC) method. The developed theory is considered to be of immense value to stochastic settings and...

(This article belongs to the Special Issue Deep Learning Techniques for Big Data Analysis)
  • Article
  • Open Access
28 Citations
2,416 Views
16 Pages

Decentralized Stochastic Recursive Gradient Method for Fully Decentralized OPF in Multi-Area Power Systems

  • Umair Hussan,
  • Huaizhi Wang,
  • Muhammad Ahsan Ayub,
  • Hamna Rasheed and ,
  • Muhammad Asghar Majeed,
  • Jianchun Peng and
  • Hui Jiang
  • + 2 authors+ 0 author

30 September 2024

This paper addresses the critical challenge of optimizing power flow in multi-area power systems while maintaining information privacy and decentralized control. The main objective is to develop a novel decentralized stochastic recursive gradient (DS...

(This article belongs to the Special Issue Artificial Intelligence and Game Theory)
  • Article
  • Open Access
10 Citations
3,155 Views
20 Pages

A Novel Framework for Abnormal Risk Classification over Fetal Nuchal Translucency Using Adaptive Stochastic Gradient Descent Algorithm

  • Deepti Verma,
  • Shweta Agrawal,
  • Celestine Iwendi,
  • Bhisham Sharma and ,
  • Surbhi Bhatia and
  • Shakila Basheer
  • + 1 author+ 0 author

31 October 2022

In most maternity hospitals, an ultrasound scan in the mid-trimester is now a standard element of antenatal care. More fetal abnormalities are being detected in scans as technology advances and ability improves. Fetal anomalies are developmental abno...

(This article belongs to the Special Issue Deep Disease Detection and Diagnosis Models)
  • Article
  • Open Access
10 Citations
2,318 Views
30 Pages

Photovoltaic/Hydrokinetic/Hydrogen Energy System Sizing Considering Uncertainty: A Stochastic Approach Using Two-Point Estimate Method and Improved Gradient-Based Optimizer

  • Mustafa Kamal,
  • Renzon Daniel Cosme Pecho,
  • Hassan Falah Fakhruldeen,
  • Hailer Sharif and ,
  • Vedran Mrzljak,
  • Saber Arabi Nowdeh and
  • Igor Poljak
  • + 2 authors+ 0 author

4 November 2023

In this paper, stochastic sizing of a stand-alone Photovoltaic/Hydrokinetic/Hydrogen storage energy system is performed with aim of minimizing the cost of project life span (COPL) and satisfying the reliability index as probability of load shortage (...

  • Article
  • Open Access
3 Citations
6,280 Views
13 Pages

31 July 2017

Due to the lack of powerful model description methods, the identification of Hammerstein systems based on the non-uniform input-output dataset remains a challenging problem. This paper introduces a time-varying backward shift operator to describe per...

  • Article
  • Open Access
20 Citations
3,716 Views
20 Pages

2 October 2024

This paper presents an innovative control strategy for robot arm manipulators, utilizing an adaptive sliding mode control with stochastic gradient descent (ASMCSGD). The ASMCSGD controller significant improvements in robustness, chattering eliminatio...

  • Article
  • Open Access
48 Citations
5,505 Views
25 Pages

17 March 2022

Landslides frequently occur along the eastern margin of the Tibetan Plateau, which poses a risk to the construction, maintenance, and transportation of the proposed Dujiangyan city to Siguniang Mountain (DS) railway, China. Therefore, four advanced m...

(This article belongs to the Special Issue Landslides Analysis and Management: From Data Acquisition to Modelling and Monitoring)
  • Proceeding Paper
  • Open Access
1,895 Views
9 Pages

Existing algorithms like nested sampling and annealed importance sampling are able to produce accurate estimates of the marginal likelihood of a model, but tend to scale poorly to large data sets. This is because these algorithms need to recalculate...

(This article belongs to the Proceedings of The 39th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering)
  • Article
  • Open Access
7 Citations
2,868 Views
23 Pages

Identification of Multi-Innovation Stochastic Gradients with Maximum Likelihood Algorithm Based on Ship Maneuverability and Wave Peak Models

  • Yang Liu,
  • Qiang Zhang,
  • Longjin Wang,
  • Shun An and ,
  • Yan He,
  • Zhimin Fan and
  • Fang Deng
  • + 2 authors+ 0 author

This paper investigates the problem of real-time parameter identification for ship maneuvering parameters and wave peak frequency in an ocean environment. Based on the idea of Euler discretion, a combined model of ship maneuvering and wave peak frequ...

(This article belongs to the Section Ocean Engineering)
  • Feature Paper
  • Article
  • Open Access
9 Citations
7,381 Views
23 Pages

An Exploration Algorithm for Stochastic Simulators Driven by Energy Gradients

  • Anastasia S. Georgiou,
  • Juan M. Bello-Rivas,
  • Charles William Gear,
  • Hau-Tieng Wu and ,
  • Eliodoro Chiavazzo and
  • Ioannis G. Kevrekidis
  • + 1 author+ 0 author

22 June 2017

In recent work, we have illustrated the construction of an exploration geometry on free energy surfaces: the adaptive computer-assisted discovery of an approximate low-dimensional manifold on which the effective dynamics of the system evolves. Constr...

(This article belongs to the Special Issue Understanding Molecular Dynamics via Stochastic Processes)
  • Article
  • Open Access
2 Citations
2,076 Views
26 Pages

31 July 2024

In this paper, we propose a stochastic primal-dual adaptive method based on an inexact augmented Lagrangian function to solve non-convex programs, referred to as the SPDAM. Different from existing methods, SPDAM incorporates adaptive step size and mo...

(This article belongs to the Special Issue Stochastic System Analysis and Control)
  • Article
  • Open Access
2 Citations
2,433 Views
31 Pages

Stochastic Zeroth-Order Multi-Gradient Algorithm for Multi-Objective Optimization

  • Zhihao Li,
  • Qingtao Wu,
  • Moli Zhang,
  • Lin Wang and ,
  • Youming Ge and
  • Guoyong Wang
  • + 1 author+ 0 author

14 February 2025

Multi-objective optimization (MOO) has become an important method in machine learning, which involves solving multiple competing objective problems simultaneously. Nowadays, many MOO algorithms assume that gradient information is easily available and...

  • Article
  • Open Access
2,362 Views
18 Pages

27 May 2024

For large-scale optimization that covers a wide range of optimization problems encountered frequently in machine learning and deep neural networks, stochastic optimization has become one of the most used methods thanks to its low computational comple...

  • Article
  • Open Access
2,930 Views
21 Pages

Stochastic Recursive Gradient Support Pursuit and Its Sparse Representation Applications

  • Fanhua Shang,
  • Bingkun Wei,
  • Yuanyuan Liu,
  • Hongying Liu and ,
  • Shuang Wang and
  • Licheng Jiao
  • + 1 author+ 0 author

30 August 2020

In recent years, a series of matching pursuit and hard thresholding algorithms have been proposed to solve the sparse representation problem with ℓ0-norm constraint. In addition, some stochastic hard thresholding methods were also proposed, such as s...

(This article belongs to the Section Sensing and Imaging)
  • Article
  • Open Access
11 Citations
6,361 Views
15 Pages

9 March 2022

In recent years, deep neural networks (DNN) have been widely used in many fields. Lots of effort has been put into training due to their numerous parameters in a deep network. Some complex optimizers with many hyperparameters have been utilized to ac...

(This article belongs to the Topic Machine and Deep Learning)
  • Article
  • Open Access
8 Citations
3,321 Views
15 Pages

8 April 2020

Complex problems require considerable work, extensive computation, and the development of effective solution methods. Recently, physical hardware- and software-based technologies have been utilized to support problem solving with computers. However,...

(This article belongs to the Section Applied Industrial Technologies)
  • Feature Paper
  • Article
  • Open Access
28 Citations
5,536 Views
15 Pages

Parallel Implementation on FPGA of Support Vector Machines Using Stochastic Gradient Descent

  • Felipe F. Lopes,
  • João Canas Ferreira and
  • Marcelo A. C. Fernandes

Sequential Minimal Optimization (SMO) is the traditional training algorithm for Support Vector Machines (SVMs). However, SMO does not scale well with the size of the training set. For that reason, Stochastic Gradient Descent (SGD) algorithms, which h...

(This article belongs to the Section Circuit and Signal Processing)
  • Article
  • Open Access
1 Citations
2,349 Views
24 Pages

Modification of Learning Ratio and Drop-Out for Stochastic Gradient Descendant Algorithm

  • Adrian Teso-Fz-Betoño,
  • Ekaitz Zulueta,
  • Mireya Cabezas-Olivenza,
  • Unai Fernandez-Gamiz and
  • Carlos Botana-M-Ibarreta

28 February 2023

The stochastic gradient descendant algorithm is one of the most popular neural network training algorithms. Many authors have contributed to modifying or adapting its shape and parametrizations in order to improve its performance. In this paper, the...

  • Article
  • Open Access
1,846 Views
44 Pages

21 May 2025

Recently, many stochastic Alternating Direction Methods of Multipliers (ADMMs) have been proposed to solve large-scale machine learning problems. However, for large-scale saddle-point problems, the state-of-the-art (SOTA) stochastic ADMMs still have...

  • Article
  • Open Access
13 Citations
2,854 Views
16 Pages

23 October 2021

This paper focuses on the nonlinear system identification problem, which is a basic premise of control and fault diagnosis. For Hammerstein output-error nonlinear systems, we propose an auxiliary model-based multi-innovation fractional stochastic gra...

(This article belongs to the Special Issue Deep Learning-Based Machinery Fault Diagnostics)
  • Article
  • Open Access
5 Citations
3,277 Views
13 Pages

Forex Investment Optimization Using Instantaneous Stochastic Gradient Ascent—Formulation of an Adaptive Machine Learning Approach

  • Iqbal Murtza,
  • Ayesha Saadia,
  • Rabia Basri,
  • Azhar Imran and ,
  • Abdullah Almuhaimeed and
  • Abdulkareem Alzahrani
  • + 1 author+ 0 author

18 November 2022

In the current complex financial world, paper currencies are vulnerable and unsustainable due to many factors such as current account deficit, gold reserves, dollar reserves, political stability, security, the presence of war in the region, etc. The...

(This article belongs to the Special Issue Business, Innovation, and Economics Sustainability)

of 18