Hardware-Aware Neural Architecture Search for Real-Time Video Processing in FPGA-Accelerated Endoscopic Imaging

Abstract

Medical endoscopic video processing requires real-time execution of color component acquisition, color filter array (CFA) demosaicing, and high dynamic range (HDR) compression under low-light conditions, while adhering to strict thermal constraints within the surgical handpiece. Traditional hardware-aware neural architecture search (NAS) relies on fixed hardware design spaces, making it difficult to balance accuracy, power consumption, and real-time performance. A collaborative “power-accuracy” optimization method is proposed for hardware-aware NAS. Firstly, we proposed a novel hardware modeling framework by abstracting FPGA heterogeneous resources into unified cell units and establishing a power–temperature closed-loop model to ensure that the handpiece surface temperature does not exceed clinical thresholds. In this framework, we constrained the interstage latency balance in pipelines to avoid routing congestion and frequency degradation caused by deep pipelines. Then, we optimized the NAS strategy by using pipeline blocks and combined with a hardware efficiency reward function. Finally, color component acquisition, CFA demosaicing, dynamic range compression, dynamic precision quantization, and streaming architecture are integrated into our framework. Experiments demonstrate that the proposed method achieves 2.8 W power consumption at 47 °C on a Xilinx ZCU102 platform, with a 54% improvement in throughput (vs. hardware-aware NAS), providing an engineer-ready lightweight network for medical edge devices such as endoscopes.

1. Introduction

Medical endoscopic video processing faces extreme thermal constraints requiring power consumption below 3 W to maintain surgical handpiece temperatures at or under 50 °C [1,2,3], driven by the miniaturization of devices with diameters not exceeding 3 cm [4,5]. Simultaneously, it demands real-time, frame-by-frame execution of computationally intensive tasks, including high dynamic range compression [6,7,8,9] and continuous color reconstruction under low-light conditions [10] to ensure diagnostic-quality video output for clinical decision-making. Data acquisition modules in medical devices such as white light/fluorescence endoscopes must be deployed directly within the surgical handpiece, where thermal management and power consumption are severely limited [10]. Excessive temperature may burn patient tissue and interfere with surgical operation, and the miniaturized design (diameter ≤3 cm) further restricts heat dissipation capability [4]. Moreover, to achieve clinically usable image quality under low illumination, the system must perform complex processing at the lens end, including color filter array (CFA) acquisition, real-time exposure adjustment, tone mapping, and high dynamic range (HDR) compression [6]. Conventional solutions relying on CPU/GPU computation often exceed 10 W of power consumption (e.g., NVIDIA Jetson AGX Xavier (NVIDIA, Santa Clara, CA, USA)) [11], causing the handpiece surface temperature to exceed 60 °C [1], which fails to meet surgical safety requirements.

Hardware-aware neural architecture search (NAS) has shown significant potential in medical imaging, particularly for low-light endoscopic image reconstruction tasks, such as CFA demosaicing and dynamic range compression [12,13]. It can automatically generate lightweight network architectures to achieve high accuracy and real-time processing under strict power constraints while avoiding the overparameterization issues of manually designed networks [14,15,16,17,18]. Deep learning methods have been widely adopted in image reconstruction in recent years and significantly outperform traditional algorithms in terms of reconstruction quality [19,20,21]. However, manually designed networks often suffer from overparameterization and are difficult to adapt to hardware resource limits [22,23]. Neural architecture search (NAS), which automates network structure generation and optimizes the balance between accuracy and efficiency for specific scenarios, offers a viable path to overcome the bottlenecks of medical edge computing [14]. However, most existing NAS methods rely on GPU-based searches and do not consider the joint constraints of power, temperature, and area [24,25], making the generated models difficult to deploy directly on medical devices [26,27].

In the context of medical endoscopic imaging, FPGAs have become the hardware of choice due to their reconfigurability, high parallel computing capability, and ultralow power consumption, making them essential for meeting the ≤3 W power constraint in surgical handpieces as seen in

Table 1. Performance comparison of hardware platforms for endoscopic imaging.

Platform	Power (W)	Temperature (°C)	Latency (ms)	Frame Rate (fps)	CPSNR (db)	Efficiency (fps/W)
NVIDIA-Jetson-AGX	18.2	65	12.3	81.3	38.8	4.47
Xilinx-Zynq-(FPGA)	2.8	48	8.1	123.5	38.2	44.1
ARM Cortex-A72	4.5	54	41.7	24.0	36.5	5.33
TI TDA4VM (DSP)	5.2	52	28.9	34.6	37.1	6.65
SiFive U740 (RISC-V)	3.1	49	62.4	16.0	35.8	5.16

Data sourced from official technical documentation of Xilinx, Texas Instruments (TI), and Microsoft, supplemented by Reference [29].

. Field-programmable gate array is a reconfigurable silicon chip whose circuitry can be rewired post-manufacturing—analogous to building custom Lego machinery rather than using pre-assembled toys. This enables ultra-parallel processing at 1/3 the power of CPUs/GPUs, critical for real-time medical imaging under thermal constraints. The core of an FPGA consists of programmable logic units such as Xilinx CLBs/Intel ALMs, block memory (BRAM), and digital signal processors (DSPs). Through pipelined architectures and dataflow optimization, significant improvements in energy efficiency can be achieved. For example, the Xilinx Zynq-7000 (xilinx, San, Jose, CA, USA) consumes only 1/6 the power of a comparable GPU in image processing tasks while supporting hardware-level customized computing unit design, providing fundamental support for real-time low-power medical imaging [28].

To address the conflicting challenges of thermal dissipation limitations in surgical handpieces and high-accuracy real-time imaging under low-light conditions in medical endoscopy, this paper proposes a hardware-aware NAS framework oriented toward FPGA deployment, establishing for the first time a closed-loop connection between power constraints and neural architecture search. First, FPGA resources are abstracted into unified cell units, and surgical temperature thresholds are converted into upper limits on the number of cells via a thermodynamic model. Second, a “pipeline block” is introduced as a basic unit to jointly optimize network operations and balance the interstage pipeline latency, thereby avoiding frequency degradation caused by deep pipelines. Third, an enumeration-pruning strategy is adopted to generate plug-and-play lightweight networks that achieve 38.2 dB CPSNR reconstruction accuracy under a 90-cell constraint while operating at a low temperature of 47 °C, promoting the practical deployment of endoscopic imaging systems.

2. Related Work

2.1. Technologies for Low-Power Medical Imaging

Medical endoscopic imaging requires the acquisition of color components (e.g., Bayer patterns) through a color filter array (CFA) in low-light environments [30,31], followed by real-time processing, such as demosaicing [32], dynamic range compression [33,34], and tone mapping [35], to reconstruct high-fidelity color images for clinical diagnosis. However, such computationally intensive tasks (e.g., convolution operations, feature extraction) often exceed 10 W power consumption on traditional GPU/CPU platforms [36], causing the surgical handpiece temperature to rise above 60 °C [37]—far exceeding the clinical safety threshold [38,39]. Thus, extreme power constraints and real-time requirements pose core challenges for medical edge devices [40,41,42].

Power-constrained design has become a critical need for edge medical devices [43,44]. The existing low-power techniques can be categorized into three types: (1) Dynamic precision quantization, which replaces floating-point arithmetic with fixed-point arithmetic to reduce computational complexity, achieving a 10× improvement in GPU energy efficiency in VGG8 models [45,46], albeit with a 3% accuracy loss. Layer-specific bit width adjustment applies different integer/fractional bit lengths to various network layers to balance accuracy and resources. (2) Memory optimization strategies include a pyramid-structured layer fusion, as shown as Figure 1, which reuses interlayer data in a multi-pyramid architecture, reducing off-chip memory access by 95% [47,48], although it increases dataflow control complexity. Alternatively, a streaming architecture employing “ping-pong buffers” masks data transfer latency, achieving 81.2 GFLOPS computational throughput on the ZCU102 platform with only 25% CPU power consumption [49,50,51,52]. (3) Computational optimization techniques involve loop unrolling and pipelining, which improve the convolution efficiency via parallel unrolling factors, reducing the latency from 1110 ms to 46.6 ms [53]. The use of binarized neural networks (BNNs), which replace multipliers with XNOR gates, as seen in Figure 2, achieves 10× higher energy efficiency than do GPUs at 150 MHz [54,55], although they are only suitable for low-complexity tasks [29,56,57,58].

This paper adopts proactive hardware-constrained modeling. To address the limitations of existing techniques that neglect thermal constraint quantification and pipeline balancing, we propose the following:

• Cell Abstraction and Power–Temperature Closed-Loop Model: FPGA resources (LUTs, BRAM, DSP) are normalized into cell units, and a surgical handpiece temperature formula is established.
• Latency-Balancing Hard Constraint: Interstage pipeline latency variance is constrained to prevent frequency degradation caused by deep pipelines.
• Medical Task-Specific Operator Library: For CFA demosaicing, depthwise separable convolution is prioritized, achieving 38.2 dB CPSNR under a 90-cell constraint.

2.2. Advances in Hardware-Aware Neural Architecture Search

Neural architecture search (NAS) automates the generation of network structures to address the overparameterization issues of deep learning models [59,60,61,62]. Traditional NAS methods, such as the ENAS and reinforcement learning [63], focus primarily on accuracy optimization while ignoring hardware deployment constraints, such as power consumption, latency, and resource usage, making the generated models difficult to deploy on edge devices such as endoscopic handpieces [64]. The extreme power limits and real-time requirements of medical imaging scenarios necessitate the integration of hardware characteristics into the NAS process, giving rise to the research direction of hardware-aware NAS (HW-NAS) [65,66].

Existing hardware-aware NAS techniques can be categorized into three types. 1. Accuracy-Oriented NAS: This method uses validation accuracy as a single objective of minimizing negative CPSNR and employs exhaustive or grid search to generate lightweight networks of DMCNN-VD [67]. However, it fails to co-optimize hardware metrics, resulting in low resource efficiency. 2. Hardware-Aware Extended NAS: This method introduces multi-objective evaluation functions incorporating metrics such as latency and power and tests candidate architectures for hardware efficiency on FPGAs [68]. Limitations include a fixed hardware design space and a lack of co-optimization between the network structure and hardware parallelism parameters, leading to limited throughput improvement. 3. Co-exploration Framework: Jointly searches network architecture and hardware FPGA pipeline partitioning parameters to balance accuracy and hardware efficiency. Compared with the conventional HW-NAS, the approach in Reference [1] improves throughput by 35.24% and energy efficiency by 54.05% [27,69]. This method does not quantify medical thermal constraints and relies on reinforcement learning, which requires tens of thousands of iterations, resulting in high search costs.

Traditional ENAS and hardware-aware NAS are optimized primarily for accuracy on GPU/CPU platforms and overlook hardware deployment constraints. 1. Fixed Hardware Assumption: Early hardware-aware NAS approaches [27,70,71] assume a fixed FPGA accelerator design and optimize only the network structure, leading to inefficient resource usage, where BRAM/DSP are not adapted to pipeline depth. Single-Objective Optimization Limitation: As noted in Ref. [72], most existing methods use accuracy as the sole objective and fail to co-optimize latency and power. GPU solutions consume up to 150 W, exceeding the thermal constraints of handpieces. 2. Low Search Efficiency: The sequential search process in Ref. [73], from architecture generation to hardware evaluation, requires tens of thousands of iterations, which is time-consuming and cannot guarantee deployability.

To address these limitations, this paper proposes proactive hardware-constrained modeling and thermal-aware optimization: proactive hardware-constrained modeling, which abstracts FPGA resources (LUTs, BRAM, DSP) into unified cell units. Latency Balancing Hard Constraint: Restricting interstage pipeline latency variance.

2.3. FPGA Acceleration Practices in Medical Imaging

Field-programmable gate arrays (FPGAs) consist of configurable logic blocks (e.g., LUTs, BRAM, DSP), input/output modules, and an internal interconnected structure, supporting hardware-level reconfiguration [17,74,75]. Their core advantages include low power consumption, high parallelism, and low latency, which make them suitable for computation-intensive tasks. Power consumption on a 30 W platform is only 1/5 [76] of that of a GPU AMD, which has multiple computing engines executing concurrently with a sub-millisecond response. In medical imaging, FPGAs accelerate neural network inference through customized pipeline architectures such as separating convolutional and fully connected layers [77]. For example, in image reconstruction tasks such as CFA demosaicing and dynamic range compression, operations are mapped to FPGA computing engines, leveraging parallelism to improve throughput [78]. Real-time performance is ensured through streaming data transmission to hide memory access latency, meeting the clinical requirement of ≥30 FPS for endoscopy [79].

Medical image reconstruction tasks rely on low-latency processing. Key efforts include the following. 1. Streaming Architecture Optimization: Ref. [80] designed a pipeline separating convolutional and fully connected layers, as seen in Figure 3, achieving 81.2 GFLOPS on the ZCU102 platform with only 25% CPU power consumption. 2. Memory Access Optimization: Refs. [81,82] employed “ping-pong buffers” to hide data transfer latency but did not resolve frequency degradation caused by interstage load imbalance. A single-stage delay of 8 ns forced the overall frequency to drop to 125 MHz. Existing FPGA accelerators rely on manually designed networks such as VGG/ResNet [83,84] variants and cannot adapt to changes in hardware constraints. Pipeline restructuring is required when the handpiece size is reduced.

To address these shortcomings, this paper proposes a hardware/software co-optimization framework. (1) Latency-balancing constraint: This limits the interstage pipeline latency variance and dynamically adjusts parallelism through a streaming architecture, as in seen Figure 3, increasing the frequency to 192 MHz and improving throughput by 54%. (2) Resource abstraction: This method normalizes FPGA resources into cell units and establishes a power–temperature model to ensure a power consumption ≤ 3 W.

This approach achieves a balance between accuracy and energy efficiency in medical imaging, addressing the gaps in thermal constraint quantification and real-time optimization in conventional FPGA solutions.

3. Our Approach

3.1. Problem Formulation

Clinical requirements serve as the origin of all designs. The optical endoscopic scenario imposes two core and stringent constraints: First, the temperature must remain ≤50 °C. As the surgical handpiece must be held by the surgeon’s hand, its surface temperature must be strictly controlled below 50 °C to avoid tissue burns and interference during the operation. Second, surgical operations demand strong real-time performance. The imaging system must deliver smooth real-time video, typically requiring a frame rate ≥ 60 FPS to support precise surgical manipulation.

Medical endoscopic imaging must achieve real-time reconstruction under extreme power constraints while meeting the clinical requirements for handpiece surface temperature. Traditional hardware-aware NAS suffers from two major shortcomings: First, GPU/CPU-based solutions often exceed power limits, and imbalanced interstage pipeline latency leads to hardware mismatch. Second, the sequential NAS process (architecture generation → hardware evaluation) requires tens of thousands of iterations, offers no guarantee that the generated network is deployable, and suffers from low search efficiency.

The imaging system primarily consists of a camera handle, an image processing host, and a 4K monitor. It is capable of achieving long-distance real-time transmission of ultra-high-definition video data. On the 4K camera handle side, a core processor (FPGA) receives data from the ultra-high-definition image sensor and external control input information, then simultaneously transmits both the external control input information and the image data to the image processing host. On the image processing host side, it receives both the image data and external control input, performs image preprocessing and image enhancement processing, while also implementing image caching and display driving.

Hardware constraints translate clinical requirements into quantifiable and measurable indicators that the hardware platform must adhere to through physical modeling and engineering calculations. Algorithm design further maps these hardware constraints into specific rules and search space limitations for neural architecture search. For example, the temperature constraint is derived into a maximum allowable power consumption via a thermodynamic model [85]. The real-time requirement is directly converted into a hard indicator for total system latency.

$$T_{\mathit{handpiece}}={ T}_{\mathit{environment}}+{ P}_{\mathit{total}} \times { R}_{\mathit{thermal} \mathit{resistance}}+0.1P_{\mathit{FPGA}}$$

(1)

T_handpiece represents the clinical safety threshold for handpiece surface temperature. T_env denotes the standard operating room ambient temperature. R_thermal quantifies the thermal resistance of copper heat sinks. P_total defines the upper limit of power consumption. The coefficient 0.1 is derived from measured data: for every 10 W increase in handpiece power consumption, the ambient temperature near the surgical table rises by 1 °C, and the handpiece temperature increases by 0.1 °C.

Cell abstraction normalizes power constraints and the heterogeneous resources of the FPGA through a cell abstraction model, thereby calculating the maximum allowable number of cells. This directly defines the upper limit of the “area” of the network model.

Through cell abstraction, regardless of whether the underlying FPGA is from Xilinx, Intel, or Lattice, as seen in

Table 2. Comparison of FPGA heterogeneous resources and unified cell equivalent coefficients.

Vendor	Resource Type	Property	Coefficient	Note
Xilinx	1 LUT6	Logic	1.5 Cell	Programmable combinational logic
	1 Flip-Flop (FF)	Logic	0.5 Cell	Constructs and registers sequential logic
	1 BRAM (18 Kb)	Store	20 Cell	Implements FIFO, RAM, ROM, etc.
	1 DSP48 Slice	Calculate	10 Cell	Efficiently executes multiply accumulate (MAC) operations
	1 MMCM	Clock	15 Cell	Mixed-mode clock manager (MMCM)
	1 SERDES	Interface	25 Cell	Used for high-speed serial interfaces
Intel (Altera)	1 ALM	Logic	1.8 Cell	Adaptive logic module (ALM)
	1 Register	Logic	0.5 Cell	Similar to Flip-Flop; functions as a storage unit
	1 M20 K Block	Store	22 Cell	Functionally equivalent to Xilinx BRAM
	1 DSP Block	Calculate	12 Cell	Digital signal processing (DSP) slice
	1 PLL	Clock	14 Cell	Phase-locked loop (PLL) for clock generation and management
	1 Transceiver	Interface	28 Cell	High-speed transceivers for serial protocols
Lattice	1 LUT4	Logic	1.0 Cell	4-input look-up table (LUT)
	1 PFU	Logic	4.0 Cell	Programmable functional unit supporting multiple resources
	1 EBR	Store	18 Cell	Embedded block RAM (BRAM) for storage
	1 sysDSP Block	Calculate	9 Cell	Dedicated DSP computing unit
	1 PLL	Clock	13 Cell	Phase-locked loop (PLL) for clock management

Data was sourced from the official technical documentation of Xilinx, Intel, and Lattice Semiconductor. The coefficients were estimated using the method proposed in this paper to highlight differences across manufacturers.

, the same standardized metrics can be used to evaluate the “size,” “power consumption,” and “cost” of an architecture. This enables the unification of both the design and evaluation of the hardware-aware NAS space. The abstraction methodology is as follows. 1. Utilize vendor-specific tools for design synthesis, placement, and routing. 2. Power reports and resource utilization reports are extracted under various resource configurations. 3. Accurate power and resource models can be established through methods such as regression analysis, thereby determining the final cell conversion coefficients.

Latency constraints and the pipeline characteristics of the FPGA are translated into a hard requirement for interstage latency balance, preventing any single stage’s excessive delay from degrading the overall system frequency and throughput, thereby achieving pipeline equilibrium. The neural network is partitioned into M pipeline stages (e.g., P₁, P₂, …, P_m), with each stage mapped to a pipeline block. Denoting the delay of the i-th stage as Δt_i [86], the constraint on the interstage latency variation and the latency-balancing constraint is given by

$$\Delta \mathrm{t}{\mathrm{i}}_{\max } / \Delta \mathrm{t}{\mathrm{i}}_{\min } \le 1.5$$

(2)

If the delay of any single stage is excessively high, the overall frequency is forced to decrease, resulting in reduced throughput. An independent RNN cell is allocated to each pipeline stage to predict its kernel size and parallelism hyperparameters. The hardware-aware reward function assigns a reward R_i for stage i on the basis of its hardware utilization U_i. If U_i > 1, the timing constraints are violated, triggering a negative reward to penalize the imbalanced stage.

$${\mathrm{R}}_{\mathrm{i}}=\left\{\begin{array}{ll}{\mathrm{U}}_{\mathrm{i}}& \mathrm{if} {\mathrm{U}}_{\mathrm{i}} \le 1\\ 1-{\mathrm{U}}_{\mathrm{i}}& \mathrm{if} 1 < {\mathrm{U}}_{\mathrm{i}} \le 2\\ -1& \mathrm{if} {\mathrm{U}}_{\mathrm{i}} > 2\end{array}\right.$$

(3)

All the aforementioned constraints are integrated into a unified optimization objective function, with multi-objective optimization carried out through Pareto front analysis. C1 and C2 represent all constraints from the “algorithm design” phase (e.g., maximum cell count, pipeline balance, etc.). Fitness = Accuracy—λ₁·Latency—λ₂·Power: A comprehensive reward function is formulated, which combines the objective to be maximized and minimized by weighting coefficients (λ₁, λ₂). The values of these weights are determined through Pareto front analysis to identify the optimal balance point. Furthermore, the resource occupancy and pipeline structure on the FPGA are verified to be feasible.

Objective Function:

$$\left\{\begin{array}{l}\max \mathrm{Accuracy}(\mathrm{CPSNR}/\mathrm{SSIM})\\ \min \mathrm{Latency}\left(\sum \Delta {\mathrm{t}}_{\mathrm{i}}\right)\\ \min \mathrm{Power}\left(\sum \mathrm{Cell}\times {\mathrm{P}}_{\mathrm{cell}}\right)\\ \mathrm{s}.\mathrm{t}. {\mathrm{T}}_{\mathrm{handle}}=25 + {\mathrm{P}}_{\mathrm{total}}\times {\mathrm{R}}_{\mathrm{thermal}} \le 50^{\circ }\mathrm{C}\\ \max \left(\Delta {\mathrm{t}}_{\mathrm{i}}\right)/\min \left(\Delta {\mathrm{t}}_{\mathrm{i}} \right)\le 1.5\end{array}\right.$$

(4)

In output optimal solution of validation indicators, through the aforementioned closed-loop optimization process, a Pareto-optimal neural architecture θ* is ultimately generated. This architecture θ* simultaneously satisfies high accuracy, low latency, and low power consumption.

The strength of this “multi-objective sequential model” lies in its establishment of a direct and quantifiable bridge from clinical needs to hardware implementation. It is not a post hoc verification process but rather a proactive guidance framework that ensures that every network architecture explored by NAS is, in principle, compatible with final deployment requirements, significantly improving search efficiency and the deployability of results.

3.2. Unified Co-Exploration Framework Overview

Building upon the problem formulation in Section 3.1, which defines the joint optimization objectives of maximizing accuracy under throughput constraints ≥ 30 FPS, while minimizing power consumption and thermal risk, we propose a hardware/software co-exploration framework to holistically address these challenges. This framework transcends traditional hardware-aware NAS by simultaneously exploring two coupled spaces:

Neural Architecture Search Space: Generates child networks (e.g., depthwise separable convolutions) optimized for medical imaging tasks.

Hardware Design Space: Explores FPGA pipeline configurations (e.g., stage partitioning, resource allocation) tailored to each architecture through a two-level exploration strategy:

Fast Exploration (FE): Prunes architectures exceeding thermal cell limits or violating latency balancing without training, reducing search space.

Slow Exploration (SE): Trains FE-filtered networks, updating a reinforcement learning controller to maximize a multi-objective reward.

This integrated approach pushes the Pareto frontier in accuracy–efficiency tradeoffs, achieving higher throughput and better energy efficiency versus hardware-aware NAS. The subsequent sections detail the framework components, beginning with the hardware-aware NAS design.

As shown in Figure 4, its core value is to describe a closed-loop, system-level design approach, where the process is constraint-driven, meaning all design activities stem from strict clinical and safety constraints on the far left, ensuring the usability of the final product. Multidisciplinary collaboration means that it is no longer simply about software or hardware design, but rather about combining thermodynamics, electronic engineering, computer architecture, and artificial intelligence for collaborative design and analysis. Finally, quantitative trade-offs are made through multi-objective optimization and Pareto front, transforming subjective “design choices” into objective “decisions” based on data, in order to scientifically find the best possible solution under given constraints. This is precisely the core of our research work: how to synergistically optimize clinical requirements, hardware limitations, and algorithm performance to ultimately obtain a truly safe and efficient medical device that can be used in the operating room.

3.3. Hardware-Aware NAS Framework

An innovative controller structure is proposed, where independent RNN units are allocated to each pipeline stage, enabling independent verification of interstage hardware constraints, such as cell count and latency. A dynamic balancing penalty mechanism calculates the ratio max(Δt_i)/min(Δt_j) in real time during the front-end (FE) stage; architectures exceeding the limit trigger a negative reward and are eliminated. The medical thermal constraint is integrated, with the power–temperature model serving as a hard elimination criterion within the NAS process.

This workflow, through hardware-aware real-time verification and multi-objective reinforcement learning coordination, addresses the resource mismatch problem of traditional NAS when deployed to medical edge devices.

3.3.1. FPGA-Centric Search Space Initialization with Medical Constraints

(1)

Hardware Constraint Definition and FPGA-Friendly Operator Library

Cell Resource Upper Bound: The thermal constraint of the surgical handpiece and power consumption is mapped to an upper limit of total FPGA resources cells. Power consumption ≤ 3 W is derived from medical clinical constraints. Pipeline Balancing Constraint: The interstage latency disparity ratio max(Δt_i)/min(Δt_j) ≤ ρ_th. Pipeline Block Definition: The basic unit B_i = (L_i, H_i), where

Li is the operator type (e.g., depthwise separable convolution, ReLU).

Hi represents hardware attributes (cell cost, Δt_i).

Constraint means that block’s cell quota must satisfy Cell B_i ≤ N_cell-max/M.

FPGA-friendly operators include convolution layers and depthwise separable convolution. Activation functions include ReLU variants. In connection rules, cross-layer connection depth ≤ 2 which reduces BRAM usage. In operators, large kernels and high-resource operations are prohibited. FPGA-friendly operators refer to hardware-efficient computational primitives optimized for FPGA deployment, characterized by low resource footprint, pipeline-compatible dataflow with no irregular memory access, and minimal control overhead in deterministic execution latency. In our framework, depthwise separable convolution replaces standard convolution with depthwise and pointwise layers, reducing DSP usage by four. In piecewise linear HDR compression, complex tone mapping is substituted with segmented linear functions. Binary activation replaces multipliers with XNOR gates, achieving 10 energy efficiency over GPU.

(2)

Load-balanced Pipeline Stage Partitioning

Our objective is to identify a network architecture that meets the accuracy requirements within given hardware resource constraints. Partitioning the hardware design space into pipeline stages necessitates considering two key issues: (1) the number of stages that can be partitioned under finite resources and the number of pipeline levels; (2) the relationship between the latency of each stage and the total latency.

The core trade-off in the number of pipeline stages is that increasing the number of stages can improve throughput, but each stage’s logical function becomes simpler, potentially hindering its ability to achieve the desired computational effect. The core trade-off in the number of pipeline levels is that increasing levels increase throughput but consume more registers and logic resources. The complexity of each stage should be balanced to equalize the computational load across all levels, preventing any single stage from becoming a timing bottleneck. If one stage’s latency is significantly higher than that of the other stages, the overall frequency is forced to decrease, leading to reduced throughput.

Latency is categorized into first latency and throughput latency.

First, latency is the time required for data to propagate from input to output = Σ (per-stage logic delay + Register T_co). For an N-level pipeline, total first latency = N × (Tl_ogic + T_co). Throughput latency is the interval between producing two consecutive results = max(T_logic) + T_co. Increasing the number of levels (N) causes the first latency to grow linearly but improves throughput (due to increased clock frequency). Load Balancing and Critical Path Optimization: By balancing the latency of each stage, max(T_logic) is minimized, thereby maximizing the operating frequency.

In the relationship between resource constraints and the number of stages, the maximum number of pipeline levels (kmax) is constrained by

$${\mathrm{k}}_{\mathrm{m}\mathrm{a}\mathrm{x}}=\min \left({\displaystyle \frac{{\mathrm{R}}_{\mathrm{F}\mathrm{F}}}{\mathsf{\alpha }}},{\displaystyle \frac{{\mathrm{R}}_{\mathrm{L}\mathrm{U}\mathrm{T}}}{\mathsf{\beta }}},{\displaystyle \frac{{\mathrm{R}}_{\mathrm{D}\mathrm{S}\mathrm{P}}}{\mathsf{\gamma }}}\right)$$

(5)

where, RFF, RLUT, RDSP: Total available register, LUT, and DSP resources on the FPGA. α, β, γ: Register, LUT, and DSP consumption per pipeline level.

Inserting an additional pipeline level requires extra registers to store intermediate results, causing resource consumption to scale linearly with the number of levels (k). If k exceeds kmax, place-and-route will fail because of resource exhaustion.

In the relationship between computational complexity and single-stage latency, the timing convergence is governed as follows:

$$\mathrm{Tstage} = \mathrm{Tlogic} + \mathrm{Tco} + \mathrm{Trouting} \le \mathrm{Tclk}-\mathrm{Tsetup}$$

(6)

T_stage is the total delay of a single pipeline stage (combinational logic delay); T_clk is the target clock period; and Tsetup is the register setup time margin. The complexity of the combinational logic determines T_logic. If T_stage > T_clk, the logic must be partitioned further, or the algorithm must be optimized.

In relationship between single-stage latency and total latency, the first latency is defined as follows:

$$L_{\mathrm{first}} = k\cdot (T_{\mathrm{stage}}+T_{\mathrm{skew}})$$

(7)

T_skew is the clock skew caused by routing differences. It is positively and linearly correlated with the number of pipeline levels (k). This approach is critical for real-time systems, requiring fast response.

The throughput latency is defined as follows:

$$T_{\mathit{throughput}} = \mathit{max}(T_{\mathit{stage},1},T_{\mathit{stage},2},\dots ,T_{\mathit{stage},k}) + T_{\mathit{co}}$$

(8)

Determined by the delay of the slowest pipeline stage, it dictates the system’s maximum throughput (1/Tthroughput). Pipeline partitioning improves the frequency by splitting Tcomb, but this gain is limited by the register overhead Treg. The factors affecting design are shown as

Table 3. Factors affecting design.

Design Objective	Optimization Formula	Constraint Condition
Maximize Throughput	min(Tthroughput) = min(max(Tstage))	k ≤ kmax(Resource upper bound)
Minimize First Latency	min(Lfirst) = min(k·Tstage)	Tstage ≥ Tclk, min(Process limit)
Control Resource Consumption	min(αk + βk + γk)	Tstage ≤ Tclk

.

Therefore, our final design balancing rule is as follows:

$${\mathrm{k}}_{\mathrm{opt} }=\arg \underset{\mathrm{k}}{\max }\left({\displaystyle \frac{1}{{\mathrm{T}}_{\mathrm{throughput} }}}\right) \mathrm{subject} \mathrm{to} \mathrm{k} \le {\mathrm{k}}_{\mathrm{msc}},{\mathrm{T}}_{\mathrm{stage},\mathrm{i}}\approx \mathrm{constant}$$

(9)

Optimal level (k) selection is based on the throughput being maximized by balancing the delay of all stages (T_stage,i approaches a constant value) within the resource upper bound. If k is too large, causing the routing delay Trouting to become excessively high due to routing congestion in deep pipelines, the achievable frequency may be lower than the theoretical prediction.

3.3.2. Multi-Task Joint Optimization for Endoscopic Imaging in NAS Framework

In CFA Demosaicing, endoscopes utilize a Bayer pattern CFA to capture color components [86], where each pixel only acquires a single channel (R/G/B). Full-color image reconstruction requires interpolation. Under low-light conditions, this process is prone to zipper effects and false color artifacts [87], which can blur vascular textures and compromise diagnostic accuracy. To address this, an edge-sensitive convolution module, as seen in Figure 5, can be customized within the search space, with constrained operator types including the following:

search_space = {

“edge_conv”: {“kernel”: [3, 5], “dilation”: [1, 2]},  # Preserves edge details

“bilateral_filter”: {“sigma”: 0.8},         # Suppresses false color

”skip_connect“: {}                   # Avoids oversmoothing

}

Figure 5. Functional module division of the imaging system and the position of FPGA in the system.

Figure 5. Functional module division of the imaging system and the position of FPGA in the system.

For dynamic range compression (HDR compression), surgical scenarios often involve uneven illumination, with dark areas and highly reflective zones simultaneously present. Compression of the high dynamic range is required to prevent overexposure or underexposure [88]. A hardware-friendly operator using piecewise linear mapping is adopted instead of complex tone mapping, significantly reducing computational latency. The piecewise function is defined as follows:

$${\mathrm{I}}_{\mathrm{out}}=\left\{\begin{array}{ll}0.5\cdot {\mathrm{I}}_{\mathrm{in}}& {\mathrm{I}}_{\mathrm{in}} \le 0.3\\ 0.3 + 0.7\cdot ({\mathrm{I}}_{\mathrm{in}}- 0.3)& 0.3 <{ \mathrm{I}}_{\mathrm{in}} \le 0.7\\ 0.8 + 0.2\cdot ({\mathrm{I}}_{\mathrm{in}}- 0.7)& {\mathrm{I}}_{\mathrm{in}} > 0.7\end{array}\right.$$

(10)

When embed into the NAS pipeline block, it consumes only 0.5 cells.

The pipeline design consists of a cascaded processing engine architecture with the following stages seen in Figure 6. The CFA preprocessing stage first receives data from the input buffer, then performs Bayer pattern separation, and finally executes G-channel priority interpolation. Subsequently, the HDR compression stage calculates the local contrast, applies piecewise linear mapping, and completes the process with output fusion.

Hardware-mapped resource consumption and latency are shown in

Table 4. Hardware-mapped resource consumption and latency.

Task	FPGA Resources	Latency (ms)
CFA Demosaicing	40 Cells	3.2
HDR Compression	20 Cells	1.8

below. The total latency is 5.0 ms, meeting the real-time requirement of 30 FPSs [1].

Thus, the multi-objective reward function is designed as follows:

$$\mathrm{Reward} = \underset{\mathrm{DemosaicingQuality}}{\underbrace{0.4\cdot \mathrm{CPSNR}}} + \underset{\mathrm{HDRFidelity}}{\underbrace{0.3\cdot \mathrm{SSIM}}} + \underset{\mathrm{HardwareUtilization}}{\underbrace{0.3\cdot {\mathrm{U}}_{\mathrm{i}}}}$$

(11)

$$\mathrm{SSIM}={\displaystyle \frac{\left(2{\mathsf{\mu }}_{\mathrm{x}}{\mathsf{\mu }}_{\mathrm{y}}+{\mathrm{C}}_1\right)\left(2{\mathsf{\sigma }}_{\mathrm{xy}}+{\mathrm{C}}_2\right)}{\left({\mathsf{\mu }}_{\mathrm{x}}^2+{\mathsf{\mu }}_{\mathrm{y}}^2+{\mathrm{C}}_1\right)\left({\mathsf{\sigma }}_{\mathrm{x}}^2+{\mathsf{\sigma }}_{\mathrm{y}}^2+{\mathrm{C}}_2\right)}}$$

(12)

Reference [55] validated its robustness under low-light conditions. It is verified in medical scenarios via a low-light porcine endoscopic dataset [69], which includes scenes with a dynamic range >100 dB. Dark areas are at 10 lux and reflective areas are at 1000 lux. The evaluation metrics include the color fidelity CPSNR, structural preservation SSIM, and false color pixel ratio. The experimental results are shown as

Table 5. Multi-task joint optimization results.

Method	CPSNR (dB)	SSIM	False Color Pixel Ratio	Power (W)
Bilinear Interpolation	34.2	0.82	12.3%	1.5
GPU-HDR (Tone Mapping)	37.1	0.86	8.7%	4.2
Our NAS Scheme	38.2	0.89	4.1%	2.8

below.

Under the 90-cell constraint, false color artifacts are reduced by 67%, whereas the temperature remains ≤48 °C [69]. The CPSNR increased by 1.1 dB. Our method achieved 38.2 dB, outperforming GPU-HDR by 1.1 dB. This is primarily due to the edge-sensitive convolution module, as seen in Figure 4, explored by NAS, which effectively suppresses zipper effects under low-light conditions. The SSIM improved to 0.89, which is 0.07 higher than that of traditional methods, demonstrating that piecewise linear mapping [53] in dynamic range compression successfully preserves tissue structures as intestinal folds, reducing errors by 52%. Power consumption of 2.8 W: This is the only method that satisfies the surgical handpiece thermal constraint, owing to the precise resource allocation enabled by the cell abstraction model. The false color pixel ratio is 4.1%; this value is 53% lower than that of the GPU-based scheme, as the NAS custom operator library excludes high-error operators as complex tone mapping, meeting clinical diagnostic requirements.

3.3.3. Hardware/Software Co-Exploration with Dynamic Medical Constraints

The framework filters candidate architectures that satisfy hardware constraints without training, eliminating designs with low resource utilization. It adopts a two-level co-exploration approach which is the fast exploration and the slow search proposed in Reference [85]. The main contribution of this reference is a framework for co-exploring the architectural search space and hardware design space, as shown in Figure 1. Specifically, the framework identifies the optimal hardware to be tailored for candidate architectures during the search process. This approach yields a set of high-quality architecture–hardware design pairs along the Pareto frontier of accuracy and hardware efficiency.

However, the description in the reference is inaccurate: (1) The most effective networks typically require substantial hardware resources, which may be impractical or prohibitively expensive in reality. (2) The search order is reversed: The hardware constraints (application context) should be determined first, and then the best network deployable under those constraints should be sought. (3) While the reference describes multi-FPGA implementation as mainstream, the number of FPGAs should not be the focus. Multi-FPGA solutions should be considered only when the resource demands exceed the capacity of a single FPGA. The actual number depends on real-world FPGA resources—a single FPGA may suffice for smaller requirements. The overall framework of NAS is shown in Figure 7.

(1)

Fast exploration (FE): Hardware efficiency first.

Independent RNN units generate candidate operations for each pipeline stage, including stage 1 of depthwise separable Conv + ReLU, along with hyperparameters such as kernel size, channel count, and parallelism. In resource validation, the cell consumption of the current architecture is computed. Latency balance is validated:

$$\mathrm{Ui} > 1 \Rightarrow \mathrm{Architecture} \mathrm{Discarded}$$

(13)

where U_i denotes hardware utilization, exceeding 1 triggers elimination. Architectures exceeding resource limits or exhibiting latency imbalance are immediately discarded, significantly reducing the search space.

(2)

Slow exploration (SE): Accuracy optimization.

Train architectures filtered by FE on medical datasets of low-light endoscopic images.

$$\mathrm{Reward}=\mathrm{Accuracy}-\mathsf{\beta }·\mathrm{Latency}$$

(14)

where accuracy-prioritized β = 0.7 or energy efficiency-prioritized β = 0.3. The RNN weights are updated via policy gradient reinforcement learning, which favors high-reward architectures.

Training-free hardware validation resource of utilization Ui is performed, eliminating noncompliant architectures where U_i > 1 trigger penalty and reducing the search space. It trains only FE-filtered candidate networks, eliminating redundant computations and dramatically improving search efficiency. The pipeline block combines pipeline stages and network blocks, simultaneously incorporating embeddings from both network search and hardware pipeline partitioning.

3.4. FPGA-Centric Optimization Components

3.4.1. Cell-Abstracted Resource Modeling with Thermal Constraints

Different FPGAs are composed of different fundamental units. While both Xilinx and Intel FPGAs employ LUTs as basic programmable elements, their resource organization differs: Xilinx groups LUTs into configurable logic blocks (CLBs). Intel groups LUTs into adaptive logic modules (ALMs). To enable vendor-agnostic optimization, our cell abstraction model maps all resources to normalized LUT6 equivalents. For Intel ALMs, fractional cell values are permitted during NAS, with final deployment rounding to the nearest integer. This handles architectural granularity while ensuring thermal safety. The cell serves as a normalized unit for quantifying power consumption across different platforms. In applications with stringent power constraints—such as medical imaging acquisition systems, which are often integrated into the handpiece—excessive power consumption can lead to elevated temperatures, hindering normal surgical procedures. Therefore, accurate modeling of FPGA resources is essential.

Data collection and feature definition are performed for all core resource types of the FPGA. The hardware feature vector is constructed as follows:

$${\mathrm{X}}_{\mathrm{i}} = \left[{\mathrm{LUT}}_{\mathrm{i}}, {\mathrm{FF}}_{\mathrm{i}}, {\mathrm{Bram}}_{\mathrm{i}}, {\mathrm{DSP}}_{\mathrm{i}}, {\mathrm{PLL}}_{\mathrm{i}}, {\mathrm{Serdes}}_{\mathrm{i}}, {\mathrm{f}}_{\mathrm{i}}, {\mathrm{V}}_{\mathrm{i}}\right]$$

(15)

Cell Abstraction Definition:

Let the cell equivalent coefficient of resource j be denoted as c_j. The total number of cells is then given by

$${\mathrm{N}}_{\mathrm{cell}}=\sum _{\mathrm{j}}{\mathrm{c}}_{\mathrm{j}} \times {\mathrm{Q}}_{\mathrm{j}}$$

(16)

Q_j: The utilization amount of resource j.

Power consumption regression equation:

$${\mathrm{P}}_{\mathrm{dynamic}} ={ \mathsf{\beta }}_0 + \sum _{\mathrm{j}}{\mathsf{\beta }}_{\mathrm{j}}·{\mathrm{Q}}_{\mathrm{j}} + {\mathsf{\beta }}_{\mathrm{f}}·\mathrm{f}·{\mathrm{V}}^2 + \mathsf{ϵ}$$

(17)

The coefficient β_j corresponds to the unit power consumption of resource j, and the cell coefficient c_j is proportional to β_j.

The normalized cell equivalent coefficients are defined with LUT as the baseline (i.e., cLUT = 1). Thus, the relationship is given by

$${\mathrm{c}}_{\mathrm{j}}={\displaystyle \frac{{\mathsf{\beta }}_{\mathrm{j}}}{{\mathsf{\beta }}_{\mathrm{LUT}}}}$$

(18)

where the design matrix X, response vector Y, and coefficient vector β are defined as follows:

$$\mathrm{X} = \left[\begin{array}{ccccc}1& {\mathrm{LUT}}_1& {\mathrm{BRAM}}_1& \dots & (\mathrm{f}\cdot {\mathrm{V}}^2{)}_1\\ 1& {\mathrm{LUT}}_2& {\mathrm{BRAM}}_2& \dots & (\mathrm{f}\cdot {\mathrm{V}}^2{)}_2\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& {\mathrm{LUT}}_{\mathrm{n}}& {\mathrm{BRAM}}_{\mathrm{n}}& \dots & (\mathrm{f}\cdot {\mathrm{V}}^2{)}_{\mathrm{n}}\end{array}\right]\mathrm{Y} = \left[\begin{array}{c}{\mathrm{P}}_{{\mathrm{dynamic}}_1}\\ {\mathrm{P}}_{{\mathrm{dynamic}}_2}\\ ⋮\\ {\mathrm{P}}_{{\mathrm{dynamic}}_{\mathrm{n}}}\end{array}\right]\mathsf{\beta } = \left[\begin{array}{c}{\mathsf{\beta }}_0\\ {\mathsf{\beta }}_1\\ ⋮\\ {\mathsf{\beta }}_6\end{array}\right]$$

(19)

where n represents the number of experimental data points, with each row corresponding to a set of measured results for a specific resource combination. The coefficients β_j are solved via the least squares method, and the cell coefficients are subsequently calculated. Comparison of cross-vendor cell calibration coefficients is shown as

Table 6. Comparison of cross-vendor cell calibration coefficients.

Resource Type	Xilinx (Cell)	Intel (Cell)	Lattice (Cell)	Calibration Basis
1 LUT	1.0	1.0	1.0	Base Unit
1 FF	0.5	0.6	0.4	Power Ratio
1 BRAM (18 Kb)	15	18	12	βj/βLUT
1 DSP	10	12	8	MACEnergy Efficiency
1 PLL	8	10	6	Clock Mgmt Complexity
1 SERDES	20	25	15	I/O Speed Weight

Coefficient = FPGA total available resources/actual resource usage × 100%.

.

$${\mathrm{c}}_{\mathrm{Bram}}={\displaystyle \frac{{\mathsf{\beta }}_{\mathrm{Bram}}}{{\mathsf{\beta }}_{\mathrm{LUT}}}}$$

(20)

Similarly, we use thermal couples to collect the chip surface temperature T_i and then apply the temperature regression equation and the constraint T_chip ≤ 50°C [87] to back-calculate the maximum allowable power consumption P_max, which in turn determines the upper limit of the number of cells, N_cell-max.

$${\mathrm{T}}_{\mathrm{chip}}={\mathsf{\gamma }}_0+{\mathsf{\gamma }}_1·{\mathrm{P}}_{\mathrm{total}}+{\mathsf{\gamma }}_2·{\mathrm{A}}_{\mathrm{DeatDisspation}}^{-1}+\mathsf{ϵ}$$

(21)

Therefore, the cell coefficient is essentially a normalized value of the resource power ratio β_j/β_LUT, which is objectively calibrated through regression analysis. In the medical constraint closed-loop system, the temperature model T = f(P)→power consumption upper limit P_max→cell count upper limit N_cell-max. This enables cross-platform adaptation, where resources from different vendors are uniformly scaled via the cell metric, ensuring the generalizability of hardware-aware NAS. We seamlessly integrate regression analysis, thermodynamic modeling and clinical constraints to establish a quantifiable hardware abstraction standard for medical edge devices.

3.4.2. Latency Balancing via Projected Gradient Descent

If the delay of one stage is significantly greater than that of the other stages, the overall frequency is forced to decrease, resulting in reduced throughput and a limited operating frequency.

Fast stages remain idle while waiting for slow stages, leading to underutilization of hardware resources in FPGAs and low resource efficiency. Deep pipelines exacerbate interstage routing delays, further degrading timing performance and causing routing congestion.

In neural architecture search, pipeline latency balancing optimization refers to constraining the interstage latency differences to prevent any single stage from becoming a timing bottleneck, thereby improving the overall throughput and hardware efficiency.

Balancing optimization can be achieved by embedding hardware constraints into the NAS space to limit the ratio max(Δti)/min(Δtj), ensuring an upper bound on latency variation. Specifying efficient resource allocation rules, the number of cells per stage is allocated proportionally to the computational load. For example, convolutional layers are of high computational load and allocated more DSPs, whereas fully connected layers are prioritized for BRAM usage.

Reference [88] adopted a streaming architecture design with ping-pong buffers and a double buffering mechanism to enable parallel computation and data transfer. While the convolutional layer writes to Buffer A, the fully connected layer reads from Buffer B. Buffers are switched in the next cycle to hide data transfer latency. The stages are decoupled: the convolutional and fully connected layers are separated into independent computing engines to avoid resource conflicts.

Therefore, our optimization objective is to minimize the maximum interstage latency difference.

$$\underset{\Delta \mathrm{t}}{\min }\left(\underset{\mathrm{i}}{\max }\Delta {\mathrm{t}}_{\mathrm{i}}-\underset{\mathrm{j}}{\min }\Delta {\mathrm{t}}_{\mathrm{j}}\right)$$

(22)

Subject to the following constraints, the upper bound on the interstage latency variation is as follows:

$${\displaystyle \frac{\max (\Delta {\mathrm{t}}_{\mathrm{i}})}{\min (\Delta {\mathrm{t}}_{\mathrm{j}})}} \le 1.5⟺\max (\Delta {\mathrm{t}}_{\mathrm{i}}) \le 1.5\min (\Delta {\mathrm{t}}_{\mathrm{j}})$$

(23)

The total latency is fixed, which is a real-time constraint:

$$\sum _{\mathrm{i} = 1}^{\mathrm{M}}\Delta {\mathrm{t}}_{\mathrm{i}} = {\mathrm{T}}_{\mathrm{total}}$$

(24)

The lower bound on single-stage latency is the minimum hardware clock period:

$$\Delta {\mathrm{t}}_{\mathrm{i}} \ge {\mathrm{t}}_{\mathrm{clk}-\min }$$

(25)

Therefore, the feasible region is defined as follows:

$$\mathrm{C} = \left\{\Delta \mathrm{t}\in {\mathrm{R}}^{\mathrm{M}} \left| \begin{array}{ll}& \Delta {\mathrm{t}}_{\mathrm{i}} \ge {\mathrm{t}}_{\mathrm{clk}-\min }\\ & \sum \Delta {\mathrm{t}}_{\mathrm{i}} = {\mathrm{T}}_{\mathrm{total}}\\ & \underset{\mathrm{i}}{\max }\Delta {\mathrm{t}}_{\mathrm{i}} \le 1.5\underset{\mathrm{j}}{\min }\Delta {\mathrm{t}}_{\mathrm{j}}\end{array}\right.\right\}$$

(26)

Since the feasible region C is a convex set, as the latency-balancing constraints define a convex set, the projected gradient method ensures that the iterative points always satisfy the constraints. The key lies in the construction of the projection operator and the design of the gradient direction [85].

Steps of the projected gradient descent Algorithm 1 is as follows:

Algorithm 1 Projected Gradient Descent

def projected_gradient_descent (Δt_init, η = 0.01, max_iter = 100):

Δt = Δt_init  # Initial delay vector

for k in range(max_iter):

# 1. Compute gradient: Objective function f(Δt) = max(Δt) − min(Δt)

grad = compute_gradient(Δt)  # Gradient calculation (see below).

# 2. Gradient descent update: Δt_new = Δt − η × grad

Δt_new = Δt − η × grad.

# 3. Projection onto feasible set: Δt = Proj_(Δt_new)

Δt = project_to_feasible_set(Δt_new).

# 4. Convergence check: If ||grad|| < ε, break loop

return Δt.

Key components are explained as follows:

• Δt_init: Initial interstage latency vector [Δt₁, Δt₂..., Δt_m];
• η: Learning rate controlling step size;
• Compute_gradient(): Computes gradient of max(Δt) − min(Δt);
• Project_to_feasible_set(): Projects update latencies onto convex constraint set.

Forces max(Δt_i)/min(Δt_j) ≤ threshold, maintaining a fixed total latency ∑Δt_i = T_total, which preserves Δt_i ≥ t_min. In convergence, it stops when the gradient norm falls below tolerance ε. In gradient calculation, the subgradient of the objective function f(Δt) = max_iΔt_i − min_jΔt_j is given by

$$\nabla \mathrm{f}(\Delta \mathrm{t})=\left\{\begin{array}{ll}{\mathrm{e}}_{{\mathrm{i}}^{\ast }}-{\mathrm{e}}_{{\mathrm{j}}^{\ast }},& \mathrm{if} \Delta {\mathrm{t}}_{{\mathrm{i}}^{\ast }} > \Delta {\mathrm{t}}_{{\mathrm{j}}^{\ast }}\\ 0,& \mathrm{otherwise}\end{array}\right.$$

(27)

where

• i* = argmaxiΔti (index of the stage with maximum delay);
• j* = argminjΔtj (index of the stage with minimum delay);
• ek denotes the standard basis vector (1 at position k, 0 elsewhere).

With projection operator ProjC implementation, the projection problem is solved:

$${\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{j}}_{\mathcal{C}}\left(\mathbf{y}\right)=\mathit{arg}\underset{\mathbf{x}\in \mathcal{C}}{\min }\Vert \mathbf{x}-\mathbf{y}{\Vert }_2$$

(28)

The projection is solved through the equality-constraint projection ${\mathbf{x}}^{\prime }=\mathbf{y}+{\displaystyle \frac{T_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}-\sum y_i}{M}}\mathbf{1}$. Shift y to satisfy ∑xi = Ttotal.

If max(xi′) > min(xj′), it is expressed as follows:

$$\left\{\begin{array}{l}{x}_i^{\mathrm{new}} = \max \left(t_{\mathrm{clk}-\min },\min \left(x_i^{\prime },1.5\mu \right)\right)\\ \mu =\mathrm{median}\left(x_i^{\prime }\right)(\mathrm{median} \mathrm{reference})\end{array}\right.$$

(29)

Finally, rescale to ∑x_inew = T_total.

The solution for this projection step is theoretically supported by the convergence guarantees of convex optimization.

3.4.3. Real-Time Constrained NAS Space

Controller-level reorganization and independent RNN allocation: an independent RNN controller is allocated to each pipeline stage, as seen in Figure 4, to predict stage-specific hyperparameters.

Reward Function:

$$R_i = \left\{\begin{array}{ll}{U}_i& \mathit{if} U_i \le 1\\ 1-U_i& \mathit{if} 1 < U_i \le 2\\ -1& \mathit{if} U_i > 2\end{array}\right.$$

(30)

Ui represents hardware utilization. Exceeding limits triggers a negative reward to penalize imbalanced stages.

Multi-objective joint optimization objective function is expressed as follows:

$$\mathrm{Fitness} = \mathrm{Accuracy}-{\mathsf{\lambda }}_1\cdot \mathrm{Latency}-{\mathsf{\lambda }}_2\cdot \max (0,\Delta {\mathrm{t}}_{\mathrm{ratio}}-1.5)$$

(31)

where Δt_ratio = max(Δt_i) / min(Δt_i) denotes the latency disparity ratio.

In terms of experimental validation and performance improvement, constrained optimization significantly improves the frequency and throughput while achieving more balanced resource usage. Latency balancing constraint improves the throughput by 54.1%, outperforming traditional schemes, as seen in

Table 7. Latency Balancing Effect.

Metric	Unconstrained Scheme	Constrained Scheme (Δtratio ≤ 1.5)	Improvement
Max Frequency	125 MHz	192 MHz	+53.6%
Throughput (FPS)	19.6	30.2	+54.1%
Resource Utilization (LUT)	81%	76%	−6.2%

.

Hard constraint for latency balancing is shown in

Table 8. Comparison with SOTA methods.

Method	Δtratio	Throughput (FPS)
Hardware-Aware NAS (Fixed Pipeline)	2.1	19.6
Co-Exploration (Ours)	1.3	30.2

. This introduces a hard constraint, with max interstage latency ratio ≤1.5, which is in the NAS space, overcoming the limitations of traditional methods that optimize only single-stage latency. Convergence guarantees that when the loss function L(θ) satisfies Lipschitz continuity and the search space Θ is compact, exhaustive search converges to the global optimal solution. Medical scenario adaptation, combined with real-time constraints, ensures the clinical feasibility of endoscopic surgery. Problem resolution addresses frequency bottlenecks and routing congestion caused by deep pipelines, providing a theoretical foundation for low power medical imaging. This framework bridges theoretical optimization with clinical deployment, enabling efficient and scalable NAS for medical edge devices.

3.4.4. Experimental Validation in Medical Scenarios

Experimental setup and platform are as follows: Xilinx ZCU102 used a medical endoscopic imaging task, with a focus on initial latency, and Δt_init = [3–5,8]ns (Max/Min Ratio = 2.67); constraints included (1) max(Δt_i)/min(Δt_j) ≤ 1.5 L; 2) ∑Δt = 20 ns, which corresponded to 50 MHz system frequency, and the optimization results are presented in

Table 9. Optimization results.

Iteration Step	Δt1	Δt2	Δt3	Δt4	Disparity Ratio (max/min)
0	8.0	3.0	5.0	4.0	2.67
10	6.2	4.1	5.0	4.7	1.51
20	5.8	4.3	5.0	4.9	1.35

.

Performance improved and frequency increased from 125 MHz to 172 MHz, so a 37.6% improvement is achieved. Throughput increased from 15.6 FPS to 22.5 FPS, meeting the real-time requirement of ≥30 FPS. Theoretical innovation and engineering contributions included the following. 1. Manifold Constraint for Latency Balancing: The proposed manifold constraint max(Δt_i)/min(Δt_j) ≤ 1.5 is transformed into a convex feasible region C, providing a theoretical foundation for projected gradient descent (PGD). 2. Guaranteed Medical Real-Time Performance: Combined with the total latency constraint ∑Δt_i = 33 ms, endoscopy systems achieved ≥30 FPS. 3. Hardware-Friendly Implementation: The projection operator can be implemented in hardware via comparator trees and scaling circuits on FPGAs, supporting dynamic runtime adjustment.

3.5. FPGA-Optimized Multi-Task Implementation Summary

By jointly exploring the neural architecture space and the hardware design space, this work breaks through the fixed hardware constraints of traditional hardware-aware NAS, pushing the Pareto frontier upwards and to the right.

The key innovations include the following:

• Cell-based Resource Unification: FPGA heterogeneous resources are normalized into unified cell units, enabling cross-platform quantification of resource consumption and mathematical modeling of hardware constraints.
• Pipeline Balancing Constraints: Interstage latency variation is constrained to prevent frequency degradation caused by deep pipelines, significantly improving throughput. The average hardware utilization reached 96.15% under these constraints.
• Medical Scenario Adaptation: Under low power constraints, the architecture achieves low latency with limited cells, meeting real-time requirements for endoscopy.
• Power–Temperature Model: A unified model that integrates the surgical handpiece’s heat dissipation area, power consumption, and temperature to ensure clinical safety.

In short, co-exploration breaks down the barriers between hardware and software design, achieving a breakthrough in the “impossible triangle” of accuracy, real-time performance, and power consumption. This provides a lightweight, engineer-ready solution for medical edge devices such as endoscopes.

The neural architecture search framework generates optimized parameter sets that serve as universal computational kernels, enabling efficient deployment across diverse medical imaging algorithms:

• HDR Compression: NAS-derived parameters configure piecewise linear mapping operators that dynamically compress high dynamic ranges while preserving tissue textures in low-light endoscopy.
• CFA Demosaicing: The same parameters drive edge-sensitive convolution kernels, suppressing zipper artifacts and reducing false color ratios in Bayer pattern reconstruction.

The detailed HDL code implementations are provided in Appendix A.

4. Experimental Results

To rigorously validate the proposed hardware-aware NAS framework, which integrates cell-based resource abstraction, pipeline balancing constraints, and thermal-aware optimization, we conducted comprehensive experiments on medical endoscopic imaging tasks, such as CFA demosaicing and HDR compression. The experimental design focuses on three critical dimensions: accuracy–quality trade-off, hardware efficiency and clinical safety.

4.1. Experimental Design

To validate the effectiveness of the proposed hardware-aware NAS framework under the constraints of low power consumption, real-time performance, and thermal safety, experiments were conducted on medical imaging, specifically endoscopic CFA demosaicing and dynamic range compression. The proposed method was evaluated via a low-light porcine intestinal dataset containing 1000 images and deployed on three test platforms: the Xilinx xczu7ev-ffvc1156-2-i (FPGA), the NVIDIA Jetson AGX Xavier (GPU), and the Intel Core i7--8700K (CPU). The evaluation metrics included accuracy measures such as the CPSNR (color peak signal-to-noise ratio) and SSIM (structural similarity index), as well as hardware efficiency indicators such as latency (ms), power consumption (W), temperature (°C), resource utilization (LUT/BRAM/DSP), and frame rate (FPS). The experimental use cases are designed as shown in

Table 10. Experimental use cases.

Variable	Range of Values	Testing Purpose
Max Cell Count	70/80/90/100 Cells	Validate the accuracy of the power–temperature model.
Pipeline Stages (M)	2/4/6 stages	Analyze the impact of the number of stages on throughput and latency balance.
Parallelism	1/2/4/8 channels	Explore the trade-off between computational resources and accuracy.

.

The experimental validation utilizes the EndoVis-WCE public dataset from the MICCAI EndoVis Challenge, comprising 120 wireless capsule endoscopy (WCE) videos (totaling 8 h) capturing small intestine scenes. Key features include the following:

1.

Precise Annotations: Frame-level labels for bleeding points and vascular malformations, and tumor locations validated by three gastroenterologists (κ-coefficient > 0.85).

2.

Technical Challenges:

• HDR Compression: Scenes with extreme illumination variance (>100 dB dynamic range) due to fluid occlusion and tissue reflectivity.
• CFA Artifacts: Bayer pattern demosaicing complications under low-light conditions (SNR < 10 dB) inducing zipper effects in 23% of frames.

3.

Acquisition: Available at endovis.grand-challenge.orgunder CC-BY-NC-SA 4.0 license (Dataset ID: EV-WCE-2021).

4.2. Experimental Comparison of the Intermediate Process Before and After Optimization

A comparison of the hardware efficiency in

Table 11. Effectiveness of pipeline-balancing optimization.

Metric	Before Optimization (Fixed Pipeline)	After Optimization (Latency-Balancing Constraint)	Improvement
Interstage Latency Variation (max/min)	2.1 (8 ns vs. 3.8 ns)	1.3 (5.2 ns vs. 4.0 ns)	38.1%
Maximum Frequency (MHz)	125 (Limited by the 8 ns stage)	192	+53.6%
Throughput (FPS)	19.6	30.2	+54.1%

between the fixed pipeline scheme and the balanced constraint scheme shows that after constraining max(Δti) / min(Δtj) ≤ 1.5, the interstage latency variation was reduced from 2.1× to 1.3×. The maximum frequency increased from 125 MHz to 192 MHz, and the throughput improved by 54% from 19.6 FPS to 30.2 FPS. This optimization eliminates single-stage bottlenecks, as an 8 ns convolutional layer delay, demonstrating that the latency-balancing constraint is critical for avoiding frequency collapse in deep pipelines. By preventing any single stage from becoming a bottleneck, the frequency was increased by 53.6%, meeting the real-time requirement of 30 FPSs.

Experimental validation data for the power–temperature model shows that as the cell count increased from 70 to 100, the power consumption increased from 2.1 W to 3.2 W, as seen in

Table 12. Thermal constraint model validation.

Cell Count	Theoretical Power (W)	Measured Temperature (°C)	Temperature Prediction Error
70	2.1	42.3	+0.2 °C
80	2.5	45.1	−0.4 °C
90	2.8	47.8	+0.3 °C
100	3.2	52.6 (Exceeded)	−0.2 °C

, and the temperature increased from 42.3 °C to 52.6 °C. At 90 °C, the temperature was 47.8 °C (error +0.3 °C), which strictly satisfied the ≤50 °C threshold. This model successfully translates the surgical thermal constraint into a quantifiable cell upper limit for NAS. The power–temperature model has an error within the range of −0.5 °C to +0.5 °C, confirming that 90 cells constitute the safe upper limit.

A parallelism of four channels achieved the optimal balance under the 90-cell constraint, delivering a CPSNR of 38.0 dB and a frame rate of 30.2 FPS as

Table 13. Resource–accuracy trade-off.

Parallelism	CPSNR (dB)	LUT Utilization	Frame Rate (FPS)
1	38.5	42%	18.3
2	38.2	58%	26.7
4	38.0	76%	30.2
8	37.8	93% (Exceeded)	32.1

.

4.3. Comparison with Existing Methods

A breakdown of latency across pipeline stages reveals that demosaicing and tone mapping constitute the critical path, accounting for 38% and 32% of the total delay, as seen in

Table 14. Comparison of accuracy and hardware efficiency.

Method	CPSNR (dB)	Power (W)	Temp. (°C)	Latency (ms)
Manually Designed Model (VGG8)	38.5	4.1	55 (Exceeded)	22.4
GPU-NAS (ENAS)	38.8	18.2	65 (Exceeded)	12.3
FPGA-NAS (Fixed Pipeline)	37.9	3.0	46	51.2
Proposed Method	38.2	2.8	48	8.1

, respectively. Through streaming architecture optimization (ping-pong buffers + parallel MAC units), the total latency was reduced from 51.2 ms to 8.1 ms, and the frame rate increased from 19.6 FPS to 123.5 FPS, far exceeding the 30 FPS requirement for real-time surgery.

The cross-platform performance on the medical imaging task (endoscopic demosaicing) shows that the FPGA solution achieved 123.5 FPS at 2.8 W power and a low temperature of 47 °C, significantly outperforming the GPU and CPU solutions. The temperature model error is ≤0.5 °C, and it is the only solution that met the clinical safety requirement of a handpiece temperature ≤50 °C, highlighting the irreplaceability of FPGAs for medical edge devices.

Under strict power constraints, the proposed method achieved 0.3 dB higher accuracy than FPGA-NAS and an 84.2% reduction in latency.

With similar resource utilization as

Table 15. Comparison of resource utilization.

Method	LUT Utilization	BRAM Utilization	Throughput (FPS)
FPGA-NAS (Ref. [38])	81%	78%	19.6
HASIC (Ref. [42])	76%	72%	28.3
Proposed Method	76%	68%	30.2

, the proposed method achieved a 54.1% higher throughput than the SOTA FPGA-NAS.

On the low-light porcine endoscopic dataset, the proposed method achieved a CPSNR of 38.2 dB, as seen in

Table 16. Medical scenario adaptability.

Method	Low-Light CPSNR (dB)	Motion Blur SSIM	Thermal Safety (≤50 °C)
Traditional Interpolation (Bilinear)	34.2	0.82	true
GPU-HDR (Tone Mapping)	37.1	0.86	error (65 °C)
Proposed Method	38.2	0.89	true (48 °C)

, which is superior to that of traditional interpolation (34.2 dB) and the GPU-NAS scheme. Under motion blur scenarios, it achieved an SSIM of 0.89, which is 0.03 higher than that of the GPU, demonstrating the advantage of hardware-aware NAS in preserving edge details and meeting diagnostic-grade imaging requirements.

The proposed method delivers superior reconstruction quality in low-light and dynamic scenes while ensuring surgical safety.

5. Conclusions

This study establishes a hardware-aware NAS framework for low-power endoscopic video processing. Key innovations include (1) cell-based resource abstraction unifying BRAM/DSP/LUT under thermal constraints; (2) projected gradient-driven latency balancing capping interstage delay variance; (3) two-stage co-exploration accelerating search efficiency. Experimental validation confirms 38.2 dB CPSNR at 2.8 W power, outperforming GPU solutions by 54.1% in energy efficiency.

Future work will focus on enhancing the adaptability and intelligence of the hardware-aware co-exploration framework. One critical direction is extending the NAS method to support more complex and multimodal medical imaging tasks, such as real-time video segmentation or the fusion of white-light and fluorescence endoscopy. The automation of constraint modeling also warrants further investigation, especially the incorporation of learned predictive models in dynamically adjusting the multi-objective weights during search, which would improve generalizability across different clinical scenarios. In addition, developing lightweight and transplantable projection operators for latency and thermal constraints is essential for achieving low-overhead dynamic adaptation on edge FPGAs. Finally, building an open and cross-platform benchmarking framework would accelerate replication and comparison for medical hardware-aware NAS research.