A Novel Approach to Retinal Blood Vessel Segmentation Using Bi-LSTM-Based Networks

Abstract

The morphology of blood vessels in retinal fundus images is a key biomarker for diagnosing conditions such as glaucoma, hypertension, and diabetic retinopathy. This study introduces a deep learning-based method for automatic blood vessel segmentation, trained from scratch on 44 clinician-annotated images. The proposed architecture integrates Bidirectional Long Short-Term Memory (Bi-LSTM) layers with dropout to mitigate overfitting. A distinguishing feature of this approach is the column-wise processing, which improves feature extraction and segmentation accuracy. Additionally, a custom data augmentation technique tailored for retinal images is implemented to improve training performance. The results are presented in their raw form—without post-processing—to objectively assess the method’s effectiveness and limitations. Further refinements, including pre- and post-processing and the use of image rotations to combine multiple segmentation outputs, could significantly boost performance. Overall, this work offers a novel and effective approach to the still unresolved task of retinal vessel segmentation, contributing to more reliable automated analysis in ophthalmic diagnostics.

1. Introduction

The human retina offers a unique window into systemic health, where subtle microvascular changes—arteriolar narrowing, venular widening, increased tortuosity—signal early warning signs of hypertension, diabetes, and cardiovascular disease [1]. These biomarkers can be detected years before clinical symptoms [2] and are quantified by segmentation of the retinal vessels. However, despite this diagnostic potential, automated segmentation remains an unsolved challenge: errors propagate into critical biomarkers such as the arteriole-to-venule ratio (AVR), a known predictor of stroke risk [3], and they actively undermine clinician and patient trust in AI systems [4].

Current approaches, dominated by convolutional neural networks (CNNs) and vision transformers (ViTs), achieve impressive precision on curated benchmarks such as DRIVE and STARE [5]. In the real world, however, baseline U-Net–style models under-segment faint capillaries, often missing entire small vessels [6,7]; hallucinate vessel-like artifacts near pathological lesions [8]; and suffer catastrophic performance drops in unseen devices or demographics [9]. Mounting evidence traces these failures to a fundamental mismatch between the need for global vascular continuity [10] and the inherently local, grid-based processing of standard CNN/transformer architectures [11].

Competing explanations. One line of work claims that annotation noise and domain shift, rather than model design, are the real bottlenecks: when masks are cleaned and multidevice data are used, standard CNNs already have near-perfect $F_1$ scores [9,12]. A second view blames local receptive field of convolutions and pushes for global context solutions: self-attention transformers or explicit connectivity losses. A recent transformer model reports markedly improved vessel connectivity on DRIVE, STARE and CHASE_DB1 [13], while connectivity-aware losses also close gaps [11]. Our sequence-first Bi-LSTM offers an orthogonal test of these competing hypotheses.

1.1. The Data Efficiency Dilemma

Deep CNNs are notoriously data-hungry, with large reviews noting the need for thousands of pixel-accurate masks [14], yet the largest vessel datasets still top out at 500–800 expert-annotated images [15,16]. However, interobserver variability exceeds 20% for capillaries [9,17], introducing noise into the training. Although synthetic data and augmentation mitigate this, they risk distorting vascular morphology or hiding failures behind post-processing. Common low-level transforms—JPEG compression, aggressive rotation or scaling—can bias vessel caliber and fractal metrics [18] and may erase capillaries unless augmentation is vessel aware [19]; comprehensive reviews echo these pitfalls across medical modalities [20]. A paradigm shift is needed: methods that learn structural priors (e.g., vessel connectivity) from limited data while preserving interpretability [21].

1.2. Sequential Reasoning with Bi-LSTMs

Bidirectional Long Short-Term Memory (Bi-LSTM) networks, originally devised for sequential signals such as language and speech, offer an unconventional solution for fundus analysis. Recent work by Marti-Puig et al. demonstrated that a column-wise Bi-LSTM can extract continuous shorelines from time-lapse coastal imagery with high fidelity [22]. Motivated by that success, we treat each retinal image column as a 1-D sequence so that a Bi-LSTM can propagate evidence bidirectionally, enforcing the global vascular continuity that patch-based CNNs often overlook [11]. Earlier hybrids, most notably BCDU-Net [23], grafted Bi-LSTM blocks onto a U-Net backbone but still inherited convolutional biases. In the present work, we eliminate convolutions entirely: raw columnar sequences are processed by a pure Bi-LSTM backbone, forcing the network to infer retinal topology from sequential context—a structural prior that aligns naturally with vascular anatomy.

1.3. Transparency Through Minimalism

Our approach prioritizes simplicity and interpretability.

• Data efficiency: Processing each 1024-pixel column as an independent sequence inflates the 44 training images (drawn from the 54 publicly released RETA-IDRiD images) into 45,056 sequences, reducing annotation demands.
• Rotation ensembles: A single right-angle rotation (0°, 90°, 180°, 270°) is always applied. For data-limited experiments, we also used eleven off-axis rotations (15°–315°, see Section 2.6), which yielded 540,672 sequences. These angles preserve vessel morphology better than arbitrary elastic warps.
• Raw outputs: Publicly released probability maps expose errors (e.g., optical disc glare, hemorrhages), fostering trust and guiding clinical refinements.

1.4. Contributions and Implications

This work demonstrates the following:

• In our internal test fold of ten images (18% of the accessible data), the best Bi-LSTM attains 95.6% pixel accuracy and an MCC of 0.70 (see Tables 5 and 7). All results are reported before any morphological post-processing. These first-cut results do not yet surpass strong CNN/ViT baselines on the official 27-image RETA leader board, but they show that pure sequence-first reasoning can reach parity on fully public data, warranting hybrid architectures in future work.
• Data-efficient training through sequential processing, sidestepping the need for massive annotated datasets.
• Failure-aware transparency, critical for clinical adoption, by exposing edge cases like tortuous vessels.

Although not yet surpassing state-of-the-art CNNs, our results validate sequence-first reasoning as a complementary paradigm. Future work could integrate Bi-LSTMs with CNNs in hybrid architectures, combining local texture analysis with global continuity modeling, a promising direction for robust, interpretable diagnostic tools.

From this point forward, the article is organized as follows. Section 2 details the RETA-IDRiD dataset, the 44/10 data split, CIE–Lab color space conversion, and rotation-based data augmentation and describes the full column-wise processing pipeline, including three Bi-LSTM variants and the exact MATLAB 2024b training protocol. Section 3 presents pixel-level performance metrics for all models on the held-out RETA fold, accompanied by raw (pre-morphology) confusion matrices and color-coded overlays for true positives, false positives, and false negatives. Section 4 discusses common failure modes of the model, such as missed capillaries, lesion-induced hallucinations, and glare artifacts, relating these issues to architectural design choices and domain-shift vulnerabilities. Finally, Section 5 summarizes the key findings.

2. Materials and Methods

2.1. Datasets

Primary data. All experiments use the RETA–IDRiD vascular tree subset, released with the RETA benchmark, [12,24]. It contains 81 color fundus images (45° field of view (FOV), 4288 × 2848 px, saved as 1024 × 1024 crops in our pipeline) with dense annotations for vessels, artery/vein identity, bifurcations, and skeletons. We follow the official split: 54 images for training/validation and 27 for hold-out testing. IDRiD images are deidentified and publicly licensed for research; no additional ethical approval is required.

Of the 54 RETA-IDRiD images labeled, 27 are reserved by the organizers as a challenge test set—the masks are not publicly released and evaluation is possible only by blind submission to the leaderboard. To keep our study fully reproducible, we therefore work solely with the 54 accessible images. Ten (18%) are left as an internal test fold, stratified to match the prevalence of lesions in the entire set (two heavily diseased eyes, eight mild/normal). The remaining 44 form the training pool. This 80/20 split is standard practice for small medical datasets and avoids data leakage while preserving enough samples for model fitting and augmentation.

Why RETA? Compared with legacy sets such as DRIVE [25], STARE [26] or CHASE_DB1  [27], RETA offers (i) cleaner masks (semi-automated, adjudicated), (ii) lesion variability (exudates, hemorrhages) that stresses false positive control, and (iii) structural labels enabling future benchmarks at the artery/vein or tree level. For video-based generalization, we point the reader to the recent RVD dataset, [28], left for future work.

2.2. Pre-Processing

Each image is center-cropped to the 1024 × 1024 field supplied by RETA and converted to CIE–Lab. A simple circular mask excludes the black background. No color normalization is applied—deliberately—to expose any domain-shift brittleness.

2.3. Image Processing Strategy: Column-Based Approach

As mentioned above, the distinctive aspect of our segmentation approach lies in the processing of retinal images column by column. This involves dividing each image into column vectors and, for each of these vectors, using a neural network to determine which pixels correspond to the retinal background and which correspond to blood vessels. In our case, the images have a resolution of 1024 × 1024 pixels, meaning that we process 1024 columns per image. These columns are subsequently reassembled in the correct order to reconstruct the segmentation of the original image.

Deep learning-based networks require large amounts of data for effective training, and the annotation process is time consuming and must be carried out by domain experts. The annotated dataset available for training is therefore limited. By changing the training unit from full images to individual image columns, we effectively increase the number of training samples by a factor of 1024. This significantly expands the annotated dataset and provides sufficient data to train deep learning models.

A key step in our approach is interpreting each column as a time series and processing it using neural networks based on LSTM layers, which are particularly well suited for handling sequential data.

The image processing workflow is illustrated in Figure 1, which shows how the image is decomposed into individual columns, how an estimate is obtained for each column, and how the final vessel-background segmentation is reconstructed by combining the estimations from all columns.

2.4. Long Short-Term Memory (LSTM) Networks

LSTM networks operate based on two core state vectors. The first, the hidden state vector ${\mathbf{h}}_t$, represents both the internal state and the output of the LSTM layer at time step t. The second, the cell state vector ${\mathbf{c}}_t$, is designed to preserve longer-term dependencies across the sequence. At each time step, the network selectively updates ${\mathbf{c}}_t$ by adding or removing information through gated mechanisms.

An LSTM cell typically consists of four gates: (1) the input gate, denoted by the vector ${\mathbf{i}}_t$, which controls how much new information is incorporated into the cell state; (2) the forget gate, ${\mathbf{f}}_t$, which modulates the removal of past information from the cell state ${\mathbf{c}}_t$; (3) the cell candidate gate, ${\mathbf{g}}_t$, which provides new candidate values to be added to the state; and (4) the output gate, ${\mathbf{o}}_t$, which determines the extent of information to be passed to the hidden state. These gates operate in conjunction to maintain and update the internal memory of the LSTM.

Figure 2a illustrates the internal architecture of an LSTM cell and the gate outputs at time t, from left to right: ${\mathbf{f}}_t$, ${\mathbf{g}}_t$, ${\mathbf{i}}_t$, and ${\mathbf{o}}_t$. The figure also displays the input vector ${\mathbf{x}}_t$ and the computations that update both ${\mathbf{h}}_t$ and ${\mathbf{c}}_t$ using their respective values ${\mathbf{h}}_{t-1}$ and ${\mathbf{c}}_{t-1}$ from the previous time step ($t-1$).

The LSTM network has three types of parameters: input weights in matrix $\mathbf{W}$, the recurrent weights in $\mathbf{R}$, and biases in the vector $\mathbf{b}$. The matrix $\mathbf{W}$ combines information from the input ${\mathbf{x}}_t$, while $\mathbf{R}$ controls the contribution of the previous hidden state ${\mathbf{h}}_{t-1}$, while $\mathbf{b}$ adds a bias. Each of these parameters is partitioned into subcomponents corresponding to the four gates:

$$\mathbf{W}=\left[\begin{array}{c}{\mathbf{W}}_i\\ {\mathbf{W}}_f\\ {\mathbf{W}}_g\\ {\mathbf{W}}_o\end{array}\right],\mathbf{R}=\left[\begin{array}{c}{\mathbf{R}}_i\\ {\mathbf{R}}_f\\ {\mathbf{R}}_g\\ {\mathbf{R}}_o\end{array}\right],\mathbf{b}=\left[\begin{array}{c}{\mathbf{b}}_i\\ {\mathbf{b}}_f\\ {\mathbf{b}}_g\\ {\mathbf{b}}_o\end{array}\right]$$

(1)

Each gate computes its output based on the current input ${\mathbf{x}}_t$ and the hidden state from the previous time step ${\mathbf{h}}_{t-1}$ according to the following:

$${\mathbf{i}}_t=\sigma \left({\mathbf{W}}_i{\mathbf{x}}_t+{\mathbf{R}}_i{\mathbf{h}}_{t-1}+{\mathbf{b}}_i\right)$$

(2)

$${\mathbf{f}}_t=\sigma \left({\mathbf{W}}_f{\mathbf{x}}_t+{\mathbf{R}}_f{\mathbf{h}}_{t-1}+{\mathbf{b}}_f\right)$$

(3)

$${\mathbf{g}}_t=tanh\left({\mathbf{W}}_g{\mathbf{x}}_t+{\mathbf{R}}_g{\mathbf{h}}_{t-1}+{\mathbf{b}}_g\right)$$

(4)

$${\mathbf{o}}_t=\sigma \left({\mathbf{W}}_o{\mathbf{x}}_t+{\mathbf{R}}_o{\mathbf{h}}_{t-1}+{\mathbf{b}}_o\right)$$

(5)

Notice, for instance, the input gate vector ${\mathbf{i}}_t$ is calculated as the weighted input ${\mathbf{W}}_i{\mathbf{x}}_t$ plus the recurrent contribution ${\mathbf{R}}_i{\mathbf{h}}_{t-1}$ and the bias ${\mathbf{b}}_i$. The activation function $\sigma (·)$, a logistic sigmoid defined as $\sigma \left(x\right)={(1+e^{-x})}^{-1}$, is applied element-wise to the resulting vector. The remaining gate vectors are computed analogously with the particularity that in Equation (4), the activation function is the hyperbolic tangent, $tanh(·)$, which also is applied element-wise.

The gate outputs obtained in Equations (2)–(5) with the cell state ${\mathbf{c}}_{t-1}$ are used to update the cell and hidden vector states ${\mathbf{c}}_t$ and ${\mathbf{h}}_t$ as follows:

$${\mathbf{c}}_t={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot {\mathbf{g}}_t$$

(6)

$${\mathbf{h}}_t={\mathbf{o}}_t\odot tanh\left({\mathbf{c}}_t\right)$$

(7)

where ⊙ denotes the Hadamard (element-wise) product. Notice that the cell state ${\mathbf{c}}_t$ is updated by selectively retaining prior information (${\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}$) and incorporating new candidate content (${\mathbf{i}}_t\odot {\mathbf{g}}_t$). The hidden state ${\mathbf{h}}_t$, which serves as the LSTM output at time t, is derived from the cell state in the form $tanh\left({\mathbf{c}}_t\right)$ pondered by the output gate vector ${\mathbf{o}}_t$.

As mentioned, the LSTM network processes image columns, where each column is represented as a sequence of vectors of size 1 × 3 over 1024 time steps. Each vector ${\mathbf{x}}_t$, with t ranging from 1 to 1024 (since the input images are of size 1024 × 1024), contains the features corresponding to the pixel at position t in the image column, expressed in the CIE–Lab color space. Consequently, all input sequences have a fixed length of 1024, and each input vector ${\mathbf{x}}_t$ encodes the color information of a single pixel in that column. The LSTM network produces a binary output vector of dimensions 1024 × 1, with each entry indicating whether the corresponding input pixel is classified as vessel or background. The initial hidden and cell states, denoted by ${\mathbf{h}}_0$ and ${\mathbf{c}}_0$, define the starting conditions of the network and can be initialized as zero vectors.

In the LSTM networks, the information is processed sequentially along the column. At each step t, the network leverages the information accumulated from previous steps, capturing both short-term and long-term dependencies in the sequence. As the input sequence consists of the complete set of pixels in an image column, and there is no temporal constraint on data availability, we propose using a bidirectional LSTM (Bi-LSTM) structure, which enables the model to learn dependencies in both directions along the sequence, thus improving its capacity to model contextual relationships across the entire column. The output of the Bi-LSTM is computed by combining the outputs of the two LSTM layers from an expression of the following type:

$${\mathbf{y}}_t=\sigma \left({\mathbf{W}}_{\overrightarrow{h}y}{\overrightarrow{\mathbf{h}}}_t+{\mathbf{W}}_{\overleftarrow{h}y}{\overleftarrow{\mathbf{h}}}_t+{\mathbf{b}}_y\right),$$

(8)

where ${\overrightarrow{\mathbf{h}}}_t$ are the outputs of the LSTM cell that processes the column in the forward direction and ${\overleftarrow{\mathbf{h}}}_t$ are the outputs of the LSTM cell that processes the image column in the backward direction. The symbol $\sigma$ stands for the activation function. The matrices ${\mathbf{W}}_{\overrightarrow{h}y}$ and ${\mathbf{W}}_{\overleftarrow{h}y}$ contain the weights used to combine the outputs of the forward and backward cells in the Bi-LSTM layer and ${\mathbf{b}}_y$ the biases. Figure 2b provides a graphical representation of the information flow within a bidirectional LSTM (Bi-LSTM) layer.

2.5. Network Architectures

Figure 3 presents the three Bi-LSTM architectures explored in this study, each corresponding to a distinct experiment designed to explore the impact of depth and regularization.

• Architecture 1 (Figure 3a): A simple configuration consisting of a single Bi-LSTM layer with 40 hidden units, serving as our baseline to test whether minimalist column-wise processing can achieve satisfactory performance (14.2 K learnable parameters).
• Architecture 2 (Figure 3b): A deeper configuration with three Bi-LSTM layers (50-25-25 hidden units) interleaved with 50% dropout layers to improve robustness and prevent overfitting (62.7 K learnable parameters).
• Architecture 3 (Figure 3c): Similar to Architecture 2 but with reduced dropout (20%) between the first two layers and no dropout after the third layer, exploring the effect of relaxed regularization constraints (62.7 K learnable parameters).

For baseline comparisons, we also implement the following.

• U-Net: Standard U-Net architecture with 61 layers processing full 1024 × 1024 fundus images, totaling 31M parameters.
• DeepLab v3+: A ResNet-18 based DeepLab v3+ with 99 layers and 20.6 M learnable parameters.
• ViT-Tiny: We adopt the standard vit_tiny_patch16_224 encoder (12 transformer blocks, hidden size 192, three attention heads per block) with 5.72 M trainable parameters as originally reported by Touvron et al. [29] and subsequently summarized in the lightweight-ViT survey of Wang et al. [30].

2.6. Data Augmentation, Training and Hyperparameter Selection

To train and evaluate the models, the dataset is divided into two subsets: one for model training and one for testing. In all experiments, the first 44 images were used for training, while the remaining 10 were used for testing. This deterministic split allows for a consistent comparison across experiments and different architectures by ensuring that all models are trained and evaluated on exactly the same data. This avoids the variability that could be introduced by random partitioning, which might otherwise affect model training and test results.

2.6.1. Bi-LSTM Training

The network was trained using the ADAM optimizer with a gradient decay factor of 0.9 and a squared gradient decay factor of 0.999. The initial learning rate was set to ${10}^{-3}$ and remained constant throughout training, as no learning rate schedule was applied. Training was performed for a maximum of 8 epochs with a mini-batch size of 2048, and data was shuffled once at the beginning of training. The L2 regularization coefficient was set to ${10}^{-4}$ to mitigate overfitting, and gradient clipping was applied using the L2 norm method with a threshold of 1. The training was executed on a CPU environment.

Figure 4 depicts the training workflow of the proposed approach, which relies on column-based image processing. So, once the model is trained, it is applied to the test set. The testing procedure consists of processing the labeled images reserved for testing, as described in Figure 1, and comparing the results provided by the network with the corresponding target values by performing a quantitative evaluation and also a qualitative assessment for interpreting the results and validating the model’s performance.

2.6.2. U-Net Training

The U-Net model was trained using the ADAM optimizer. The initial learning rate was set to $1\times {10}^{-4}$, and no learning rate scheduling was applied. The optimizer’s hyperparameters included a gradient decay factor of 0.9 and a squared gradient decay factor of 0.999, with a small epsilon value ($1\times {10}^{-8}$) to ensure numerical stability. L2 regularization was applied with a weight of $1\times {10}^{-4}$ to reduce overfitting. Training was performed over a maximum of 50 epochs with a mini-batch size of 4. The training data was shuffled at the beginning of each epoch. The training was executed on a CPU environment.

2.6.3. DeepLab v3+ Training

The network was trained using the Stochastic Gradient Descent with Momentum (SGDM) optimizer. The training configuration included a momentum of 0.9 and an initial learning rate of 0.001, with a total of 60 epochs. Mini-batches of size 8 were used, and the data were shuffled at every epoch. L2 regularization was set to ${10}^{-4}$ to prevent overfitting. The training was executed on a CPU environment.

2.6.4. Vision Transformer Training

All hyperparameters reflect recurrent experiments: class-weighted cross-entropy, ADAMW ($\mathrm{lr}=1\times {10}^{-3}$, weight decay $1\times {10}^{-4}$) and the deterministic 44/10 RETA–IDRiD split. A single epoch sweeps ≈0.5 M patches—∼12× more gradient steps than one full-image epoch of the Bi-LSTM. During inference, softmax probabilities are stitched back to a single ${1024}^2$ mask and thresholded at 0.5, exactly as for the Bi-LSTM pipeline.

So, each image in the training group and each corresponding target is decomposed into columns, and all the vectors obtained from the set of training and all the corresponding target vectors are aggregated and used to train the model. This operation yields 44 × 1024 = 45,056 labelled instances.

As a data augmentation strategy, given the circular shape of the retinas centered within square images, the most natural approach is to apply image rotations. By simply rotating the images by 90°, 180°, and 270°, the number of labeled instances can be increased by a factor of four without the need for additional image processing. This augmentation helps reduce overfitting by exposing the model to different orientations of the same anatomical structures, which improves its ability to generalize to unseen data. If further (non-right-angle) rotations are introduced, minimal additional processing of both the images and their corresponding target masks will be required to preserve alignment. However, in all the experiments, the data augmentation strategy consisted of applying the same set of rotations, specifically at angles ${15}^{\circ }$, ${45}^{\circ }$, ${60}^{\circ }$, ${90}^{\circ }$, ${110}^{\circ }$, ${135}^{\circ }$, ${150}^{\circ }$, ${180}^{\circ }$, ${225}^{\circ }$, ${270}^{\circ }$, and ${315}^{\circ }$. As a result, a total of $\mathrm{540,672}$ labeled instances were generated.

2.7. Performance Evaluation

In the present case, we compare the pixel classification results of each reconstructed image after being processed column by column with its corresponding target. To evaluate binary classification performance, the confusion matrix, a 2 × 2 contingency table, is used. In this matrix, pixels that are correctly classified as positive are referred to as true positives (TPs), while negative pixels incorrectly classified as positive are called false positives (FPs). Similarly, pixels that are correctly classified as negative are termed true negatives (TNs), and positive pixels misclassified as negative are known as false negatives (FNs). In addition to evaluating the results using TPs, TNs, FPs, and FNs, we use these values to compute the following performance metrics.

$$\begin{array}{cc}\hfill Accuracy& ={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill Recall(Sensitivity,TruePositiveRate)& ={\displaystyle \frac{TP}{TP+FN}}\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill Specificity\left(TrueNegativeRate\right)& ={\displaystyle \frac{TN}{TN+FP}}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill Precision\left(PositivePredictiveValue\right)& ={\displaystyle \frac{TP}{TP+FP}}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill F_1\text{-}Score& ={\displaystyle \frac{2·TP}{2·TP+FP+FN}}={\displaystyle \frac{2·Precision·Recall}{Precision+Recall}}\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill FalsePositiveRate& ={\displaystyle \frac{FP}{FP+TN}}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill MatthewsCorrelationCoefficient& ={\displaystyle \frac{TP·TN-FP·FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}}\hfill \end{array}$$

(15)

These metrics provide a comprehensive evaluation of binary classification performance, especially in contexts where class imbalance may affect simple accuracy-based assessments. While accuracy gives an overall sense of correctness, it may be misleading when one class dominates. In such cases, recall and specificity become particularly important, as they reflect the model’s ability to correctly identify positive and negative instances, respectively.

Precision complements recall by indicating the reliability of positive predictions, which is essential in applications where false positives carry a significant cost. The F1-score, as the harmonic mean of precision and recall, offers a single metric that balances both aspects and is especially useful when the dataset has an uneven class distribution.

The false-positive rate (FPR) highlights the proportion of negative cases incorrectly labeled as positive and is a key component in Receiver Operating Characteristic (ROC) analysis. Finally, the Matthews Correlation Coefficient (MCC) provides a robust evaluation that considers all four components of the confusion matrix (TPs, TNs, FPs, FNs), making it a particularly valuable metric for assessing model performance on imbalanced datasets.

3. Results

This section is organized as follows. Section 3.1 provides an in-depth ablation of our primary contribution, the proposed Bi-LSTM model. Section 3.2, Section 3.3 and Section 3.4 present results from three state-of-the-art baselines, a U-Net, a DeepLab v3+ (both full-image CNNs), and a parameter-matched Vision Transformer (ViT-Tiny), for direct comparison against our approach. We conclude with a comprehensive summary table (Table 11) enabling side-by-side comparison of all key metrics. Standard convergence diagnostics—including (a) the training loss curve and (b) the precision–recall (P–R) curve for the ten-image test fold—are reported per method.

3.1. Bi-LSTM Ablation Study

We trained three variants that differ only in depth and dropout.

(a)

One-layer Bi-LSTM (50 hidden units), no dropout;

(b)

Three-layer Bi-LSTM + 50% dropout (Drop I);

(c)

Three-layer Bi-LSTM + dropout plus one inter-layer $3\times 3$ convolution (Drop II).

Table 1. Mean test-fold performance for the three Bi-LSTM variants (Section 3.1). Best values in bold.

Model	F1	MCC	FPR (%)	Params
(a) 1-layer	0.645	0.646	2.3	14,323
(b) 3-layer + Drop I	0.703	0.698	1.4	62,153
(c) 3-layer + Drop II	0.706	0.701	1.2	62,153

reports mean test performance; per-image metrics appear in

Table A1. Extended classification metrics from U-Net, trained on the set of 44 images (IDRiD_01.jpg–IDRiD_44.jpg).

NAME	Acc. (%)	TP (%)	TN (%)	FP (%)	FN (%)	R	S	P	F1	FPR	MCC
means	94.412	5.3561	89.056	2.561	3.027	0.63147	0.97206	0.68418	0.64806	0.027943	0.6429

Acc. = Accuracy, TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative, R = Recall, S = Specificity, P = Precision, $F_1$ = F1-Score, FPR = False-Positive Rate, MCC = Matthews Correlation Coefficient.

. Depth alone yields only a marginal +0.02 F₁, while adding dropout lifts F₁ by +0.06 and reduces the false-positive rate. Variant (c) (“Bi-LSTM-Drop II”) is therefore carried forward as our best Bi-LSTM. Figure 7 shows improved suppression of spurious thin branches without sacrificing connectivity.

3.1.1. Experiment 1

Column-wise processing using a single Bi-directional LSTM.

Table 2. Extended classification metrics from Experiment 1, trained on the set of 44 images (IDRiD_01.jpg–IDRiD_44.jpg).

	Acc. (%)	TP (%)	TN (%)	FP (%)	FN (%)	R	S	P	F1	FPR	MCC
means	94.960	4.768	90.192	1.425	3.615	0.561	0.985	0.774	0.645	0.016	0.646

Acc. = Accuracy, TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative, R = Recall, S = Specificity, P = Precision, $F_1$ = F1-Score, FPR = False-Positive Rate, MCC = Matthews Correlation Coefficient.

and

Table 3. Extended classification metrics computed per image (rounded to two decimals), obtained during the test phase of Experiment 1.

NAME	Acc. (%)	TP (%)	TN (%)	FP (%)	FN (%)	R	S	P	F1	FPR	MCC
IDRiD_45.jpg	95.78	4.64	91.14	0.65	3.57	0.57	0.99	0.88	0.69	0.01	0.69
IDRiD_46.jpg	90.41	6.54	83.87	7.08	2.51	0.72	0.92	0.48	0.58	0.08	0.58
IDRiD_47.jpg	95.12	5.71	89.41	1.40	3.47	0.62	0.98	0.80	0.70	0.02	0.69
IDRiD_48.jpg	95.04	2.78	92.26	1.34	3.62	0.43	0.99	0.68	0.53	0.01	0.54
IDRiD_49.jpg	94.95	5.77	89.18	1.84	3.21	0.64	0.98	0.76	0.70	0.02	0.69
IDRiD_50.jpg	96.15	2.45	93.70	0.63	3.22	0.43	0.99	0.80	0.56	0.01	0.58
IDRiD_51.jpg	94.70	6.43	88.27	2.67	2.63	0.71	0.97	0.71	0.71	0.03	0.70
IDRiD_52.jpg	94.80	6.83	87.97	2.49	2.71	0.72	0.97	0.73	0.72	0.03	0.71
IDRiD_53.jpg	95.48	4.75	90.72	1.44	3.09	0.61	0.98	0.77	0.68	0.02	0.67
IDRiD_54.jpg	95.48	6.32	89.16	1.44	3.08	0.67	0.98	0.81	0.74	0.02	0.73
means	94.791	5.223	89.568	2.096	3.112	0.612	0.977	0.741	0.660	0.023	0.657

Acc. = Accuracy, TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative, R = Recall, S = Specificity, P = Precision, $F_1$ = F1-Score, FPR = False-Positive Rate, MCC = Matthews Correlation Coefficient.

present the training and test results, respectively. Figure 5 illustrates the qualitative analysis.

3.1.2. Experiment 2

Column-wise processing using a network based on Bi-directional LSTM and dropout layers. Solution I.

Table 4. Extended classification metrics from Experiment 2, trained on the set of 44 images (IDRiD_01.jpg–IDRiD_44.jpg).

NAME	Acc. (%)	TP (%)	TN (%)	FP (%)	FN (%)	R	S	P	F1	FPR	MCC
means	95.665	5.359	90.306	1.311	3.024	0.631	0.986	0.797	0.703	0.014	0.698

Acc. = Accuracy, TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative, R = Recall, S = Specificity, P = Precision, $F_1$ = F1-Score, FPR = False-Positive Rate, MCC = Matthews Correlation Coefficient.

and

Table 5. Extended classification metrics computed per image (rounded to two decimals), obtained during the test phase of Experiment 2.

NAME	Acc. (%)	TP (%)	TN (%)	FP (%)	FN (%)	R	S	P	F1	FPR	MCC
IDRiD_45.jpg	96.25	5.16	91.09	0.70	3.05	0.63	0.99	0.88	0.73	0.01	0.73
IDRiD_46.jpg	93.46	6.68	86.78	4.18	2.37	0.74	0.95	0.62	0.67	0.05	0.66
IDRiD_47.jpg	95.54	6.28	89.26	1.55	2.91	0.68	0.98	0.80	0.74	0.02	0.73
IDRiD_48.jpg	95.31	2.93	92.38	1.22	3.47	0.46	0.99	0.71	0.56	0.01	0.56
IDRiD_49.jpg	95.31	6.28	89.02	2.00	2.70	0.70	0.98	0.76	0.73	0.02	0.72
IDRiD_50.jpg	96.47	3.17	93.29	1.03	2.50	0.56	0.99	0.75	0.64	0.01	0.64
IDRiD_51.jpg	95.71	6.86	88.86	2.08	2.21	0.76	0.98	0.77	0.76	0.02	0.75
IDRiD_52.jpg	96.08	7.22	88.86	1.60	2.32	0.76	0.98	0.82	0.79	0.02	0.77
IDRiD_53.jpg	95.55	5.45	90.10	2.06	2.39	0.70	0.98	0.73	0.71	0.02	0.70
IDRiD_54.jpg	96.06	6.84	89.22	1.38	2.56	0.73	0.98	0.83	0.78	0.02	0.77
means	95.575	5.688	89.887	1.778	2.647	0.670	0.981	0.766	0.710	0.020	0.704

Acc. = Accuracy, TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative, R = Recall, S = Specificity, P = Precision, $F_1$ = F1-Score, FPR = False-Positive Rate, MCC = Matthews Correlation Coefficient.

present the training and test results, respectively. Figure 6 presents the qualitative evaluation.

3.1.3. Experiment 3

Column-wise processing using a network based on Bi-directional LSTM and dropout layers. Solution II.

Table 6. Extended classification metrics from Experiment 3, trained on the set of 44 images (IDRiD_01.jpg–IDRiD_44.jpg).

NAME	Acc. (%)	TP (%)	TN (%)	FP (%)	FN (%)	R	S	P	F1	FPR	MCC
means	95.741	5.219	90.522	1.094	3.164	0.613	0.988	0.8194	0.700	0.012	0.697

Acc. = Accuracy, TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative, R = Recall, S = Specificity, P = Precision, $F_1$ = F1-Score, FPR = False-Positive Rate, MCC = Matthews Correlation Coefficient.

and

Table 7. Extended classification metrics computed per image (rounded to two decimals), obtained during the test phase of Experiment 2.

NAME	Acc. (%)	TP (%)	TN (%)	FP (%)	FN (%)	R	S	P	F1	FPR	MCC
IDRiD_45.jpg	96.21	4.97	91.25	0.55	3.24	0.61	0.99	0.90	0.72	0.01	0.73
IDRiD_46.jpg	93.83	6.42	87.41	3.54	2.62	0.71	0.96	0.64	0.68	0.04	0.66
IDRiD_47.jpg	95.68	6.07	89.62	1.20	3.12	0.66	0.99	0.84	0.74	0.01	0.73
IDRiD_48.jpg	95.37	2.80	92.57	1.02	3.61	0.44	0.99	0.73	0.55	0.01	0.56
IDRiD_49.jpg	95.44	6.16	89.28	1.74	2.82	0.69	0.98	0.78	0.73	0.02	0.72
IDRiD_50.jpg	96.43	2.86	93.57	0.75	2.81	0.50	0.99	0.79	0.62	0.01	0.62
IDRiD_51.jpg	95.84	6.71	89.13	1.80	2.35	0.74	0.98	0.79	0.76	0.02	0.75
IDRiD_52.jpg	95.80	7.15	88.65	1.81	2.39	0.75	0.98	0.80	0.77	0.02	0.76
IDRiD_53.jpg	95.69	5.38	90.31	1.85	2.46	0.69	0.98	0.74	0.71	0.02	0.70
IDRiD_54.jpg	96.15	6.81	89.34	1.26	2.59	0.72	0.99	0.84	0.78	0.01	0.77
means	95.645	5.532	90.113	1.551	2.803	0.650	0.983	0.786	0.706	0.017	0.701

Acc. = Accuracy, TP = True Positive, TN = True Negative, FP = False Positive, FN = False Negative, R = Recall, S = Specificity, P = Precision, $F_1$ = F1-Score, FPR = False-Positive Rate, MCC = Matthews Correlation Coefficient.

present the training and test results, respectively. Figure 7 presents the qualitative evaluation.

3.2. U-Net Baseline

To benchmark the proposed Bi-LSTM against a standard CNN, we trained a 61-layer U-Net that processes full $1024\times 1024$ fundus images (architecture in Figure 8c).

Two variants were explored.

(a)

Plain—no data augmentation;

(b)

+ Rotation augmentation—the same 15-angle policy used for the Bi-LSTM experiments.

The augmented U-Net attains 0.719 F₁ with 34 M parameters (

Table 8. Mean metrics for the U-Net baseline with rotation augmentation (Section 3.2).

Metric	Acc. (%)	F1	MCC	FPR (%)
Mean	95.92	0.719	0.715	1.4

); the plain model is 0.07 F₁ lower. Complete experimental details, including

• Training configuration (ADAM optimizer, 50 epochs);
• Metrics for both variants (plain and augmented);
• Per-image test results for all configurations.

are provided in Appendix A.

3.3. DeepLab v3+ Baseline

The ResNet-18 DeepLab v3+ (99 layers, 40 M parameters) improves F₁ by +0.001 over U-Net but at 6× the parameter count of Bi-LSTM (

Table 9. Mean metrics for the DeepLab v3+ (ResNet-18) baseline (Section 3.3).

Metric	Acc. (%)	F1	MCC	FPR (%)
Mean	95.89	0.720	0.716	1.4

). Full experimental details, including

• Training configuration (SGDM optimizer, 60 epochs, data augmentation);
• Extended training metrics (Table 1);
• Per-image test metrics (Table 2).

are provided in Appendix B.

3.4. ViT-Tiny Baseline

A lightweight ViT-Tiny encoder (5.72 M parameters) plus a four-stage U-Net decoder totals 6.1 M parameters, matching the Bi-LSTM scale. Despite the wider receptive field, ViT-Tiny lags the best Bi-LSTM by 0.207 F₁ and doubles the false-positive rate (

Table 10. Mean metrics for the ViT-Tiny baseline (Section 3.4).

Metric	Acc. (%)	F1	MCC	FPR (%)
Mean	93.89	0.612	0.580	2.9

). Complete experimental details, including

• Architecture design (ViT-Tiny encoder + U-Net decoder);
• Training protocol (patch-based processing, AdamW optimizer);
• Quantitative results (Table 1 and Table 2);
• Qualitative error analysis.

are provided in Appendix C.

3.5. Global Comparison

Table 11. Side-by-side comparison of the best Bi-LSTM versus all baselines.

Model	Params (M)	F1	MCC	FPR (%)
BiLSTM (Drop II)	6.4	0.706	0.701	1.2
U-Net (+ rot. aug.)	34	0.719	0.715	1.4
DeepLab v3+ (ResNet-18)	40	0.720	0.716	1.4
ViT-Tiny (+ U-Net dec.)	6.1	0.612	0.580	2.9

unifies the best Bi-LSTM and all three baselines on the most informative metrics, plus model size and pixel-wise inference speed. The proposed Bi-LSTM offers the highest F₁/MCC at the lowest parameter count, validating sequence-first reasoning as a competitive alternative to both CNN and transformer designs.

4. Discussion

4.1. Principal Findings

This study demonstrates that a parameter-efficient, purely recurrent architecture can achieve competitive retinal vessel segmentation. The best Bi-LSTM configuration obtained a mean accuracy over 95.6% and a Matthews Correlation Coefficient (MCC) over 0.703 on the held-out RETA–IDRiD test subset (Table 5, Table 6 and Table 7). Notice that all of these results are obtained with no pre- or post-processing.

Our models exhibit a significantly lower number of trainable parameters, confirming their lightweight nature while maintaining competitive performance.

Table 12. Model architecture summary: U-Net, DeepLab, Bi-LSTM, and ViT-Tiny.

Model	Learnable Parameters	Layers
Bi-LSTM (a)	14,323	4
Bi-LSTM (b)/(c)	62,153	9/8
DeepLab v3+ (with pretrained ResNet-18)	20,607,636	99
U-Net	31,031,810	61
ViT-Tiny	5,720,000	12

below summarizes the architectural complexity of the proposed Bi-LSTM variants compared to U-Net, DeepLab v3+, and ViT-Tiny.

4.2. Methodological Contributions

(a)

Column-wise sample inflation. Decomposing every image into 1024 vertical sequences increased the effective training set from 44 to 45,056 instances without synthetic texture manipulation, supporting robust optimization on a limited corpus.

(b)

Transparent probability maps. The model outputs full-resolution 16-bit probability maps rather than hard binary masks. Releasing these logs alongside the code allows for exact reproduction of every metric in Table 3 and Table 5, Table 6 and Table 7 and facilitates independent error analysis (e.g., threshold sweeps, calibration curves or post hoc connectivity pruning) without rerunning training.

4.3. Limitations

• Dataset scope. All images originate from a single-camera model and demographic pool. Generalizability remains unverified; a planned external validation on the DRIVE and RVD datasets will be reported in a subsequent submission.
• Resolution constraints. The fixed 1024-pixel sequence length may degrade on ultra-wide-field fundus images; hierarchical or overlapping-window LSTMs will be explored.
• Absence of post-processing. Although deliberate, omitting morphological refinement marginally depresses the $F_1$ score relative to state-of-the-art CNN pipelines. A lightweight and connectivity-sensitive refinement module is in development.

4.4. Clinical Relevance

Accurate and computationally inexpensive vessel maps facilitate point-of-care applications, including automated arteriole-to-venule ratio (AVR) estimation and longitudinal microvascular monitoring in primary care settings. The low memory footprint positions the model for deployment on embedded retinal cameras.

Although current models have achieved very good segmentation performance, they still fall short of 100% accuracy, indicating that the problem remains open and subject to further improvement. At present, these models serve as diagnostic support tools in the clinical context. As observed in other fields, immediate improvements in these techniques are likely to come from the fusion of the outputs of multiple models. In this context, it is valuable to provide diverse approaches to the problem—whether through CNN-based networks, visual transformers, or slightly different strategies such as the one proposed in this work. These models should benefit from the integration of both classical image processing techniques—similar to the approaches described in [31,32]—and modern deep learning strategies, leveraging the strengths of each paradigm to enhance segmentation accuracy and robustness.

4.5. Future Work

• Post processing (1): Remove background artifacts—false positives, shown in red in Figure 5, Figure 6 and Figure 7—via object-based analysis. This leverages the morphological contrast between rounded blobs (artifacts) and elongated, thin structures (vessels) (see Figure 9a,b).
• Post-processing (2): Improve true-positive detection using rotation-based inference. From qualitative inspection, it can be observed that processing the images column-wise prioritizes the detection of capillaries that evolve horizontally in the image, while hindering the detection of those oriented vertically—i.e., aligned with the direction of the columns. One way to enhance the detection of these vertically oriented capillaries is to apply 90-degree rotations to the image and reapply the method, thereby revealing structures that may have remained hidden. The method can be further improved by combining information obtained from multiple rotations, as certain capillaries that remain undetected in some regions may become visible under different orientations.
  The input image is rotated and processed multiple times, and then the outputs are realigned and aggregated to better capture fine vascular structures (see Figure 9a,c).
• Cross-device validation: evaluates the model on DRIVE, RVD, and other camera sources.
• Shallow CNN front-end: prepends a few $3\times 3$ convolutions to capture fine-scale texture before Bi-LSTM.
• Domain-shift mitigation: applies adversarial training so that precision is maintained in the new groups of patients.
• Ablation study: quantifies the impact of rotation policy, sequence length, and LSTM type before camera-agnostic release.

5. Conclusions

We propose a new approach to automated retinal vessel segmentation. The proposed solution processes the images in a column-wise manner. While the final solution to this task will likely involve a combination of different methods, introducing a conceptually distinct approach into the existing toolbox represents a meaningful contribution to the field. The results, obtained through three experiments, have been presented in raw form—without any pre- or post-processing—in order to clearly evaluate the potential of the proposed strategy.

Although a full-scale evaluation on the complete dataset has not yet been performed, preliminary observations suggest that post-processing involving the removal of false positives combined with multiple rotated inferences and realignment could significantly enhance segmentation performance. The proposed method supports an arbitrary number of inferences based on different image rotations, enabling the detection of small vessels from varying orientations.

It should also be noted that the models presented can be trained from scratch using a relatively small number of annotated images (44 in our case). This is a significant advantage compared to other methods, as image annotation requires highly specialized and labor-intensive manual work. So, some future work will focus on refining the post-processing pipeline and evaluating the method on a broader set of datasets, with the goal of developing a robust, lightweight solution suitable for real-world clinical deployment.