High-Level Hessian-Based Image Processing with the Frangi Neuron

Abstract

The Frangi neuron proposed in this work is a complex element that allows high-level Hessian-based image processing. Its adaptive parameters (weights) can be trained using a minimum number of training data. In our experiment, we showed that just one image is enough to optimize the values of the weights. An intuitive application of the Frangi neuron is to use it in image segmentation process. In order to test the performance of the Frangi neuron, we used diverse medical datasets on which second-order structures are visualized. The Frangi network presented in this paper trained on a single image proved to be significantly more effective than the U-net trained on the same dataset. For the datasets tested, the network performed better as measured by area under the curve receiver operating characteristic (ROC AUC) than U-net and the Frangi algorithm. However, the Frangi network performed several times faster than the non-GPU implementation of Frangi. There is nothing to prevent the Frangi neuron from being used as part of any other network as a component to process two-dimensional images, for example, to detect certain second-order features in them.

1. Introduction

Neural networks that utilize convolutional and MaxPooling layers dominate modern multi-scale digital image processing. It is an effect of high approximation capabilities of such architectures and relative simplicity. In order to prepare the network to start solving a particular problem, it is required to gather a sufficiently large training set due to which the network parameters can be trained. In some problems, gathering a sufficiently large dataset is difficult and expensive, and sometimes even impossible. This is especially true for medical data, for example, where even a large medical unit may not have abundant data for a certain disease. Also, data augmentation may not be sufficient if the collected data are not sufficiently representative. It should not be forgotten that before the era of deep neural networks, there were many effective mathematical models that could be used effectively for signal processing. They require a much smaller number of adaptive parameters to be specified in order to operate effectively than is the case with convolutional neural networks. The trade-off is expense of computational complexity and sophistication of these algorithms. On the other hand, with the use of modern computational frameworks, many complex algorithms can be implemented as elements of neural networks and then used as integral parts of the structure of neural networks that learn along with the overall architecture of the network. Vessel enhancement filters (aka vesselness) have been part of angiographic image processing for many years [1]. However, the application of these methods is not limited to blood vessels. Successful applications of such filters have also been reported in detecting facial wrinkles [2] or saliva ferning prediction [3]. Blood vessel filtering uses a variety of methods. Among the most popular are deep neural networks [4], particularly U-network-based architectures [5], directional convolutional kernels [6], various approaches to region growing [7] and applied mathematical morphology [8]. In recent years, there have been several proposals to improve the response of hessian-based methods by, for example, using swarm optimization [9], genetic programming and other approaches [10,11,12,13,14,15,16,17]. In this work, we propose so-called Frangi neurons that allow filtering second-order structures from digital images and for optimizing parameters of which a minimal training set consisting of even a single image is required. We show that the use of a Frangi neuron in a properly constructed neural network facilitates better performance in the task of segmenting second-order structures than the state-of-the-art solution U-net [18] and allows training the model using only one training image. Frangi filtering [19] is a commonly used method, most often in situations where tubular or linear structures such as blood vessels need to be segmented. So-called Vesselness Filters [20] are used in liver [21], retinal [22], brain [23] finger [24], face wrinkles [25], and fabric defect [26] imaging. Frangi filtering is used effectively in optoacoustic (photoacoustic) imaging [13], computed tomography [27], magnetic resonance [23] or simple digital photography [25]. In the literature, methods have been proposed to optimize the parameters of the Frangi filter; for example, work [28] uses an ordered logit model for this purpose. In our work, these parameters are optimized while training the neural network and no additional optimization procedure is needed.

The construction of a neural network inspired by Frangi’s algorithm is described in paper [29]. In practice, it is a convolutional network in which layers are initialized with parameters so that they compute the Hessian of the input image. Those parameters are optimized during training. In total, there are nine kernels and six parameters to update during training. In our approach, there is no restriction to the number of parallel Frangi blocks, and the Hessian calculation is fixed. In this way, we obtain a significantly larger filter response to second-order structures with different diameters. Furthermore, in this work, we define a single Frangi neuron that can stand alone as a component of any neural network and not just one particular architecture.

Our proposed method, like one-shot semantic segmentation networks, allows tuning the parameters to adapt to a new class of images. However, unlike the models proposed, for example, in works [30,31], the parameters of the segmentation networks are not based on convolutional layers, but on a ready-made hessian-based model. This results in a significant reduction in network tuning time at the expense of reducing the flexibility of the segmentation model.

The hessian-based second-order structure detection algorithm is, in fact, a well-known approach. An important novelty that we propose in this paper is to enable the use of this technique in the form of a dedicated neural layer. The recent developments in the design of new types of deep learning architectures have made it possible to directly optimize the parameters of such filters in a deep learning framework. Lately, other layers using efficient signal processing algorithms were defined, such as, for example, Zernike layers [32,33,34] or wavelet transform layers [35]. The theoretical principles of these methods are well known, as in the case of Frangi’s filter. Creating neural network layers based on them allowed us their effective use in deep learning models. We hope that a publication showing the effectiveness and possible applications of the second-order Hessian approach encapsulated in the neural layer will be useful to the scientific community.

2. Materials and Methods

In this section, we present the Frangi neuron mathematical model and the datasets on which we test its effectiveness in practice.

2.1. Frangi Filter

One of the most influential methods on multiscale tubular structure filtering has been described by Frangi et al. in [19]. This method uses the analysis of the second partial derivative of an image (Hessian) to detect local structures of this image. In the remainder of this paper, we consider the two-dimensional case. The Frangi filter follows Algorithm 1.

Algorithm 1: Frangi filtering algorithm.

Calculation of eigenvalues from a symmetric matrix of size 2 × 2 can be performed according to Equation (1).

$$\left\{\begin{array}{c}{\lambda }_1=\frac{H_{xx}+H_{yy}+\sqrt{{(H_{xx}-H_{yy})}^2+4X_{xy}^2}}{2}\\ {\lambda }_2=\frac{H_{xx}+H_{yy}-\sqrt{{(H_{xx}-H_{yy})}^2+4X_{xy}^2}}{2}\end{array}\right..$$

(1)

The computations performed in a loop of Algorithm 1 that uses different values of $sigma$ from the set $sigma$ are independent of each other and can be performed in parallel. Only the last operation $R=max(R,V_{I_{sigma}})$ aggregates the partial results via element-wise (pixel-wise) maximum. This fact can be used to build a parallel neural network architecture inspired by the performance of Frangi’s original algorithm.

2.2. Frangi Neuron

Let us define a Frangi neuron as an indivisible block of operations that takes a tensor of size (n,m) as its input and returns a tensor of identical size. The operations performed by the Frangi neuron are identical to the operations performed in the individual loop iterations of Algorithm 1. Each loop iteration of Algorithm 1 uses three parameters: $\sigma$, $\beta$, and $\gamma$. In the case of the Frangi algorithm, these are invariant and are defined before the filter is run. However, we can optimize them to achieve the desired effect by minimizing the network’s loss function. Each Frangi neuron can thus independently have three parameters subject to optimization: $\sigma$, $\beta$, and $\gamma$. We now discuss the practicality of optimizing each parameter in the neural network learning process given a potential implementation of the Frangi neuron.

Parameter $\sigma$ is a parameter of the Gaussian kernel. Its value affects the diameter of the structures detected by the filter. In practice, the value of $\sigma$ determines the window size of the Gaussian filter $k_{dim}$, for example according to Equation (2) (see source codes of [36]).

$$k_{dim}=2\ast ⌊4\sigma +0.5⌋+1,$$

(2)

where ⌊ ⌋ is the rounding down operator. This fact causes, in the case of processing data in a single instruction multiple data pipeline, parameter $\sigma$ to become troublesome because the implementation of the convolution filter, e.g., in Tensorflow, requires a predefined filter window size. For this reason, the Frangi neuron architecture proposed in this work has a fixed value of $\sigma$ that is not subject to optimization. Parameters $\beta$ and $\gamma$ are threshold values for blob-like structures and the second-order structureness ratio. They are easy to optimize because changing them does not change the structure inside the neuron. The values of $\beta$ and $\gamma$ must be different from zero and there is no point in considering their negative values. Since they are arguments of the denominator of the exponent of the exponential function, there is a certain range of values in which they affect the performance of the filter. For this reason, we impose constraints in the optimization process: $0.05\le \beta <10$ and $0.05\le \gamma <50$. Outside this range of values, the Frangi algorithm may become stuck in the local minimum. The Frangi neuron diagram is shown in Figure 1.

2.3. Example Application: Network Using Frangi Neurons for Image Segmentation (Frangi Network)

An intuitive application of the Frangi neuron is to use it in an architecture for the detection of longitudinal structures of varying diameters. We present our proposed network architecture of this type in Figure 2. This network consists of m Frangi neurons, each initialized with different $\sigma$ values according the Equation (3). The range of sigma values depends on the range of diameters of the structures we want to detect. In practice, in the remainder of this paper, we used the set of $\sigma$ values defined according to equation

$$\sigma =[1,1.25,1.5,1.75,2,2.25,2.5,2.75,3,3.25,3.5,3.75,4,5,6,7,8,9,10],$$

(3)

which proved to be very effective for the datasets tested. Frangi neurons are shown as green circles in Figure 2.

In Algorithm 1, each loop result with a given $\sigma$ has an identical weight. In our approach, we added a single block after each Frangi neuron to scale the neuron’s response. We called it a scaling neuron. Its operation is to multiply the incoming signal by value I in the interval $I\in [0,1]$. The value of I is subject to optimization during network training. These blocks are shown in Figure 2 as yellow circles. The next step is to aggregate the results of the individual filters by computing the pixel wise maximum and scaling the results to the interval $[0,1]$. Values close to 0 indicate no detection of second-order structures. Values close to 1 indicate a high probability that second-order structures are present. The network can be trained using the stochastic gradient descent method of, for example, Adam [37]. The Frangi neuron implements complex functionality and hence needs a very small training set to optimize its parameters. Also, the Frangi network can be practically trained using only one (!) training image. In our implementation, we used an augmentation of the training set inspired by the augmentation used in the U-net [18]. We used random cropping, random brightness, contrast and saturation modification, random rotation, shearing, horizontal and vertical flipping, shifting and zooming. We used binary crossentropy as the loss function. DICE, IoU as well as binary crossentropy [38] are used to train the neural network used for segmentation. Modern scientific literature suggests that crsossentrop-based loss functions should be used first as a target function, which is universal for various segmentation problems, especially for medical images [38,39].

2.4. Datasets

In order to test the performance of a Frangi neuron, we used diverse medical datasets on which second-order structures are visualized. The following datasets were used in other research work and the license allows us their use in our work. We did not perform any research on humans to gather these data. Data do not contain personally identifiable information or offensive content.

2.4.1. Retinopathy (Retina)

First dataset that we used is a retinopathy database [40] that was gathered to enable comparative studies on segmentation of blood vessels in retinal images. That dataset consists of 40 images in total. Overall, 20 images are in the training group, while the remaining 20 images are in the test group. Both the training and test groups have reference manual segmentations and binary masks that indicate only the region containing the eye ball. Only the masked area is evaluated during segmentation quality testing. The images and their manual segmentations are made publicly available at https://drive.grand-challenge.org/ (Access date: 4 September 2023).

2.4.2. X-ray Coronary Artery Angiography (Coronary Artery)

The second dataset contains four X-ray coronary angiography images [41]. The images show blood vessels of various sizes. A high level of background noise as well as the blood vessels themselves are evident. A manual segmentation is provided for each image. Since there is no distinction between training and test set, training on this set is performed by selecting its subset to be used in training. In terms of image quality as well as diameter variation, this dataset is much more difficult to segment than the retina one.

2.4.3. Brain MRA (Brain)

The last dataset contains five images of brain magnetic resonance angiography (MRA) [42]. Images depict vessels of varying thickness. MRA reconstructions show different axial slices of human brain. Manual segmentation is provided for each image. Since there is no separation between training and test set, training on this set is performed by selecting a subset of it to be used for training.

3. Results

We implemented our solution using Python 3.8, tensorflow 2.8.0, keras 2.8.0, tensoflow-addons 0.16.1 (for Gaussian smoothing). The source code of the proposed algorithm can be downloaded from the GitHub repository https://github.com/browarsoftware/FrangiNeuron (Access date: 4 September 2023). We tested the performance of the Frangi neuron in the Frangi network (see Section 2.3) by comparing its performance to the Frangi filter and the U-net. We used the Frangi filter implementation from the scikit-image package 0.19.2. For the Frangi filter, we set $\sigma$ values according to Equation (4).

$$\Sigma =[1,1.25,1.5,1.75,2,2.25,2.5,2.75,3,3.25,3.5,3.75,4,5,6,7,8,9,10].$$

(4)

Weobtained the U-net implementation from the GitHub repository, https://github.com/jocicmarko/ultrasound-nerve-segmentation (Access date: 4 September 2023). This source code is available under the MIT licence. Both the Frangi network and the U-net were trained on a single image in the training set at 200 epochs. We repeated the training on every single image in the set, and then the remaining images were used to test the results. For training purposes, we used the Adam optimizer [37,43] with a learning rate equal to ${10}^{-2}$ and a batch size equal to 16. In all cases, only one image was used for training, and the rest wete used for validation (one-shot learning). We applied cross-validation, so after the Frangi network and the U-net were trained and validated, the network weights were randomly initialized, and another image was selected for training and the rest of the dataset was selected for validation. We performed 20 trainings and validations for the retina dataset, 4 trainings and validations for the artery dataset and 5 trainings and validations for the brain dataset. The quality of segmentation was evaluated using the Area Under the Receiver Operating Characteristic Curve (ROC AUC) measure, which is an appropriate metric for evaluating this type of task. We used the scikit-learn 1.0.2 package for this. Our implementation and the data used are presented in the https://github.com/jocicmarko/ultrasound-nerve-segmentation (accessed on 4 September 2023). The calculations were performed on Intel Core i7 3.00 GHz CPU, 64 GB RAM, NVIDIA GeForce RTX 2060 GPU, Windows 10 OS.

The results of the segmentation performance of each method can be seen in Figure 3, Figure 4 and Figure 5 and in

Table 1. ROC AUC value of test dataset calculation for various segmentation algorithms. The bottom row contains the average execution time of the algorithm on the retina set given in seconds.

	Frangi Network	Frangi	U-Net
Retina	0.8684	0.7883	0.7769
Coronary artery	0.9117	0.9110	0.7264
Brain	0.9397	0.9297	0.5720
Exec time [S]	0.160	1.187	0.143

,

Table 2. Coefficients assessing the quality of segmentation of test datasets calculated for the Frangi network.

	Sensitivity	Specificity	IoU	VOE	F1-Score/Dice
Retina	0.6648	0.9557	0.4251	0.5749	0.5966
Coronary artery	0.9891	0.9143	0.4842	0.5158	0.6526
Brain	0.9242	0.9133	0.4768	0.5232	0.6451

,

Table 3. Coefficients assessing the quality of segmentation of test datasets calculated for the Frangi filter.

	Sensitivity	Specificity	IoU	VOE	F1-Score/Dice
Retina	0.6244	0.9441	0.3648	0.6352	0.5348
Coronary artery	0.9551	0.9144	0.4677	0.5323	0.6377
Brain	0.9139	0.9166	0.4804	0.5196	0.6487

and

Table 4. Coefficients assessing the quality of segmentation of test datasets calculated for the U-net network.

	Sensitivity	Specificity	IoU	VOE	F1-Score/Dice
Retina	0.6506	0.9152	0.3130	0.6870	0.4767
Coronary artery	0.8088	0.8242	0.2577	0.7423	0.4095
Brain	0.6457	0.6751	0.1430	0.8570	0.2506

. Values in the tables are averaged. The last row of Table 1 contains the average execution time of the algorithm on the retina set given in seconds. We calculated Sensitivity, Specificity, Intersection of Union (IoU/Jaccard), Volumetric Overlap Error (VOE) [44] and F1-score/Dice to make the results more comprehensive. We found those metrics most appropriate. Other popular scorings like Average Symmetric Surface Distance (ASSD) that measures the average of all distances for every point from one object to the other and vice versa are not suitable for tubular structure segmentation but rather for regions or volumes segmentation [45,46].

4. Discussion

Both the results obtained in Table 1, Table 2, Table 3 and Table 4 and the visualizations in Figure 3, Figure 4 and Figure 5 prove that the Frangi neuron fulfills its role as a detector of second-order structures. Both AUC ROC and other evaluated metrics consistently show that the proposed network based on Frangi neurons performs better, and, in borderline cases, generally no worse than the stationary Frangi filter. On the other hand, the qualitative advantage of the obtained segmentations over the U-net architecture is strongly evident. This is not due to the weakness of this universal and efficient architecture like U-net, which is a kind of state-of-the-art solution for segmentation tasks. It is simply due to the fact that model parameter fitting is performed on the basis of a one-element learning set. It can be said that the proposed network based on Frangi neurons can be classified among one-shot segmentation approaches. This reveals the great flexibility of the proposed architecture and may be its advantage over universal models such as U-net in some specific applications. The results on the retina set are much worse than, for example, in [47], because the U-net was trained on only one image (one-shot learning) and not on the full 20-element training set. The limitation on the size of the training set was intended to show the limitations of the models purely based on convolutional filters (U-net-like architectures). The Frangi filter performances on all datasets in not poor as it might be expected. As can be seen in Table 2 and Table 3, specificity, IoU and F1-score factors are quite similar to those of the Frangi network. In the case of the Frangi filter, however, the sensitivity value is lower, which means that the filter without the optimization procedure performs worse in identifying true positive values. On the other hand, the fact is that the Frangi filter on the tested datasets performs better than the U-net if this deep neural network is not trained on a sufficiently large dataset. A single image in the training set is enough for the Frangi network using the Frangi neuron to optimize its weights to detect the relevant image fragments. Compared to the Frangi filter alone, the results returned by the network have much stronger contrast. Frangi filtering has the disadvantage of highlighting the boundaries of an area of solid colour as the location of second-order structures. This is clearly visible in the retina set (see Figure 3). This phenomenon cannot be eliminated in a network architecture using only Frangi neurons because it is a feature of a Hessian-based approach. If the filtered area does not have such clear boundaries, this effect does not occur as can be seen, for example, in the coronary artery dataset (see Figure 5). In Figure 3 and Figure 4, the proposed Frangi neuron model might have bias for segmenting the contour. This is typical behaviour of any method that uses various types of edge detectors. In practice, if an image is known to have a globally definable edge separating the area of interest from the background, it is eliminated from the segmentation process by applying an image mask. Such an edge removal procedure was also used in [40]. Thus, this area does not affect the physician or clinician’s decision, since its removal is a well-known practice. For the datasets we considered, a single image was insufficient for the U-net to produce generalizations of the segmentation process despite the fact that it uses image augmentation analagous to that of the Frangi network. Trained on a single image, the U-net either removes too many parts of the image or leaves fractions of the image that do not contain the desired structures. The ROC AUC of the Frangi network has the largest (best) values of all tested methods for all datasets. The computation execution time on the U-net is about $10\%$ faster than on the Frangi net. The Frangi net performs more than seven times faster than the Frangi filter using the same $\Sigma$ set. We did not optimize the speed of the Frangi neuron because we used the available standard Tensorflow and Keras functions in its implementation, so we assume that its performance can be improved. In summary, the Frangi neuron met the expectations and is an effective component of the network performing segmentation of various types of second-order structures. In most advanced deep neural one-shot network models for image segmentation, the authors use pretrained models on large and comprehensive image datasets [48,49]. This is an important limitation because the most popular image databases such as imagenet [48] and COCO [50] do not contain medical modalities. In contrast, solutions based on Hessian matrix analysis are immediately ready to detect tubular structures. In medical practice, we often have a very limited number of medical images presenting the modality of interest to physicians, on which a particular biological structure can be seen. In such case, even transfer learning is problematic due to the high complexity of the deep learning model and the shortage of training data. The remedy might be the use of image augmentation, which we also applied in this work for the U-net.

5. Conclusions

The neuron proposed in this work is a complex element that allows high-level Hessian-based image processing. Its adaptive parameters (weights) can be trained using a minimal amount of training data. In our experiment, we showed that just one image is enough to optimize the values of the weights. An intuitive application of the Frangi neuron is to use it in image segmentation process. The Frangi network presented in this paper trained on a single image proved to be much more effective than the U-net trained on the same dataset. For the tested datasets, the network achieved the best results as measured by ROC AUC as the Frangi algorithm runs, however, several times faster than the non-GPU implementation of Frangi. The Frangi network implementation prepared for this work uses a massive GPU parallelized algorithm. For this reason, it runs several times faster than the non-GPU implementation of the Frangi filter. The U-net, on the other hand, runs at a speed on the GPU comparable to that of the proposed Frangi network. Ultimately, the goal should be to have every algorithm that performs image processing implemented multithreaded, if it can be accelerated in this way. There is nothing preventing the Frangi neuron from being used as a part of any other network as a processing element for two-dimensional images, for example, to detect some second-order features in them. We proposed a method for the detection of second-order features.

Our proposed method has some limitations caused by the implementation. It would be beneficial to implement the ability to optimize the values in the $\Sigma$ set. This would allow us the use of a much smaller set of $\Sigma$ values which would speed up the computation. We can manually overcome such problems by re-implementing Tensorflow’s two-dimensional Gaussian filtering. Tensorflow’s built-in Gaussian filtering requires the float value sigma, which cannot be a Tensorflow variable. For this reason, sigma is not directly optimized in the GPU processing pipeline. However, we decided that Tensorflow’s current built-in implementation of Gaussian filtering is sufficient to demonstrate the effectiveness of the proposed method. The second limitation is that the Frangi neuron is less flexible compared to a typical convolutional architecture. The Frangi neuron is designed to detect certain second-order structures and cannot cope with other types of scenarios. On the other hand, it is an excellent alternative to convolutional networks, when we have a small number of training data pieces, which is very common for medical data, where gathering and manual segmentation of the training set is a difficult (due to small number of available cases) and expensive task. Frangi neuron is an image processing method that is dedicated to the segmentation of tubular structures. The most intuitive application is the detection of various types of bio structures. In practice, a wide variety of blood vessels are often analyzed [51,52,53], including those that form tree-like structures (for example, lungs) [54]. Tubular structures also form the intestine [55,56] and trachea [57,58]. We hope that the Frangi neuron will find its application in the above-mentioned areas.