Next Article in Journal
Effects of Cation Exchange in Rhodamine B Photocatalytic Degradation Using Peroxo-Titanate Nanotubes
Previous Article in Journal
Self-Reconstructed Metal–Organic Framework-Based Hybrid Electrocatalysts for Efficient Oxygen Evolution
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Nanomaterials
Volume 14
Issue 14
10.3390/nano14141169
nanomaterials-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Efficientand Robust Automated Segmentation of Nanoparticles and Aggregates from Transmission Electron Microscopy Images with Highly Complex Backgrounds

by
Lishi Zhou
1,
Haotian Wen
1,
Inga C. Kuschnerus
1,2 and
Shery L. Y. Chang
1,2,*
1
School of Materials Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia
2
Electron Microscope Unit, Mark Wrainwright Analytical Centre, University of New South Wales, Sydney, NSW 2052, Australia
*
Author to whom correspondence should be addressed.
Nanomaterials 2024, 14(14), 1169; https://doi.org/10.3390/nano14141169
Submission received: 25 February 2024 / Revised: 26 June 2024 / Accepted: 2 July 2024 / Published: 9 July 2024
(This article belongs to the Section Theory and Simulation of Nanostructures)
Browse Figures
Versions Notes

Abstract

:
Morphologies of nanoparticles and aggregates play an important role in their properties for a range of applications. In particular, significant synthesis efforts have been directed toward controlling nanoparticle morphology and aggregation behavior in biomedical applications, as their size and shape have a significant impact on cellular uptake. Among several techniques for morphological characterization, transmission electron microscopy (TEM) can provide direct and accurate characterization of nanoparticle/aggregate morphology details. Nevertheless, manually analyzing a large number of TEM images is still a laborious process. Hence, there has been a surge of interest in employing machine learning methods to analyze nanoparticle size and shape. In order to achieve accurate nanoparticle analysis using machine learning methods, reliable and automated nanoparticle segmentation from TEM images is critical, especially when the nanoparticle image contrast is weak and the background is complex. These challenges are particularly pertinent in biomedical applications. In this work, we demonstrate an efficient, robust, and automated nanoparticle image segmentation method suitable for subsequent machine learning analysis. Our method is robust for noisy, low-electron-dose cryo-TEM images and for TEM cell images with complex, strong-contrast background features. Moreover, our method does not require any a priori training datasets, making it efficient and general. The ability to automatically, reliably, and efficiently segment nanoparticle/aggregate images is critical for advancing precise particle/aggregate control in biomedical applications.

1. Introduction

Nanoparticles have a wide range of applications, including biomedical diagnostics [1], energy storage [2], catalysis [3,4], agricultural production [5], and environmental protection [6]. An excellent example is quantum dots, a class of nanoparticles, whose broad impact earned them recognition with the Nobel Chemistry prize in 2023. Among the applications of nanoparticles, biomedical applications [7,8,9,10] such as medical imaging contrast enhancement [11] and drug delivery [12,13] are growing rapidly. The key to the success of these applications is the control of their size, shape, and dispersions. For biomedical applications, the impact of the aforementioned factors on cellular uptake [7,8,9], biomolecule absorption [8], and their cytotoxicity [9,10] is critical in establishing the effectiveness and safety of the synthesized particles [14]. While many methods have been demonstrated to successfully control the size, shape, and dispersions [15,16,17], reliable characterization and analysis methods are still required to validate the nanoparticles to ensure that they meet the desired conditions in their working state.
Transmission electron microscopy (TEM) is a powerful technique capable of providing direct information on the nanoparticle size, shape, and structure down to the atomic level [18,19]. By utilizing TEM imaging under cryogenic conditions, it becomes possible to measure the native state of nanoparticle dispersion by plunge-freezing the dispersion. However, analyzing statistically representative numbers of particles from a large number of images is very laborious. The need for efficiently analyzing large amounts of data has driven the recent development of machine learning (ML) methods [20,21] for automatically identifying and classifying nanoparticles, garnering much attention.
In order to correctly and accurately apply machine learning methods for nanoparticle shape analysis, particle segmentation in the pre-processing step is critical in identifying and extracting nanoparticle metrology information. There are numerous methods for particle segmentation, ranging from threshold-based to more sophisticated deep-learning-based approaches.
Threshold-based image segmentation methods, including global and local thresholding methods such as Otsu’s method [22], Sauvola algorithm [23], and Niblack algorithm [24], are widely utilized. Some threshold-based methods such as cross-entropy thresholding [25] and variational theory-based methods [26] possess good adaptive capabilities, enabling automatic determination of the optimal separation threshold. Despite their prevalence, these methods, which depend on single or limited thresholds for segmenting images, frequently encounter difficulties in processing complex TEM images. Such images often present challenges like uneven background intensities [27,28] and variable particle intensities [29], hindering the achievement of optimal segmentation results.
To overcome the shortcomings of threshold-based approaches, machine-learning-based image segmentation approaches have gained much interest. Unsupervised machine learning segmentation approaches, such as k-means [30,31,32], mean shift [33,34,35,36,37], and GrabCut [38,39,40], show significant improvement over threshold-based methods. A recent report has demonstrated the effectiveness of combined template matching and image segmentation in extracting contours of platinum and core–shell nanoparticle catalysts from a complex background arising from the catalyst support [41]. However, this method may miss particles with shapes that deviate from the “template” or those with a high-contrast background.
On the other hand, supervised, deep learning methods such as convolutional neural networks (CNNs) can achieve very accurate particle segmentation even with highly complex background structures. However, deep learning methods require large amount of experimental or simulated training data. In the case of TEM images of nanoparticles, considering the varieties of imaging modalities and particle background features (such as support in the case of catalysis or cell features in nanoparticle uptake in cells), such deep learning approaches are undesirable as they require a priori training data sets, making automation for a wide range of TEM nanoparticle images very challenging.
Herein, we have developed a new method for nanoparticle segmentation using Shannon-entropy-optimized slope difference distribution (SEO-SDD). This method is an unsupervised clustering method capable of automatically finding the optimum image intensity clusters, thereby achieving particle segmentation from images. Such a method overcomes the shortcomings of intensity-threshold-based methods that generally fail when there are large background intensity variations. Moreover, as this method is unsupervised, it does not require a priori data training needed for the deep learning methods.
The effectiveness of our method is demonstrated on a deliberately chosen exemplar nanoparticle system of detonation nanodiamond (DND). DNDs are ≈5 nm diamond nanoparticles that are synthesized by the detonation process [42]. Due to their small particle size, flexible surfaces for functionalization, and biocompatibility, they have shown great promise for biomedical applications in drug delivery [43,44,45,46], gene therapy [47,48,49], tissue engineering, and bone surgery [50,51]. Because of their small size and weak contrast in TEM images, the segmentation of DNDs represents a challenging example for particle segmentation. Many existing automated particle segmentation processes for machine learning were demonstrated on nanoparticles with high image contrast, such as noble metal particles [52,53,54] and transition metal oxides [41], making them easier to identify due to stronger contrast differences from the background.
Moreover, we chose to use cryo-TEM images of frozen DND dispersion, which give noisier images compared to conventional TEM images. In addition, TEM images of DNDs taken up by cells were chosen as another example, as the complex features within the cells represent another great challenge in DND particle segmentation. The successful application of these DND examples demonstrate that our methods are efficient, effective, and robust.

2. Methodology

Generally, the workflow of machine learning (ML) morphology categorization for nanoparticles (Figure S1) is divided into two parts: the first part is pre-processing, which converts the raw image data into nanoparticle contour datasets for the subsequent ML algorithm. This part involves the denoising of raw images, followed by segmentation of nanoparticles. The second part is the application of the ML algorithm for shape categorization. As mentioned in the introduction, the focus of this work is on particle segmentation. In this section, we firstly describe our new Shannon-entropy-optimized slope difference distribution (SEO-SDD) particle segmentation method. Then, the ML algorithm, the sample preparation, and image acquisitions will be described.

2.1. Particle Segmentation: Automated Shannon-Entropy-Optimized Slope Difference Distribution (SEO-SDD) Method

The first step of the pre-processing is denoising. Here, we employed a non-local mean (NLM) denoising algorithm [55] to reduce noise levels. Subsequently, we applied an SEO-SDD particle segmentation method (Figure 1) to obtain an accurate binary image of the nanoparticle.
As TEM images can exhibit various types of noise [56] arising from sample preparations, electron beam configuration, electron doses, and the detectors [57,58,59], some TEM images of nanoparticles can be particularly noisy. Examples include electron-beam-sensitive materials that require imaging at a low-electron dose, low-atomic-number materials such as carbon-based nanoparticles, and cryo-TEM images of frozen biological and nanoparticle dispersion samples [60]. These factors contribute to a low signal-to-noise ratio in the images, resulting in a significant amount of noise and low particle contrasts.
We chose NLM denoising as it is an effective approach for noise reduction and has been demonstrated to work well in electron microscopy images and spectroscopy maps [61,62]. The key principle of NLM is to find similar regions within the image and replace the original pixel values with the weighted mean pixel values of these regions, aiming to achieve denoising.
After applying NLM denoising, we obtain a higher signal-to-noise ratio TEM image, as shown in Figure 1a. Next, we use a fully automated Shannon-entropy-optimized slope difference distribution (SEO-SDD) method that we have developed to extract the particles in the image and convert it into a binary image (Figure 1g).
For a TEM image, the intensity of the image generally changes rapidly near the edges of particles. This means that there is an intensity gradient near the edges. Such a feature is common for TEM images (and other types of microscopy images) and is independent of the background intensity, contrast, and intensity variation within the particles. The principle of the SDD method utilizes such a feature to remove the background, thereby identifying the particles.
The SDD method, firstly developed by Wang [63], is a method for determining the center of data clusters in unsupervised machine learning (a more detailed description of the SDD method is given in Supplementary Materials Section I). It has been applied in fluorescence and scanning electron microscopy images [64]. It has primarily been used for particle and cell counting rather than for automated particle contour extraction. The workflow of the SEO-SDD method is shown in Figure 1, and the formulation of the SDD method is described below.
For a given image (Figure 1a), I ( x ) , its corresponding gradient image (Figure 1b), I g ( x ) , can be computed by combining the magnitude and the direction of the gradient image
I g x = I i 2 x + I j 2 x
Θ = a t a n 2 I j x , I i x
where I i denotes the gradient along x-coordinates and I j along y-coordinates. The fundamental principle of the SDD method is to find the extrema of the slope difference distribution function, s ( x ) , from the gradient image, I g ( x ) . The normalized gradient image intensity histogram distribution, P ( n ) , is defined as
P n = f n f m a x ,
where f n denotes the frequency of a pixel intensity value n and f m a x is the maximum frequency of a pixel value. For a given point n in the histogram distribution P ( n ) , two slopes—one on the left, a L ( n ) , and the other on the right, a R ( n ) —of the point adjacent to n can be calculated. The slope difference distribution function, s ( n ) , is then defined as
s n = a L n − a R n .
The segmentation of particles in an image can then be treated as finding the centers of different clusters from s ( n ) by setting the derivative of s ( n ) to zero, given as
d s n d n = 0 ; n ( x ) > 0
By sorting and selecting the intersections of the derivative with the horizontal axis, we can then obtain a set of values as the centers of the “intensity clusters” (marked as red crosses in Figure 1c), which are the candidate intensity thresholds for P ( n ) . Therefore, this method provides a more flexible approach for nanoparticle segmentation compared to conventional histogram-threshold-based approaches [21,65,66,67]
Having obtained a set of candidate threshold values based on Equation (5), the next challenge of the SDD method is to select the optimum thresholds. Previously, such a parameter requires manual input and can suffer from overly harsh background intensity removal [65,67] or excessive remaining noise. In order to achieve fully automated and robust particle segmentation, we propose a Shannon-entropy-optimized [68] binary search algorithm to determine the optimum intensity thresholds. It should be noted that the conventional binary search algorithm searches for the value sequentially when the value being sought is known [69]. In our case, since the optimal threshold is not known, a figure-of-merit evaluation function is needed to determine the optimum thresholds.
Here, we chose Shannon entropy [68], H, as the evaluation criterion. Shannon entropy is used to quantify the complexity or uncertainty of the data. For an image with a pixel intensity value, n, its Shannon entropy can be defined as
H = − Σ i p ( n i ) log 2 p ( n i )
where p ( n i ) is the probability that the pixel value takes on a specific intensity value n i . The total entropy for the image therefore is given as
H ( N ) = H N a n o p a r t i c l e s + H B a c k g r o u n d
The larger the value of the entropy, the more information an image contains. In other words, the data have lower entropy when an image contains more similar information. Therefore, by maximizing H ( N ) , it is possible to achieve automated and robust particle segmentation.
The steps of the Shanon entropy optimization in our SEO-SDD method are described below:
1.
Set the initial range of threshold selection by choosing the appropriate upper, B, and lower, A, limits. The range can be read from the curve of the differential of slope difference (shown in Figure 1c). The upper limit B differentiates the background from particles, and the lower limit A differentiates particles with different intensities.
2.
Calculate the median and its corresponding Shannon entropy. The essence of the binary search algorithm lies in iteratively narrowing the search range by comparing the median value, m i d = ( A + B ) / 2 , with the currently considered optimal threshold. As the actual optimal value remains to be identified, we compute the Shannon entropy of the median, H ( m i d ) , at each iteration and compare it with the entropy from the previous iteration.
3.
Narrow the search range by comparing H ( m i d ) and H ( m i d + 1 ) . If the entropy obtained using the current threshold m i d is higher than m i d + 1 , then adjust the search range to between A and m i d . Otherwise, adjust the search range to between m i d + 1 and B.
4.
Reaching the maximum H when A = B . The threshold value that maximizes H ( N ) is the best threshold for nanoparticle segmentation when A is equal to B.
Figure 1d–g illustrate this iterative optimization process by plotting the Shannon entropy value of each iteration and the corresponding segmented images from the selective 1st, 3rd, and final iterations.
In order to demonstrate the effectiveness of the SEO-SDD method, we estimate the accuracies of particle counting with respect to the simulated images with a known number of particles, which can then be served as the ground truth, given as
A c c u r a c y = 100 % − | c o u n t s S E O − S D D − c o u n t s G r o u n d t r u t h | c o u n t s G r o u n d t r u t h
However, for the experimental TEM images, the exact number of particles is to be determined. In these instances, we use expert manual counting as a benchmark for evaluating the segmentation results of the image. As manual counting inherently introduces biases, it is therefore not suitable to be used as “ground truth”. Instead, we calculate the relative error (RE) for each segmentation result with respect to the manual counting result. A smaller RE means smaller differences in particle counting between the segmentation method and manual counting, thereby indicating a greater accuracy of the result. RE here can be defined as
R E = | c o u n t s s e g m e n t a t i o n − c o u n t s M a n u a l | c o u n t s M a n u a l × 100 %

2.2. Unsupervised Machine Learning for Nanoparticle/Aggregate Shape Categorization

After particle segmentation, the edges of the particles (shapes) were identified by applying the Canny edge detection algorithm [70]. This is a widely used method due to its simplicity and robustness [71,72].
Having converted the TEM image data into the particle contour dataset, we then apply an unsupervised ML categorization algorithm that we have previously developed for the particle and aggregate shape categorization [21,67,73] (see Supplementary Materials Section II). As our method is an unsupervised machine learning algorithm (distinct from, e.g., “deep learning” methods), it does not require a priori simulated and experimental training datasets and is therefore generally applicable to any nanoparticle/aggregate materials.
In brief, we apply the hierarchical agglomerative clustering method with the average linkage [67] to categorize the parameterized particle shapes. This clustering method builds a hierarchy of clusters and therefore does not require pre-determined optimum numbers of shape clusters. Thus, the hierarchical clustering method allows for full automation. Finally, the optimum numbers of clusters are determined automatically by applying internal cluster validity indexes (CVIs), with the optimum numbers being the common local extrema of each CVI.
It is worth mentioning that, in the case of aggregates of DNDs, as the attributes for the aggregates are different from the isolated particles, we have developed a different clustering method to categorize the aggregates [73]. Briefly, this method firstly identifies the aggregates by dividing the pre-processed image into grid cells. The choice of the grid cell is optimized to achieve high accuracy in shape categorization by testing a pre-determined range of grid-cell sizes. The grid-cell intensity histograms were initially categorized into three groups (kn) to differentiate the image background (k1), aggregate edges (k2), and aggregate interiors (k3), using the hierarchical agglomerative clustering method with the average linkage. Next, in order to distinguish the morphology of different aggregates, we need to perform a second categorization to include the first and second order nearest neighbors of a given grid cell. The optimum number of clusters of 3 was determined, which were termed as clusters, ropes, and chains. Clusters are defined as aggregates larger than three DNDs in diameter, ropes smaller than three DNDs in width, and chains approximately equal to one DND in width. More details can be seen in Supplementary Materials Section II C.

2.3. Sample Preparation and TEM Imaging

DND dispersions were prepared via a freeze-plunging method for cryo-TEM imaging. Approximately 1 wt% of each sample in deionized water was sonicated for ca. 30 min first and then 4.5 µL of the sample was deposited as a droplet onto the glow-discharged grid (R2/2 Quantifoil copper grids, Jena, Germany) using a Leica grid plunger.
The TEM samples of quantum dot nanoparticles were prepared by drop-casting nanoparticle dispersions onto the holey-carbon-film-coated Cu TEM grids (results in Supplementary Materials Section IV).
For the HeLa cell uptake of DNDs, DNDs were firstly ingested by HeLa cells using the standard process described in detail in the Supplementary Information. For preparing the TEM sample, the cells were postfixed in 1% OsO4 in 0.1 mol Na cacodylate buffer using a BioWave Pro + Microwave Tissue Processor (Ted Pella, Inc., Redding, CA, USA), washed again in Milli Q water, dehydrated with a graded series of ethanol, infiltrated with resin (ProSciTech Pty Ltd., Kirwan, QLD, Australia), and polymerized at 60 °C overnight. Ultrathin sections (70 nm) were collected onto carbon-coated copper TEM grids, which were then post stained with uranyl acetate (2%) and lead citrate (2%). Afterward, images were acquired using a JEOL TEM-1400 (Tokyo, Japan) operating at 120 keV.
The quantum dots were imaged using JEOL F200 (Tokyo, Japan), operated at 200 kV, using the bright-field TEM (BF-TEM) mode. The frozen DND dispersions were imaged using a Talos Arctica TEM (Thermo Fisher Scientific, Waltham, MA, USA) with an acceleration voltage of 200 keV.

3. Results and Discussion

In this section, we firstly validate the effectiveness of the SEO-SDD method by using a simulated nanoparticle image. We then test the method on two challenging types of TEM images: highly noisy cryo-TEM images of frozen DND dispersion in water and BF-TEM images of DND uptake in HeLa cells. We deliberately chose DND particles as an example due to the challenges that they present in terms of small particle size and lower image contrast (compared to the typically demonstrated noble metal particles), as mentioned in the introduction.
Having demonstrated the effectiveness of the SEO-SDD method, we then show how the results can be applied for accurate particles and aggregate shape categorization using unsupervised machine learning methods that we have reported previously [21,67,73] on the examples of quantum dot particles (see Supplementary Materials Section IV) and, again, DND aggregates.

3.1. Effectiveness of SEO-SDD Method

3.1.1. Simulated Image

Figure 2a presents a simulated image containing 1,028 circular-shaped nanoparticles, each distinct in size and intensity levels. The background intensity of this image is intentionally uneven, incorporating several regions of sharp-intensity transitions. Figure 2e shows the distribution of slope differences based on the gradient histogram (yellow curve) and the differential curve of the slope difference distribution ( d s ( n ) d n , blue curve). The zeros of the d s ( n ) d n are candidate threshold values and the red rectangle indicates the selection range for the optimal thresholds. Figure 2b–d display the segmentation results using the SEO-SDD method, histogram thresholding, and dynamic thresholding, respectively. It can be observed that histogram thresholding fails to adequately mitigate the effects of the uneven background, thereby falsely identifying nanoparticles in these regions. Although dynamic thresholding marginally diminishes the impact of the uneven background by accentuating the contrast between the background and the particles, it largely leaves the background unaltered in the segmentation output. In the SEO-SDD results, despite few instances where closely spaced particles overlap, the particle outlines are predominantly extracted, with a good accuracy of 97%. The accuracy, determined using Equation (8), is listed in Table 1. The findings demonstrate that SEO-SDD achieves high accuracy. Subsequently, this methodology was applied to the experimental TEM images.

3.1.2. Experimental TEM Images of DND Aggregates

Figure 3a shows a cryo-TEM image of frozen DND dispersion on a “quantifoil” amorphous carbon film. It can be seen that the DNDs form aggregates in water, with polydispersity and complex aggregate morphologies composed of elongated rope-like and cluster-shaped aggregates, as reported previously [74]. It has been reported that rope aggregates promote better drug biomolecule absorption and can be beneficial for cellular uptake through penetration of the membranes [75]. The cluster-shaped aggregates, on the other hand, can be utilized as nanoporous structures for slower and controlled drug release [76].
In this test experimental TEM image given in Figure 3a, its center shows a distinct ring-shaped uneven background area, which is a circular hole in the “quant foil film”, supporting the DND dispersion. Figure 3b shows an enlargement of the red square region in Figure 3a, highlighting the different background intensity levels between the hole and the carbon film. The segmentation result using the SEO-SDD method is given in Figure 3d in comparison with two other threshold-based methods.
It can be seen that, in the process of removing the uneven background, the histogram threshold method (Figure 3e) loses many nanoparticles as well as sacrifices the particle contours. On the other hand, dynamic thresholding (Figure 3f) appears to deal with such a background better, though it nonetheless misses many pores within the DND aggregates that were successfully identified by the SEO-SDD method (Figure 3d).
In order to quantify the number of nanoparticles in the images, we referred to the machine-learning-based method specifically designed for the classification of DND aggregates [73], briefly mentioned in Section 2.2 and detailed in Supplementary Materials Section II C. The key aspect of counting DND particles when they are aggregates is to utilize the “grid cells” applied to the image. The selection of grid cell size is optimized and is close to the size of DND particles. Thus, by categorizing the intensity histograms within each grid cell (details in Supplementary Materials Section II C), we can then deduce the numbers of DND particles. The manual counting of DND particles was conducted in a similar fashion. The “grid cells” were applied to the image as shown in Figure 3c, and then the DND particles were manually counted by counting the grid cells with intensities covering more than 50% of its area. The relative differences (REs) can then be directly compared among different segmentation methods with respect to the raw image. To reduce the errors inherited in manual counting, we employed the average of multiple manual counts as the reference for comparison.
The inaccuracies in particle segmentation from the threshold-based methods are reflected by means of particle counting relative differences, listed in Table 2. It shows that the SEO-SDD method is most accurate, the dynamic thresholding slightly under-filtering with similar performance, and the histogram thresholding grossly over-filtering.
Figure 4 shows the second example of DNDs uptake by HeLa cells. The BF-TEM image of a slice of negatively stained HeLa cells, given in Figure 4a, demonstrates the highly uneven contrast arising from the complex features of cells as well as the darker contrast from the DND aggregates. As shown in Figure 4b, the red arrows point to the synapse of the cell. Due to the staining, the features of cells have relatively stronger contrast compared to typical TEM images of nanoparticles supported on thin amorphous carbon films. This imposes a greater challenge in segmenting nanoparticles from the background.
It can be seen that, even with complex and irregular cell features as the “background” present in the image, our SEO-SDD method can still successfully segment DND particles (Figure 4d) with very good accuracy, as listed in Table 3. The histogram thresholding method (Figure 4e), while achieving the removal of uneven backgrounds from the cell features by over-filtering, unfortunately sacrifices the correct segmentation of DNDs. Therefore, its RE is relatively high, exceeding 59%.
On the other hand, dynamic thresholding (Figure 4f) experiences difficulties in filtering the background in this case. It incorrectly identified almost all cellular structures in the image as nanoparticles. The RE presented in Table 3 exceeds 790%, indicating that dynamic thresholding in this case is highly unreliable.

3.2. Particle and Aggregate Shape Categorization through Unsupervised Machine Learning

Having demonstrated the effectiveness and robustness of our SEO-SDD particle segmentation method, in this section, we then apply this method in the pre-processing for particle/aggregate contour extraction for the unsupervised machine learning that we have reported previously [21,67,73]. Again, we have chosen DNDs dispersion and cell uptake as examples here. As these two examples are DND aggregates, we have shown the ML analysis of the nanoparticle system of quantum dots in the Supplementary Information Section IV for a better flow of the manuscript.

3.3. Cryo-TEM Image of DND Aggregates in PBS

For the use of DNDs in biomedical applications, their control of the aggregation behavior is critical to their performance. In order to understand the DNDs aggregation state in biomedical-relevant media, here, we show that the SEO-SDD segmentation method can successfully identify and extract DND aggregates for ML morphology categorization. Figure 5a shows a cryo-TEM image of DND dispersion in phosphate buffered saline (PBS), which is a common medium for cell cultures. It can be seen that the image is noisy and the background appears to be “textured”. This is due to the presence of salt in the solution. Under cryogenic conditions, these ions are likely to precipitate from the solution and form small crystals. These crystals produce a strong scattering effect, thereby reducing the clarity and contrast of the image. This is precisely because there are a large number of ions in PBS adsorbed on DNDs, therefore promoting the aggregation of DNDs. As a result, most of the DNDs in the image exist in the form of large aggregates, with small pores (an example of the pore is marked by the yellow arrow in Figure 5a) among the DND particles. This can result in small-contrast differences between the particles and the pores (background), giving difficulties in particle/aggregate segmentation.
Figure 5b shows the DND aggregate segmentation using the SEO-SDD method. We can see that, even in areas where DNDs are highly aggregated, SEO-SDD can still accurately extract the contours. The small pores between the aggregates are also accurately determined. Based on the segmented image given in Figure 5b, the aggregate morphologies were then categorized using our ML method [73]. In this approach, the results of image segmentation undergo erosion and dilation processing before being aligned with three distinct types of aggregate frameworks: clusters, ropes, and chains. Subsequently, the classification outcome depicted in Figure 5c is achieved. This method is described in Section II C of the Supplementary Information. The aggregates were divided into three morphology groups, labelled in blue, green, and red colors. Their corresponding morphology fractions (in terms of areal concentration) and size distributions are plotted in Figure 5d–g, respectively. The aggregate morphology analysis shows that the most of the aggregates, not surprisingly, are cluster-shaped aggregates (blue). The other two types of aggregate shapes, which we termed “ropes” (green) and “chains” (red), have lower fractions and exhibit relatively short lengths (mostly < 60 nm). Such a result is expected as a high-salt-concentration environment can cause nanoparticles, including DNDs, to form dense aggregates.

3.4. DND Uptake in HeLa Cells

For the use of nanoparticles for any biomedical application, studies of the cell uptake of nanoparticles are important in assessing the viabilities before any further development. Moreover, the shape of particles/aggregates has been established to be an important factor for the uptake [7]. Hence, here, we again use the DNDs as an example to show that our SEO-SDD segmentation method is a robust pre-processing method for the analysis of aggregate morphology distribution even in the cells.
Figure 6a shows the BF-TEM image of the chemically fixed and negatively stained HeLa cell with DNDs ingested after 12 h. The DNDs can be seen as present both inside and outside the cell. The resultant DND segmentation is given in Figure 6b. It can be seen that even the background intensity has large variations, most of the background can still be removed equally using the SEO-SDD method. Moreover, the cell structure contrasts, including synapses, microvesicles, cell membranes, and endosomes, marked by a dashed line in Figure 6a, are successfully removed. However, we note that small fractions of cellular structures in the upper left corner and upper center of the image are not correctly removed. This is primarily due to the high contrast of the membrane structures in the cell. In order to further remove the large cell features, we have applied an additional size convexity constraint in the SEO-SDD process. Figure 6c shows the DND aggregate shape categorization analysis with the three major shape groups color-coded and overlayed onto the segmented image using our ML method. To demonstrate the accuracy of our SEO-SDD pre-processing and the ML shape analysis, Figure 6d–g compare the morphology distributions and aggregate size histograms using manual (solid bars) and SEO-SDD segmentation (shaded bars).
Based on the quantitative morphological analysis from the ML method, we find that the outcome of SEO-SDD processing is very close to the ground truth obtained through manual processing, although some fractions of the cell structures are falsely identified as DNDs. Our result shows that the SEO-SDD method can achieve a 96.33% accuracy when processing images containing such complex cellular structures.

4. Conclusions

In this work, we have developed an efficient, robust, and highly automated nanoparticle/aggregate segmentation method from TEM images. The Shannon-entropy-optimized slope difference distribution method, based on image intensity gradient and automatic optimum threshold selection, has been demonstrated to be successful on TEM images with a highly complex background. The SEO-SDD method is more efficient compared to other existing segmentation methods as it does not require a priori training datasets, which are critical for deep-learning-based segmentation methods.
The effectiveness of our method is demonstrated on (cryo-)TEM images of very small (5 nm) detonation nanodiamond particle dispersion and uptake in cells. Such examples represent great challenges in terms of small particle size and relatively low particle contrasts with a noisy and complex background. Our results show that the SEO-SDD method that we have developed provides a highly automated and accurate analysis of size and shape distributions of nanoparticles/aggregates. Moreover, even for TEM images with nanoparticles in cells, our method is sufficiently accurate and robust for extracting and correctly categorizing their shape distributions.
The ability to statistically analyze the size and shape distributions of nanoparticle/aggregate dispersion and cell uptake from (cryo-)TEM images will certainly make a significant contribution to advancing the understanding and applications of nanoparticles for biomedical applications. Moreover, while our methodology was only demonstrated on detonation nanodiamonds and quantum dot nanoparticles, it should be directly applicable to other nanoparticle systems for nanomedicine and biomedical applications, such as liposomes and polymeric micelles.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/nano14141169/s1. In the supplementary materials, we offer comprehensive descriptions of the original SDD method, the ML algorithms used for analyzing the shapes of nanoparticles and aggregates, and the techniques for acquiring TEM images referenced in the main text. Additionally, we provide a concise overview of the Python code employed to implement our approach. Refs. [77,78,79,80,81,82,83,84,85,86] are cited in Supplementary Materials.

Author Contributions

Conceptualization, S.L.Y.C. and L.Z; methodology, L.Z., I.C.K., and H.W.; software, L.Z. and H.W.; validation, L.Z. and H.W.; formal analysis, L.Z. and H.W.; resources, S.L.Y.C. data curation, L.Z., I.C.K., and H.W.; writing—original draft preparation, L.Z.; writing—review and editing, S.L.Y.C. and L.Z.; visualization, L.Z.; supervision, S.L.Y.C.; project administration, S.L.Y.C.; funding acquisition, S.L.Y.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by ARC under project IC210100056.

Data Availability Statement

Data are contained within the article and Supplementary Materials.

Acknowledgments

All authors acknowledge the use of facilities supported by Microscopy Australia at the Electron Microscope Unit at UNSW. S.L.Y.C. acknowledges the support by ARC under project IC210100056.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Raffa, V.; Vittorio, O.; Riggio, C.; Cuschieri, A. Progress in nanotechnology for healthcare. Minim. Invasive Ther. Allied Technol. 2010, 19, 127–135. [Google Scholar] [CrossRef] [PubMed]
  2. Feng, H.P.; Tang, L.; Zeng, G.m.; Zhou, Y.; Deng, Y.C.; Ren, X.; Song, B.; Liang, C.; Wei, M.y.; Yu, J.F. Core-shell nanomaterials: Applications in energy storage and conversion. Adv. Colloid Interface Sci. 2019, 267, 26–46. [Google Scholar] [CrossRef]
  3. Xie, C.; Niu, Z.; Kim, D.; Li, M.; Yang, P. Surface and interface control in nanoparticle catalysis. Chem. Rev. 2019, 120, 1184–1249. [Google Scholar] [CrossRef] [PubMed]
  4. Li, Y.; Liu, Q.; Shen, W. Morphology-dependent nanocatalysis: Metal particles. Dalton Trans. 2011, 40, 5811–5826. [Google Scholar] [CrossRef] [PubMed]
  5. Khot, L.R.; Sankaran, S.; Maja, J.M.; Ehsani, R.; Schuster, E.W. Applications of nanomaterials in agricultural production and crop protection: A review. Crop. Prot. 2012, 35, 64–70. [Google Scholar] [CrossRef]
  6. Das, S.; Sen, B.; Debnath, N. Recent trends in nanomaterials applications in environmental monitoring and remediation. Environ. Sci. Pollut. Res. 2015, 22, 18333–18344. [Google Scholar] [CrossRef] [PubMed]
  7. Wang, W.; Gaus, K.; Tilley, R.D.; Gooding, J.J. The impact of nanoparticle shape on cellular internalisation and transport: What do the different analysis methods tell us? Mater. Horizons 2019, 6, 1538. [Google Scholar] [CrossRef]
  8. Garcia-Alvarez, R.; Hadjidemetriou, M.; Sanchez-Iglesias, A.; Liz-Marzan, L.M.; Kostarelos, K. In vivo formation of protein corona on gold nanoparticles. The effect of their size and shape. Nanoscale 2018, 10, 1256. [Google Scholar] [CrossRef]
  9. Carnovale, C.; Bryant, G.; Shukla, R.; Bansal, V. Identifying Trends in Gold Nanoparticle Toxicity and Uptake: Size, Shape, Capping Ligand, and Biological Corona. ACS Omega 2019, 4, 242. [Google Scholar] [CrossRef]
  10. Yang, W.; Wang, L.; Mettenbrink, E.M.; DeAngelis, P.L.; Wilhelm, S. Nanoparticle Toxicology. Annu. Rev. 2021, 61, 269. [Google Scholar] [CrossRef]
  11. Han, X.; Xu, K.; Taratula, O.; Farsad, K. Applications of Nanoparticles in Biomedical Imaging. Nanoscale 2019, 11, 799–819. [Google Scholar] [CrossRef] [PubMed]
  12. McNamara, K.; Tofail, S.A.M. Nanoparticles in biomedical applications. Adv. Phys. X 2017, 2, 54–88. [Google Scholar] [CrossRef]
  13. Suri, S.S.; Fenniri, H.; Singh, B. Nanotechnology-based drug delivery systems. J. Occup. Med. Toxicol. 2007, 2, 16. [Google Scholar] [CrossRef] [PubMed]
  14. Danaei, M.; Dehghankhold, M.; Ataei, S.; Hasanzadeh Davarani, F.; Javanmard, R.; Dokhani, A.; Khorasani, S.; Mozafari, M.R. Impact of Particle Size and Polydispersity Index on the Clinical Applications of Lipidic Nanocarrier Systems. Pharmaceutics 2018, 10, 57. [Google Scholar] [CrossRef] [PubMed]
  15. Wu, Z.; Yang, S.; Wu, W. Shape control of inorganic nanoparticles from solution. Nanoscale 2016, 8, 1237–1259. [Google Scholar] [CrossRef] [PubMed]
  16. Grzelczak, M.; Pérez-Juste, J.; Mulvaney, P.; Liz-Marzán, L.M. Shape control in gold nanoparticle synthesis. Chem. Soc. Rev. 2008, 37, 1783–1791. [Google Scholar] [CrossRef] [PubMed]
  17. An, K.; Somorjai, G.A. Size and shape control of metal nanoparticles for reaction selectivity in catalysis. ChemCatChem 2012, 4, 1512–1524. [Google Scholar] [CrossRef]
  18. Chang, L.Y.; Cervera Gontard, L.; Dunin-Borkowski, R.E.; Ozkaya, D. Resolving the Structure of Active Sites on Platinum Catalytic Nanoparticles. Nano Lett. 2010, 10, 3073–3076. [Google Scholar] [CrossRef] [PubMed]
  19. Cervera Gontard, L.; Chang, L.Y.; Hetherington, C.J.D.; Kirkland, A.I.; Ozkaya, D.; Dunin-Borkowski, R.E. Aberration-Corrected Imaging of Active Sites on Industrial Catalyst Nanoparticles. Angew. Chem. Int. Ed. 2007, 46, 3683–3685. [Google Scholar] [CrossRef]
  20. Ealia, S.A.M.; Saravanakumar, M.P. A review on the classification, characterisation, synthesis of nanoparticles and their application. In Proceedings of the IOP Conference Series: Materials Science and Engineering, Vellore, India, 2–3 May 2017; IOP Publishing: Bristol, UK, 2017; Volume 263, p. 032019. [Google Scholar]
  21. Wen, H.; Xu, X.; Cheong, S.; Lo, S.C.; Chen, J.H.; Chang, S.L.; Dwyer, C. Metrology of convex-shaped nanoparticles via soft classification machine learning of TEM images. Nanoscale Adv. 2021, 3, 6956–6964. [Google Scholar] [CrossRef]
  22. Bangare, S.L.; Dubal, A.; Bangare, P.S.; Patil, S. Reviewing Otsu’s method for image thresholding. Int. J. Appl. Eng. Res. 2015, 10, 21777–21783. [Google Scholar] [CrossRef]
  23. Senthilkumaran, N.; Vaithegi, S. Image segmentation by using thresholding techniques for medical images. Comput. Sci. Eng. Int. J. 2016, 6, 1–13. [Google Scholar]
  24. Senthilkumaran, N.; Kirubakaran, C. Efficient implementation of Niblack thresholding for MRI brain image segmentation. Int. J. Comput. Sci. Inf. Technol. 2014, 5, 2174–2176. [Google Scholar]
  25. Brink, A.D.; Pendock, N.E. Minimum cross-entropy threshold selection. Pattern Recognit. 1996, 29, 179–188. [Google Scholar] [CrossRef]
  26. Chan, F.H.; Lam, F.K.; Zhu, H. Adaptive thresholding by variational method. IEEE Trans. Image Process. 1998, 7, 468–473. [Google Scholar] [CrossRef] [PubMed]
  27. Pu, S.; Gong, C.; Robertson, A.W. Liquid cell transmission electron microscopy and its applications. R. Soc. Open Sci. 2020, 7, 191204. [Google Scholar] [CrossRef]
  28. Liu, K.L.; Wu, C.C.; Huang, Y.J.; Peng, H.L.; Chang, H.Y.; Chang, P.; Hsu, L.; Yew, T.R. Novel microchip for in situ TEM imaging of living organisms and bio-reactions in aqueous conditions. Lab Chip 2008, 8, 1915–1921. [Google Scholar] [CrossRef] [PubMed]
  29. Srnová-Šloufová, I.; Lednický, F.; Gemperle, A.; Gemperlová, J. Core—Shell (Ag) Au bimetallic nanoparticles: Analysis of transmission electron microscopy images. Langmuir 2000, 16, 9928–9935. [Google Scholar] [CrossRef]
  30. Krishna, K.; Murty, M.N. Genetic K-means algorithm. IEEE Trans. Syst. Man, Cybern. Part B 1999, 29, 433–439. [Google Scholar] [CrossRef]
  31. Likas, A.; Vlassis, N.; Verbeek, J.J. The global k-means clustering algorithm. Pattern Recognit. 2003, 36, 451–461. [Google Scholar] [CrossRef]
  32. Na, S.; Xumin, L.; Yong, G. Research on k-means clustering algorithm: An improved k-means clustering algorithm. In Proceedings of the 2010 Third International Symposium on Intelligent Information Technology and Security Informatics, Jinggangshan, China, 2–4 April 2010; pp. 63–67. [Google Scholar]
  33. Comaniciu, D.; Meer, P. Mean shift analysis and applications. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Corfu, Greece, 20–25 September 1999; Volume 2, pp. 1197–1203. [Google Scholar]
  34. Comaniciu, D.; Meer, P. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 2002, 24, 603–619. [Google Scholar] [CrossRef]
  35. Derpanis, K.G. Mean shift clustering. Lecture Notes 2005, 32, 1–4. [Google Scholar]
  36. Carreira-Perpinán, M.A. A review of mean-shift algorithms for clustering. arXiv 2015, arXiv:1503.00687. [Google Scholar]
  37. Wu, K.L.; Yang, M.S. Mean shift-based clustering. Pattern Recognit. 2007, 40, 3035–3052. [Google Scholar] [CrossRef]
  38. Xu, N.; Price, B.; Cohen, S.; Yang, J.; Huang, T. Deep grabcut for object selection. arXiv 2017, arXiv:1707.00243. [Google Scholar]
  39. Han, S.; Tao, W.; Wang, D.; Tai, X.C.; Wu, X. Image segmentation based on GrabCut framework integrating multiscale nonlinear structure tensor. IEEE Trans. Image Process. 2009, 18, 2289–2302. [Google Scholar] [PubMed]
  40. Wang, Z.; Lv, Y.; Wu, R.; Zhang, Y. Review of GrabCut in Image Processing. Mathematics 2023, 11, 1965. [Google Scholar] [CrossRef]
  41. Noval, J.J.S.; Gómez-Merchán, R.; Leñero-Bardallo, J.A.; Gontard, L.C. TEMAS: A Flexible Non-AI Algorithm for Metrology of Single-Core and Core-Shell Nanoparticles from TEM Images (Part. Part. Syst. Charact. 2/2023). Part. Part. Syst. Charact. 2023, 40, 2370003. [Google Scholar] [CrossRef]
  42. Krüger, A.; Kataoka, F.; Ozawa, M.; Fujino, T.; Suzuki, Y.; Aleksenskii, A.E.; Vul’, A.Y.; Ōsawa, E. Unusually tight aggregation in detonation nanodiamond: Identification and disintegration. Carbon 2005, 43, 1722–1730. [Google Scholar] [CrossRef]
  43. Ho, D.; Wang, C.H.K.; Chow, E.K.H. Nanodiamonds: The intersection of nanotechnology, drug development, and personalized medicine. Sci. Adv. 2015, 1, e1500439. [Google Scholar] [CrossRef]
  44. Chauhan, S.; Jain, N.; Nagaich, U. Nanodiamonds with powerful ability for drug delivery and biomedical applications: Recent updates on in vivo study and patents. J. Pharm. Anal. 2020, 10, 1–12. [Google Scholar] [CrossRef] [PubMed]
  45. Giammarco, J.; Mochalin, V.N.; Haeckel, J.; Gogotsi, Y. The adsorption of tetracycline and vancomycin onto nanodiamond with controlled release. J. Colloid Interface Sci. 2016, 468, 253–261. [Google Scholar] [CrossRef] [PubMed]
  46. Benson, V.; Amini, A. Why nanodiamond carriers manage to overcome drug resistance in cancer. Cancer Drug Resist. 2020, 3, 854–866. [Google Scholar] [CrossRef] [PubMed]
  47. Bertrand, J.R.; Pioche-Durieu, C.; Ayala, J.; Petit, T.; Girard, H.A.; Malvy, C.P.; Le Cam, E.; Treussart, F.; Arnault, J.C. Plasma hydrogenated cationic detonation nanodiamonds efficiently deliver to human cells in culture functional siRNA targeting the Ewing sarcoma junction oncogene. Biomaterials 2015, 45, 93–98. [Google Scholar] [CrossRef] [PubMed]
  48. Leung, H.M.; Chan, M.S.; Liu, L.S.; Wong, S.W.; Lo, T.W.; Lau, C.H.; Tin, C.; Lo, P.K. Dual-Function, Cationic, Peptide-Coated Nanodiamond Systems: Facilitating Nuclear-Targeting Delivery for Enhanced Gene Therapy Applications. ACS Sustain. Chem. Eng. 2018, 6, 9671–9681. [Google Scholar] [CrossRef]
  49. Zhang, X.Q.; Chen, M.; Lam, R.; Xu, X.; Ōsawa, E.; Ho, D. Polymer-Functionalized Nanodiamond Platforms as Vehicles for Gene Delivery. ACS Nano 2009, 3, 2609–2616. [Google Scholar] [CrossRef]
  50. Fox, K.; Ratwatte, R.; Booth, M.A.; Tran, H.M.; Tran, P.A. High Nanodiamond Content-PCL Composite for Tissue Engineering Scaffolds. Nanomaterials 2020, 10, 948. [Google Scholar] [CrossRef] [PubMed]
  51. Nunes-Pereira, J.; Silva, A.; Ribeiro, C.; Carabineiro, S.; Buijnsters, J.; Lanceros-Méndez, S. Nanodiamonds/poly(vinylidene fluoride) composites for tissue engineering applications. Compos. Part B Eng. 2017, 111, 37–44. [Google Scholar] [CrossRef]
  52. Wang, X.; Li, J.; Ha, H.D.; Dahl, J.C.; Ondry, J.C.; Moreno-Hernandez, I.; Head-Gordon, T.; Alivisatos, A.P. AutoDetect-mNP: An unsupervised machine learning algorithm for automated analysis of transmission electron microscope images of metal nanoparticles. JACS Au 2021, 1, 316–327. [Google Scholar] [CrossRef]
  53. Liang, Z.; Nie, Z.; An, A.; Gong, J.; Wang, X. A particle shape extraction and evaluation method using a deep convolutional neural network and digital image processing. Powder Technol. 2019, 353, 156–170. [Google Scholar] [CrossRef]
  54. Lee, B.; Yoon, S.; Lee, J.W.; Kim, Y.; Chang, J.; Yun, J.; Ro, J.C.; Lee, J.S.; Lee, J.H. Statistical Characterization of the Morphologies of Nanoparticles through Machine Learning Based Electron Microscopy Image Analysis. ACS Nano 2020, 14, 17125–17133. [Google Scholar] [CrossRef] [PubMed]
  55. Buades, A.; Coll, B.; Morel, J.M. Non-local means denoising. Image Process. Line 2011, 1, 208–212. [Google Scholar] [CrossRef]
  56. Kushwaha, H.S.; Tanwar, S.; Rathore, K.; Srivastava, S. De-noising filters for TEM (transmission electron microscopy) image of nanomaterials. In Proceedings of the 2012 Second International Conference on Advanced Computing & Communication Technologies, Rohtak, India, 7–8 January 2012; pp. 276–281. [Google Scholar]
  57. Russo, F. A method for estimation and filtering of Gaussian noise in images. IEEE Trans. Instrum. Meas. 2003, 52, 1148–1154. [Google Scholar] [CrossRef]
  58. Azzeh, J.; Zahran, B.; Alqadi, Z. Salt and pepper noise: Effects and removal. JOIV Int. J. Inform. Vis. 2018, 2, 252–256. [Google Scholar] [CrossRef]
  59. Boyat, A.K.; Joshi, B.K. A review paper: Noise models in digital image processing. arXiv 2015, arXiv:1505.03489. [Google Scholar] [CrossRef]
  60. Danino, D. Cryo-TEM of soft molecular assemblies. Curr. Opin. Colloid Interface Sci. 2012, 17, 316–329. [Google Scholar] [CrossRef]
  61. Mevenkamp, N.; Yankovich, A.B.; Voyles, P.M.; Berkels, B. Non-local Means for Scanning Transmission Electron Microscopy Images and Poisson Noise based on Adaptive Periodic Similarity Search and Patch Regularization. In Proceedings of the VMV, Darmstadt, Germany, 8–10 October 2014; pp. 63–70. [Google Scholar]
  62. Mevenkamp, N.; MacArthur, K.E.; Tileli, V.; Ebert, P.; Allen, L.J.; Berkels, B.; Duchamp, M. Multi-modal and multi-scale non-local means method to analyze spectroscopic datasets. Ultramicroscopy 2020, 209, 112877. [Google Scholar] [CrossRef] [PubMed]
  63. Wang, Z. Determining the clustering centers by slope difference distribution. IEEE Access 2017, 5, 10995–11002. [Google Scholar] [CrossRef]
  64. Wang, Z. A new approach for segmentation and quantification of cells or nanoparticles. IEEE Trans. Ind. Inform. 2016, 12, 962–971. [Google Scholar] [CrossRef]
  65. Kapur, J.N.; Sahoo, P.K.; Wong, A.K. A new method for gray-level picture thresholding using the entropy of the histogram. Comput. Vision, Graph. Image Process. 1985, 29, 273–285. [Google Scholar] [CrossRef]
  66. Davies, E.R. Machine Vision: Theory, Algorithms, Practicalities (Signal Processing and its Applications); Elsevier: Amsterdam, The Netherlands, 1990; Volume 2. [Google Scholar]
  67. Wen, H.; Luna-Romera, J.M.; Riquelme, J.C.; Dwyer, C.; Chang, S.L. Statistically representative metrology of nanoparticles via unsupervised machine learning of TEM Images. Nanomaterials 2021, 11, 2706. [Google Scholar] [CrossRef] [PubMed]
  68. Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef]
  69. Williams, L.F., Jr. A modification to the half-interval search (binary search) method. In Proceedings of the 14th Annual Southeast Regional Conference, Birmingham, AL, USA, 22–24 April 1976; pp. 95–101. [Google Scholar]
  70. Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 1986, PAMI-8, 679–698. [Google Scholar] [CrossRef]
  71. Sharifi, M.; Fathy, M.; Mahmoudi, M.T. A classified and comparative study of edge detection algorithms. In Proceedings of the International Conference on Information Technology: Coding and Computing, Las Vegas, NV, USA, 8–10 April 2002; pp. 117–120. [Google Scholar]
  72. Kalbasi, M.; Nikmehr, H. Noise-Robust, Reconfigurable Canny Edge Detection and its Hardware Realization. IEEE Access 2020, 8, 39934–39945. [Google Scholar] [CrossRef]
  73. Kuschnerus, I.C.; Wen, H.; Ruan, J.; Zeng, X.; Su, C.J.; Jeng, U.S.; Opletal, G.; Barnard, A.S.; Liu, M.; Nishikawa, M.; et al. Complex Dispersion of Detonation Nanodiamond Revealed by Machine Learning Assisted Cryo-TEM and Coarse-Grained Molecular Dynamics Simulations. ACS Nanosci. Au 2023, 3, 211–221. [Google Scholar] [CrossRef] [PubMed]
  74. Chang, S.L.; Reineck, P.; Williams, D.; Bryant, G.; Opletal, G.; El-Demrdash, S.A.; Chiu, P.L.; Ōsawa, E.; Barnard, A.S.; Dwyer, C. Dynamic self-assembly of detonation nanodiamond in water. Nanoscale 2020, 12, 5363–5367. [Google Scholar] [CrossRef]
  75. Schrand, A.M.; Hens, S.A.C.; Shenderova, O.A. Nanodiamond particles: Properties and perspectives for bioapplications. Crit. Rev. Solid State Mater. Sci. 2009, 34, 18–74. [Google Scholar] [CrossRef]
  76. Ho, D. Nanomaterial-based therapy: A new generation of cancer treatment. Therapy 2009, 6, 99–104. [Google Scholar] [CrossRef]
  77. Wang, D.; Zhou, S. Color image recognition method based on the prewitt operator. In Proceedings of the 2008 International Conference on Computer Science and Software Engineering, Wuhan, China, 12–14 December 2008; Volume 6, pp. 170–173. [Google Scholar]
  78. Vairalkar, M.K.; Nimbhorkar, S.U. Edge detection of images using sobel operator. Int. J. Emerg. Technol. Adv. Eng. 2012, 2, 291–293. [Google Scholar]
  79. Cherri, A.K.; Karim, M.A. Optical symbolic substitution: Edge detection using prewitt, sobel, and roberts operators. Appl. Opt. 1989, 28, 4644–4648. [Google Scholar] [CrossRef]
  80. Chaple, G.N.; Daruwala, R.D.; Gofane, M.S. Comparisions of robert, prewitt, sobel operator based edge detection methods for real time uses on fpga. In Proceedings of the 2015 International Conference on Technologies for Sustainable Development (ICTSD), Mumbai, India, 4–6 February 2015; pp. 1–4. [Google Scholar]
  81. Hu, M.-K. Visual pattern recognition by moment invariants. IRE Trans. Information Theory 1962, 8, 179–187. [Google Scholar]
  82. Arbelaitz, O.; Gurrutxaga, I.; Muguerza, J.; P´erez, J.M.; Perona, I. An extensive comparative study of cluster validity indices. Pattern Recognit. 2013, 46, 243–256. [Google Scholar] [CrossRef]
  83. Rousseeuw, P.J. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 1987, 20, 53–65. [Google Scholar] [CrossRef]
  84. Davies, D.L.; Bouldin, D.W. A cluster separation measure. IEEE Trans. On Pattern Anal. Mach. Intell. 1979, PAMI-1, 224–227. [Google Scholar] [CrossRef]
  85. Cali´nski, T.; Harabasz, J. Communications in statistics—theory and methods. Commun. Stat. 1974, 3, 1–27. [Google Scholar]
  86. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
Nanomaterials 14 01169 g001
Figure 1. Process of Shannon-entropy-optimized slope difference distribution (SEO-SDD) method for nanoparticle segmentation from TEM images. (a) Raw TEM image with its corresponding intensity histogram; (b) gradient image with the corresponding normalized gradient histogram; (c) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red crosses denote candidate threshold values, and the red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range; (d) plot of the Shannon entropy values calculated in the iterative optimization process; (e–g) segmentation result in the 1st, 3rd, and 5th iteration, giving the Shannon entropy values of 0.30282, 0.99924, and 0.99999, respectively.
Figure 1. Process of Shannon-entropy-optimized slope difference distribution (SEO-SDD) method for nanoparticle segmentation from TEM images. (a) Raw TEM image with its corresponding intensity histogram; (b) gradient image with the corresponding normalized gradient histogram; (c) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red crosses denote candidate threshold values, and the red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range; (d) plot of the Shannon entropy values calculated in the iterative optimization process; (e–g) segmentation result in the 1st, 3rd, and 5th iteration, giving the Shannon entropy values of 0.30282, 0.99924, and 0.99999, respectively.
Nanomaterials 14 01169 g001
Nanomaterials 14 01169 g002
Figure 2. Comparison of SEO-SDD segmentation and other thresholding-based segmentation methods on a simulated image including 1028 particles with uneven background. (a) Simulated image; (b) SEO-SDD segmentation result; (c) histogram thresholding result; (d) dynamic thresholding result; (e) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range.
Figure 2. Comparison of SEO-SDD segmentation and other thresholding-based segmentation methods on a simulated image including 1028 particles with uneven background. (a) Simulated image; (b) SEO-SDD segmentation result; (c) histogram thresholding result; (d) dynamic thresholding result; (e) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range.
Nanomaterials 14 01169 g002
Nanomaterials 14 01169 g003
Figure 3. Comparisons of SEO-SDD segmentation and other thresholding-based segmentation methods on a cryo-TEM image of DND suspension supported on a “quanti-foil” amorphous carbon film. (a) Raw image; (b) magnification of the red area in (a). The boundary of hole and carbon film are indicated by yellow dashed lines; (b,c) covered with red grid lines for manual counting; (d) SEO-SDD segmentation result; (e) histogram thresholding result; (f) dynamic thresholding result; (g) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range. The scale bar in (a) is 500 nm, and those in (b–f) are all 50 nm.
Figure 3. Comparisons of SEO-SDD segmentation and other thresholding-based segmentation methods on a cryo-TEM image of DND suspension supported on a “quanti-foil” amorphous carbon film. (a) Raw image; (b) magnification of the red area in (a). The boundary of hole and carbon film are indicated by yellow dashed lines; (b,c) covered with red grid lines for manual counting; (d) SEO-SDD segmentation result; (e) histogram thresholding result; (f) dynamic thresholding result; (g) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range. The scale bar in (a) is 500 nm, and those in (b–f) are all 50 nm.
Nanomaterials 14 01169 g003
Nanomaterials 14 01169 g004
Figure 4. Comparison of SEO-SDD segmentation and other thresholding-based segmentation methods on a BF-TEM image of DNDs taken up by HeLa cells. (a) Raw image; (b) magnification of the red area in (a). Cellular structure is indicated by red arrows; (b,c) covered with red gridlines for manual counting; (d) SEO-SDD segmentation result; (e) histogram thresholding result; (f) dynamic thresholding result; (g) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range. The scale bar in (a) is 500 nm, and those in (b–f) are all 100 nm.
Figure 4. Comparison of SEO-SDD segmentation and other thresholding-based segmentation methods on a BF-TEM image of DNDs taken up by HeLa cells. (a) Raw image; (b) magnification of the red area in (a). Cellular structure is indicated by red arrows; (b,c) covered with red gridlines for manual counting; (d) SEO-SDD segmentation result; (e) histogram thresholding result; (f) dynamic thresholding result; (g) process of threshold range selection. The yellow curve ( s ( n ) ) represents the distribution of slope differences obtained based on the gradient histogram, while the blue curve ( d s ( n ) d n ) represents the differential curve of the slope difference distribution. Red rectangle indicates the selection range for the optimal thresholds. A and B are respectively the upper and lower limits of the threshold selection range. The scale bar in (a) is 500 nm, and those in (b–f) are all 100 nm.
Nanomaterials 14 01169 g004
Nanomaterials 14 01169 g005
Figure 5. ML analysis of DND aggregate morphology categorization from a cryo-TEM image of DNDs in PBS. The small pore within DND particles in (a–c) is indicated by red box and yellow arrow. (a) Raw cryo-TEM image with the yellow arrow indicating a pore among the aggregates; (b) SEO-SDD segmented binary image; (c) aggregate morphology categorized groups overlay on (b), with cluster shape group shown in blue, rope shape group in green, and chain shape group in red; (d) morphology fraction of the three aggregate morphology groups; (e) diameter distribution of the cluster-shaped aggregates; (f) length distribution of rope-shaped aggregates; and (g) length distribution of chain-shaped aggregates.
Figure 5. ML analysis of DND aggregate morphology categorization from a cryo-TEM image of DNDs in PBS. The small pore within DND particles in (a–c) is indicated by red box and yellow arrow. (a) Raw cryo-TEM image with the yellow arrow indicating a pore among the aggregates; (b) SEO-SDD segmented binary image; (c) aggregate morphology categorized groups overlay on (b), with cluster shape group shown in blue, rope shape group in green, and chain shape group in red; (d) morphology fraction of the three aggregate morphology groups; (e) diameter distribution of the cluster-shaped aggregates; (f) length distribution of rope-shaped aggregates; and (g) length distribution of chain-shaped aggregates.
Nanomaterials 14 01169 g005
Nanomaterials 14 01169 g006
Figure 6. ML analysis of DND aggregate morphology categorization from a BF-TEM image of DNDs uptake in HeLa cells. (a) Raw BF-TEM image with the colored dotted lines indicating the cell internal features; (b) SEO-SDD segmented binary image; (c) aggregate morphology categorized groups overlayed onto (b); (d) fraction of the three aggregate morphology groups, with the manually segmented results shown in solid bars and the SEO-SDD results in shaded bars; (e) diameter distribution of the cluster-shaped aggregates; (f) length distribution of rope-shaped aggregates; and (g) length distribution of chain-shaped aggregates.
Figure 6. ML analysis of DND aggregate morphology categorization from a BF-TEM image of DNDs uptake in HeLa cells. (a) Raw BF-TEM image with the colored dotted lines indicating the cell internal features; (b) SEO-SDD segmented binary image; (c) aggregate morphology categorized groups overlayed onto (b); (d) fraction of the three aggregate morphology groups, with the manually segmented results shown in solid bars and the SEO-SDD results in shaded bars; (e) diameter distribution of the cluster-shaped aggregates; (f) length distribution of rope-shaped aggregates; and (g) length distribution of chain-shaped aggregates.
Nanomaterials 14 01169 g006
Table 1. Comparisons of particle counting accuracies based on particle segmentation from the simulated image using the SEO-SDD, dynamic and histogram thresholding methods.
Table 1. Comparisons of particle counting accuracies based on particle segmentation from the simulated image using the SEO-SDD, dynamic and histogram thresholding methods.
MethodsNumber of ParticlesAccuracy (%)
Ground truth1028100
SEO-SDD99796.98
Dynamic threshold91589.00
Histogram threshold74071.98
Table 2. Comparisons of the DND particle counting relative differences (with respect to manual counting) based on particle segmentation from the cryo-TEM image of DND dispersion using SEO-SDD, dynamic, and histogram thresholding methods.
Table 2. Comparisons of the DND particle counting relative differences (with respect to manual counting) based on particle segmentation from the cryo-TEM image of DND dispersion using SEO-SDD, dynamic, and histogram thresholding methods.
MethodsNumberRE (%)
Manual counting20,3620
SEO-SDD19,9342.10
Dynamic threshold20,9062.67
Histogram threshold12,36039.30
Table 3. Comparisons of the DND particle counting relative differences, REs, (with respect to manual counting) from the BF-TEM image of HeLa cell uptaking DNDs, processed using SEO-SDD, dynamic, and histogram thresholding methods.
Table 3. Comparisons of the DND particle counting relative differences, REs, (with respect to manual counting) from the BF-TEM image of HeLa cell uptaking DNDs, processed using SEO-SDD, dynamic, and histogram thresholding methods.
MethodsNumberRE (%)
Manual counting29320
SEO-SDD28163.96
Dynamic threshold26,170792.56
Histogram threshold119159.38
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Zhou, L.; Wen, H.; Kuschnerus, I.C.; Chang, S.L.Y. Efficientand Robust Automated Segmentation of Nanoparticles and Aggregates from Transmission Electron Microscopy Images with Highly Complex Backgrounds. Nanomaterials 2024, 14, 1169. https://doi.org/10.3390/nano14141169

AMA Style

Zhou L, Wen H, Kuschnerus IC, Chang SLY. Efficientand Robust Automated Segmentation of Nanoparticles and Aggregates from Transmission Electron Microscopy Images with Highly Complex Backgrounds. Nanomaterials. 2024; 14(14):1169. https://doi.org/10.3390/nano14141169

Chicago/Turabian Style

Zhou, Lishi, Haotian Wen, Inga C. Kuschnerus, and Shery L. Y. Chang. 2024. "Efficientand Robust Automated Segmentation of Nanoparticles and Aggregates from Transmission Electron Microscopy Images with Highly Complex Backgrounds" Nanomaterials 14, no. 14: 1169. https://doi.org/10.3390/nano14141169

APA Style

Zhou, L., Wen, H., Kuschnerus, I. C., & Chang, S. L. Y. (2024). Efficientand Robust Automated Segmentation of Nanoparticles and Aggregates from Transmission Electron Microscopy Images with Highly Complex Backgrounds. Nanomaterials, 14(14), 1169. https://doi.org/10.3390/nano14141169

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop