Introduction

Nanoparticle characterization is rapidly becoming a “big data” problem1,2,3,4,5,6,7,8,9, where the volume of data threatens to surpass the capabilities of analytical approaches. Recent developments have made it possible to synthesize vast arrays with millions of distinct nanoparticles on a chip10, known as combinatorial megalibraries. Manual analysis can no longer fully capture the abundance of information contained in these arrays, leading to a bottleneck in processing. To meet the challenge posed by high throughput characterization, researchers are increasingly turning to automated tools and analysis frameworks. Recently, machine learning (ML), deep learning (DL), and artificial intelligence (AI) have become valuable tools in materials science11,12,13,14,15,16,17,18,19,20. Researchers are compiling databases and libraries of material properties21,22 and developing analysis frameworks that leverage the power of AI23,24, which are increasingly driving materials design and discovery25,26,27,28.

This paper details the second phase of a project to automate key steps in nanoparticle characterization. The first step consists of a quality assessment to determine if a particle meets the criteria for further analysis. In previous work29,30,31, we built binary classification machine learning models for this task. Once a particle has been selected for in-depth study, researchers must choose where within the particle to gather data. In this work, we automate this step with a custom pipeline that returns acquisition coordinates for STEM-based analysis, such as Four-Dimensional Scanning Transmission Electron Microscopy (4D-STEM)32, Energy Dispersive X-ray Spectroscopy (EDS)33, and Electron Energy Loss Spectroscopy (EELS)34. Our process incorporates techniques from computer vision and unsupervised learning and was carefully designed and tested with input from microscopy and materials science experts on our team. We are working to deploy these automated tools in the laboratory to accelerate 4D-STEM processing.

Challenges and objectives

Current characterization workflows face significant efficiency limitations. Standard approaches collect high-resolution information using an evenly spaced grid of 128 × 128 points, dividing each image into 16,384 smaller square regions for analysis. This grid-based approach presents several challenges: it produces regions of identical size regardless of feature complexity, fails to distinguish between particle and background areas, and leads to inefficient use of both time and data storage resources. An optimized preprocessing method should, therefore, accomplish three key objectives: (1) background identification and removal; (2) automated image segmentation based on key features; and (3) appropriate placement of acquisition boxes. Figure 1 illustrates this desired workflow.

Fig. 1
figure 1

(a) The original parent image; (b) a manually created image representing the desired outcome after the background removal step; (c) a manually created image representing the ideal segmentation regions for this image; (d) hand-drawn acquisition boxes overlaid on the parent image reflecting the desired coordinates for child images. Acquisition box size corresponds to the locations of features of interest: areas of varying intensity are captured with smaller boxes. Scale bars: 20 nm.

Full size image

For clarity, we define key terms used throughout this paper. We refer to the original microscopy image as the "parent image," and the smaller derivative images as "child images." The term “acquisition box” describes the set of square coordinates that defines each child image’s position within a parent frame (Fig. 1).

We present a method that accelerates the characterization workflow by more than 25 times on average, validated using a unique dataset of 964 grayscale nanoparticle images. The pipeline is specifically designed for high-throughput acquisition to enable precise and efficient analysis of material properties at the nanoscale.

Related work

Modern image analysis often involves techniques from ML and computer vision. When training an ML model, researchers sometimes perform feature engineering steps to provide the model with additional information. These steps may include transformations, such as binning numerical data35, or augmentations, such as adding information about visual features in images36. Deep learning is a subset of machine learning that relies on many-layer (“deep”) neural networks and large amounts of training data. Deep learning can be advantageous because it can often learn patterns using simpler representations than those required for machine learning37. While DL has demonstrated considerable success in a variety of fields, including materials science7, computer vision38, medicine39, physics40, and others, the requirement for large amounts of training data can present a challenge.

Image segmentation involves partitioning images into regions of interest. Various ML and DL approaches have proved successful at this task, from convolutional neural networks (CNNs)41 to specialized architectures like U-net42; a survey of DL techniques for image segmentation appears in43. Specialized training and testing datasets to support image segmentation have been developed, such as Cityscapes44, SYNTHIA45, COCO46, ADE20K47, and the LIVECell dataset48.

In microscopy and materials science, ML and DL methods have emerged as crucial tools for materials characterization3,4. Segmentation models have been used to detect defects in steel using STEM images49 and characterize X-ray computed tomography data of sintered alloys50. Notable advances include the development of multiple output convolutional networks for multi-particle boundary detection51, comprehensive architecture comparisons for TEM image segmentation52, and genetic algorithm-based approaches for rapid morphological analysis53. These developments demonstrate the field’s progression toward automated analysis, though most focus on post-acquisition processing rather than acquisition optimization, which is the focus of this current work.

Our initial approach considered traditional ML or DL image segmentation techniques similar to those in5,6. However, several key factors led us to explore more efficient alternatives. First, our grayscale nanoparticle images differ substantially from common ML datasets, which typically consist of color images with complex scenes. This distinction made direct adaptation of existing ML models impractical. While supervised transfer learning54,55 offered a potential solution for adapting existing models to our ___domain, the absence of ground-truth segmentation labels in our dataset and the prohibitive cost of creating them made this approach infeasible.

These constraints led us to explore unsupervised learning methods, specifically 1D k-means clustering56. This approach offers several advantages: it groups similar data points without requiring labels, has well-established implementations in multiple libraries57,58,59, and maintains computational efficiency. This choice aligns with recent trends in automated microscopy analysis7,8, where unsupervised methods have shown promise in handling specialized scientific data.

For dividing segmented regions into acquisition boxes, we initially investigated geometric meshing techniques as in60,61. However, two significant limitations emerged: first, standard meshing algorithms typically generate triangular grids62, incompatible with our requirement for square acquisition boxes; second, advanced adaptive refinement features often come embedded in comprehensive software suites such as63, introducing unnecessary dependencies and potential performance overhead. These considerations led us to develop custom methods for generating acquisition coordinates (detailed in “Boxing algorithms”).

Our pipeline bridges the gap between initial microscopy imaging and high-resolution analysis techniques. The system generates adaptively-sized acquisition coordinates based on pixel intensity, optimizing data collection for regions of interest while minimizing unnecessary sampling. Our approach complements the development of other automated approaches in this field, such as9, and is expected to accelerate the characterization process. The implementation maintains high performance with minimal dependencies, ensuring versatility and longevity.

Datasets

Throughout this project, we relied on a unique set of high-angle annular dark-field (HAADF) images for development and evaluation. The images were acquired specifically to support the development of automated materials characterization. Data were collected on three dates using a JEOL JEM-ARM200CF scanning transmission electron microscope to mimic diverse laboratory conditions. The images were curated to represent a range of magnification levels and materials compositions, from one to five elements.

Since the first step in our overall nanoparticle characterization workflow involves a quality assessment, each image was analyzed by a human expert and labeled either “good” or “bad.” These labels correspond to whether or not the expert would choose to further analyze the particle under normal experimental conditions. These labels allowed us to train ML models in previous work to automate the initial quality assessment, as described in29,30,31. Additional metadata in the file names provides unique identifiers for each image. More details about the datasets and collection settings are available in29.

This work emulates the characterization step immediately following the quality check, where a particle has passed the assessment and been selected for in-depth analysis. As a result, we chose to filter our comprehensive dataset down to only the images that would reach this stage in the characterization workflow. Using the unique labels provided for each image, we focused on images labeled “good” with magnifications at or above 1,500,000x, resulting in a final dataset of 964 images for this work. The images were distributed across the three source datasets: 204 from the first, 399 from the second, and 361 from the third (Supplementary Table SI 1).

Methods

Our pipeline consists of three major components: image preprocessing, clustering, and acquisition box generation. Each component was developed with careful consideration of computational efficiency and effectiveness across diverse sample conditions. The following sections detail our procedures and design decisions for each step.

Image preprocessing steps

The first step in our pipeline involves downsizing the parent image to 128 × 128 pixels. This step provides a direct comparison with the existing 128 × 128 grid technique and allows us to standardize inputs across datasets (parent images have original resolutions of either 512 × 512 or 1024 × 1024, depending on the collection date). The downsizing step is illustrated in Fig. 2b.

Fig. 2
figure 2

The first six steps in the image preprocessing pipeline. Clockwise from top left: (a) the original image; (b) the original image resized to 128 × 128 pixels; (c) a sharpening mask applied to (b); (d) a Gaussian Blur applied to (c); (e) the binary mask resulting from Gaussian thresholding on (d), overlaid by the bounding box in red; (f) image (b) cropped to the coordinates of the bounding box in (e). The axis labels for all images show pixel dimensions; the axis bounds for (f) are relative to (b). The composition of this nanoparticle is Au40Ag10Cu10Ni40. The original image, shown in (a) has a magnification of 1,500,000 × and a resolution of 1024 × 1024 pixels.

Full size image

Next, the image is cropped to remove the background and center the nanoparticle in the foreground. This step creates a smaller image, significantly reducing processing time. To prepare the image for cropping, we apply image enhancement techniques to reduce noise and highlight features. First, a sharpening mask is applied to the image (Fig. 2c), followed by a GaussianBlur filter (Fig. 2d) and an adaptive Gaussian threshold. This last step produces a binary mask with foreground and background regions, allowing us to calculate a bounding box that fully encloses the foreground pixels (Fig. 2e). Finally, the original 128 × 128 image is cropped to the region enclosed by the bounding box, yielding a focused region of interest (Fig. 2f).

We then perform an additional step to further separate the foreground from the remaining background. We plot pixel intensity along the upper-left-to-lower-right diagonal in the cropped image, resulting in a profile that varies along with features in the image (Fig. 3a). We then compute a cutoff using information from the profile slope and total intensity range (Fig. 3b). During the design process, we implemented the same technique with the lower-left-to-upper-right diagonal, resulting in a second cutoff value. However, the differences between the two cutoffs were trivial in most cases (Fig. 3c), so we chose to use only information from the upper-left-to-lower-right diagonal to streamline the process.

Fig. 3
figure 3

To further separate the background from the foreground, we define a cutoff value by examining the pixel intensity of the upper-left-to-lower-right diagonal of the cropped image. The cropped image is shown in (a) with the diagonal overlaid in green for reference; (b) a plot of the pixel intensities along this diagonal, with the cutoff value as a red horizontal line; (c) a histogram of the difference in intensity values between the upper-left-to-lower-right and upper-right-to-lower-left diagonals across the entire dataset.

Full size image

Once the cutoff is calculated, we set the pixel values in the cropped image below the cutoff to 0, corresponding to the background, and set the remaining pixels, the foreground, to 1. This process produces a binary mask in the shape of the outline of the particle (Fig. 4b).

Fig. 4
figure 4

(a) cropped image; (b) binary mask of (a) after the background removal procedure in Fig. 3; (c) convex hull of (b); (d) the result of filtering (a) based on the mask in (c) and brightening the foreground by the amount in Eq. (1). Red ovals highlight the difference between the binary mask and the convex hull.

Full size image

Since the nanoparticles in our application should always be convex, we perform an additional check at this step. First, we calculate the convex hull of the binary mask (Fig. 4c). This step returns a new binary mask that corrects any undesirable concavity in the original mask. We then use this mask to remove the background in the cropped image. The pixel locations in the cropped image corresponding to the mask background are set to 0, and the foreground pixels are brightened slightly to further differentiate them from the background. The amount by which the foreground is brightened is relative to the intensity range of the diagonal, as shown in Eq. (1):

$${\text{Foreground Offset }} = {\text{Ceiling}}\left( {{\text{Range}}\left( {\text{Diagonal Pixel Intensities}} \right) \, *0.{2}} \right)$$
(1)

Following these steps, the image is sharpened and blurred using the same techniques as before. The output image from this step becomes the input to the k-means clustering, as described next.

Clustering

Since we use 1D k-means clustering for image segmentation, we need to estimate the number of clusters in the image as precisely as possible. We experimented with several techniques and converged on a strategy that combines information about the number of elements in the particle composition and the peak pixel intensities in the image. This approach allows us to make use of chemical information specific to each particle. Looking ahead, this approach could be extended to novel materials compositions.

We used a histogram and scipy’s find_peaks64 tool to estimate the number of pixel intensity peaks. Since values of 0 correspond to the background and are not meaningful for this step, they were removed from the histogram. A bin size of 20 was used, and criteria was set to capture any peak with a prominence greater than 5% of the maximum value on the y-axis (Fig. 5).

Fig. 5
figure 5

The process identifies one pixel intensity peak for the particle shown in Figs. 2, 3 and 4.

Full size image

Higher k values result in longer segmentation times, so it is crucial to identify the lowest possible k values that would produce reasonable segmentations. To find a good value of k for each image, we combine information about the number of peaks (p) calculated using the histogram and the number of elements (n) in the composition. Our procedure uses the following logic:

Procedure 1

If (p < 2) or (abs(p-n) >= 2):

k = n + 1

Else:

k = p + 1

We use the resulting k value to segment our image using 1D k-means clustering (Fig. 6).

Fig. 6
figure 6

(a) The cropped image, shown for reference; (b) final segmentation result after applying k-means clustering with k = 5.

Full size image

Boxing algorithms

We implemented two different algorithms for creating acquisition boxes. One approach involved finding and prioritizing the largest possible non-overlapping boxes within a given segmented region. The second approach was more efficient but did not prioritize the largest squares. The first strategy is described in Algorithm 1, and the second in Algorithm 2:

Algorithm 1

Description: Returns a list of squares T that completely fill some segmented region R inside an image array A, except the background region. Sorts the squares to prioritize the largest non-overlapping boxes. Each element of T contains upper-left-hand coordinates x, y and size s.

Input: A, a mask array the same size as the image; for some given element a \(\in\) A, a = 1 if the pixel at that ___location \(\in\) R; 0 otherwise. Let the element at position x, y be denoted A[x, y]. Let the lower-right-hand corner of A be denoted as A[maxx, maxy]. Let Q be a queue containing the x and y coordinates of all initial nonzero values of A, where the order of the coordinates is the same as when iterating across columns and then down rows of A. The maximum allowed box size is specified as m.

figure a

Algorithm 2

Description: Returns a list of squares F that completely fill some segmented region R inside an image array A, except the background region. Each element of F contains upper-left-hand coordinates x, y and size s.

Input: A, a mask array the same size as the image; for some given element a \(\in\) A, a = 1 if the pixel at that ___location \(\in\) R; 0 otherwise. Let the element at position x, y be denoted A[x, y]. Let the lower-right-hand corner of A be denoted as A[maxx, maxy]. Let Q be a queue containing the x and y coordinates of all initial nonzero values of A, where the order of the coordinates is the same as when iterating across columns and then down rows of A. The maximum allowed box size is specified as m.

figure b

In consultation with ___domain experts on our team, we decided to cap the size of the maximum possible box to 20% of the shorter side of the cropped image. For example, a cropped image of size 100 × 80 pixels would contain acquisition boxes up to a maximum size of 80*0.2 = 16 pixels. Setting this upper bound allowed us to standardize our results and prevent detail loss in large images. It also made Algorithm 2 the clear choice for this application, although we note that Algorithm 1 may be the preferred choice in other situations. Algorithm 2 was used for the following results. An example application of Algorithm 2 is shown in Fig. 7; additional examples of the process are in Supplementary Figs. SI 1–6.

Fig. 7
figure 7

(a) Acquisition boxes calculated by Algorithm 2 overlaid on the segmentation result in Fig. 6. (b) The same boxes overlaid on the cropped image.

Full size image

Results

After the final design decisions were made, we ran the code on all 964 images in our dataset. We gathered timing information from these experiments and performed a visual inspection of the results.

Timing and performance

To standardize the segmentation results on successive code runs, we set a random seed and verified that the segmentation and boxing results were unchanged between runs when the seed was applied. We used python’s cProfiler65 to identify performance bottlenecks and achieved significant speedups between successive versions of the code. The timing tests described below were conducted on a 2020 MacBook Pro, running macOS Sonoma 14.5. Run times in the laboratory may vary, depending on the software environment and other processes on the machine.

To account for timing variability, we ran the code on each image 30 times while calculating the elapsed processing time. We then calculated the median of those 30 values for each image. In the following calculations, we examined the distribution of these median values across the entire dataset. The average of the median image processing times across the entire dataset was 0.05 s, with a maximum of 0.13 s and a minimum of 0.02 s. The median value was 0.04 s. The distribution of these median processing times is shown in Fig. 8:

Fig. 8
figure 8

Histogram of the median processing times for our dataset, when each image is run 30 times. In other words, we processed each image 30 times, then calculated the median processing time for that image. This figure shows the distribution of median times over the entire dataset. This processing time includes the preprocessing, segmentation, and box calculation steps.

Full size image

Because the random seed guarantees that segmentation and boxing results are the same every time, we report the number of boxes calculated on a single run of the code. The number of boxes varied between 63 and 1819, with a mean value of 551.2 and a median of 493.0. Since the original procedure captured 16,384 child images for each parent image, our process results in 9.0 times fewer boxes in the worst-case scenario, 260.1 times fewer in the best case, and 29.7 times fewer in the average case. This streamlined process saves time and data storage space. The distribution of the number of calculated boxes across the entire dataset is shown in Fig. 9:

Fig. 9
figure 9

The distribution of boxes identified by the procedure across the dataset.

Full size image

To benchmark our process, we calculate the projected total time needed to process an image and capture 4D-STEM data using known laboratory acquisition times. For an image, we can represent the total time to process and acquire data using Eq. (2), where T is the total time, tseg is the time for the segmentation and boxing process, nb is the number of boxes, and tacq is the projected acquisition time per data point.

$$T \, = \, t_{seg} + \, n_{b} * \, t_{acq} .$$
(2)

Our benchmark 4D-STEM acquisition methods take 8, 16, 32, and 64 s for a single image and capture 16,834 individual data points. We calculate that it takes \({t}_{acq}=\frac{{s}_{b}}{\text{16,384}}\) to acquire a single data point, where sb is a known benchmark time. For example, given a current acquisition process that takes 8 s per image, we calculate \({t}_{acq}=\frac{8 seconds}{\text{16,384} points}\) = 0.488 ms per data point. To show how the speedup changes for each of these values, we calculate the distribution of T expected for each of these values (Fig. 10).

Fig. 10
figure 10

Projected total time to process and acquire images, using Eq. (2), for benchmark acquisition methods ranging from 8 to 64 s. To calculate these times, the median runtime of 30 trials was used for each image.

Full size image

For these calculations, we again used the median timing value over 30 trials for each image and then calculated statistics across the entire dataset (Table 1).

Table 1 Timing and performance statistics across the entire dataset.
Full size table

We note that the acquisition time portion (nb * tacq) dominates the total projected runtimes, and the proposed preprocessing, segmentation, and boxing time (tseg) is a relatively insignificant component of the total. Note that tseg is independent of the baseline acquisition method, i.e., it is the same for both the 8 s and 64 s baselines.

Visual inspection

Running the code on our dataset generates 964 distinct pipeline results, which we used to evaluate overall performance. We used the following criteria to judge each image: (1) the bounding box should fully enclose the entire particle, including any faint edges; (2) the segmentation step should separate any areas in the image that differ in pixel intensity; (3) the returned acquisition boxes should reflect regions of interest within the original image. Using these metrics, we performed a visual inspection on a single run of the code. We identified 925 of the final images as meeting all criteria, providing an overall success rate of 96.0%. In the absence of ground-truth labels, we acknowledge that this technique is subjective and depends on individual judgment. However, example results with ambiguous outputs were discussed with ___domain experts in our team before this final figure was calculated.

Conclusion

We present an efficient, automated image segmentation pipeline that addresses a critical bottleneck in high-throughput nanoparticle characterization. The pipeline is expected to process images 25.0 to 29.1 times faster than baseline methods and operates without human input. We evaluated the results of our pipeline on over 900 diverse images and report a 96.0% success rate based on expert-validated criteria. The system effectively handles variations in focus, magnification, and contrast, demonstrating robust performance across different experimental conditions. Moreover, our approach is expected to reduce the number of acquisition points by an average factor of 29.7, significantly saving storage space. The pipeline’s distinctive features include adaptive sizing of acquisition boxes and integration of particle composition information in the segmentation process. Our code was optimized for speed and designed to contain minimal dependencies, ensuring long-term efficiency and stability. Future work could involve extending this methodology to additional microscopy datasets and performing detailed studies of real-time laboratory deployment. This automated pipeline represents a significant step toward fully automated materials characterization, reducing human intervention in routine preprocessing tasks and accelerating data acquisition.