Introduction

The investigation of the three-dimensional (3D) structure plays a vital role in nanotechnology research, encompassing areas like nanofabrications1,2, nanorobots3,4, and nanomedicines5,6, given its critical relevance to the functional properties of micro- and nanoscale objects. The scanning electron microscope (SEM)7 is a prevalent tool for providing qualitative information about the geometry of micro- and nanostructures. This technique involves irradiating the sample with an electron beam and capturing a 2D image by detecting the intensity of secondary electrons emitted from the sample surface. Despite its widespread use and multiple advantages, SEM is a destructive method8, requires a vacuum environment, and cannot provide accurate height information in the images. In contrast, the atomic force microscope (AFM)9 acquires precise height information of the sample surface through the forces between its probe and the sample. Moreover, AFM can operate in various environments, is insensitive to the sample material, and is non-destructive.

Nonetheless, conventional AFM comes with its own set of challenges. One primary limitation is that conventional AFM can only capture 2.5D10 information instead of a complete 3D representation of the sample because the position feedback in conventional AFM systems is confined to the vertical direction9. Another challenge is the issue of tip-sample convolution11. This phenomenon arises from geometrical interactions between the AFM tip and the surface features of the sample (Supplementary Fig. 1). These interactions often lead to artifacts11,12,13 in the scanning results that are inherently difficult to differentiate from the actual sample geometry. Such limitations impede the effective use of AFM to investigate intricate 3D micro- and nanostructures and catalyze the development of advanced AFM technologies, i.e., 3D-AFM.

The advancement of 3D-AFM technology predominantly follows two distinct trajectories. The first approach involves the design of specialized probe shapes aimed at enabling the measurement of structures that are inaccessible with conventional AFM scanning. As an example, critical dimension AFM (CD-AFM)14,15, currently a prevalent method for semiconductor structures, manipulates the lateral dithering of a flared tip to capture lateral geometric information of the sample. These designs equip the AFM with the capability to image not only vertical but also undercut sidewall structures. Additionally, there are other creative designs of probes, such as the introduction of hinge structures16, orthogonal cantilevers17, and probes made of carbon nanotube with high aspect ratios18. However, the extra cost and complexity of manufacturing special probes and customized AFM scanning systems present substantial challenges to the widespread adoption of these methods.

Another common technique for 3D-AFM is the practice of tilting either the probe19,20,21,22,23 or the sample24,25 to scan micro- and nanostructures from multiple directions. These methods avoid the necessity for specialized flared tips, instead relying on integrating multiple scans into a complete 3D model. As a result, the effectiveness relies on the precision of the data stitching of multiple scans. Historically, previous methods19,20,21,22,23,24,25 predominantly apply to simple, well-defined structures, such as gratings, for their CD metrology. The grating’s relatively straightforward structure facilitates the manual removal of artifacts resulting from tip-sample convolution while only requiring two tilt scans towards both sidewalls to obtain complete geometric information. This greatly simplifies the problem of the tilting method. However, substantial challenges arise when the tilting method is applied to micro- and nanostructures of unknown and intricate shapes, such as those created by Two-photon Lithography (TPL)26,27,28,29 or comprised of diverse nanoparticles5,6,30—a scenario frequently encountered in nanotechnology research. Firstly, the complex surface geometries of these structures make it difficult to manually identify and remove artifacts from the AFM images. Secondly, the existence of these unremoved artifacts in the scans impedes the accuracy of the stitching process. Thirdly, owing to the complex overlapping relationships among multi-view data, simply stitching of these data is insufficient for the construction of a clear and accurate 3D model.

In this study, we propose MVN-AFM, a framework that is able to reconstruct the 3D surface model of a wide range of complex-shaped micro- and nanostructures without any specially shaped probes or costly modifications to the AFM system. Our framework leverages the concept of tilting samples, but we extend its application for complex structures beyond the limitations of existing methods. Specifically, we propose an iterative optimization algorithm to automatically remove AFM artifacts and improve the alignment accuracy of multi-view data from intricate micro- and nanostructures. Compared with previous tilt methods that only consider two tilt scans, our method can handle multi-view AFM data from more tilt angles, eight in our experiments. Subsequently, in order to reconstruct the 3D model of these structures by multi-view AFM data, we draw inspiration from multi-view depth fusion techniques31,32,33,34,35,36 in computer vision. We introduce the neural implicit surface reconstruction methods37,38,39,40,41,42, the recent advance in this field, to utilize a neural network to represent the 3D model of micro- and nanostructures. By employing differentiable volume rendering to train the neural implicit function with multi-view AFM data supervision, we fuse the multi-view scanning results into an accurate and comprehensive 3D model. Furthermore, we conduct extensive experiments to evaluate the capabilities of the MVN-AFM framework. In detail, we utilize the TPL technique to fabricate various 3D microstructures with distinct geometrical characteristics and prepare specimens of commonly used nanoparticles, including poly(methyl methacrylate) (PMMA) nanospheres43,44,45 and zeolitic imidazolate framework-67 (ZIF-67) nanocrystals46,47,48,49. MVN-AFM effectively eliminates artifacts present in raw AFM images and successfully reconstructs not only the overall shape but also specific hidden details that are not discernible in conventional AFM scans. The ability of MVN-AFM to provide detailed and accurate 3D reconstructions of a broad spectrum of micro- and nanostructures, coupled with its low implementation cost, positions it as a potentially valuable tool in nanotechnology research.

Results

Pipeline of MVN-AFM

The objective of MVN-AFM is to construct a generalized process that can be used for 3D reconstruction of unknown-shaped complex micro- and nanostructures based on multi-view AFM scanning data, relying only on conventional AFM systems and standard probes. MVN-AFM consists of three main steps (Fig. 1): multi-view AFM scanning, data alignment and mask solving, and neural implicit surface reconstruction.

Fig. 1: The pipeline of MVN-AFM.
figure 1

a First, we place various micro- and nanostructures on a rotatable tilt stage, including two-photon lithography (TPL) microstructures and some commonly used nanoparticles. Second, we rotate the turntable and measure the vertical heights by conventional atomic force microscopy (AFM), resulting in a set of multi-view AFM images with many artifacts. b Input raw AFM images with artifacts and iterate two sub-steps. In the data alignment process, data judged as artifacts are eliminated before alignment, and the poses of multi-view images are updated. In the mask-solving process, the solved poses transform the multi-view data, and the data consistency is cross-validated to solve the mask of artifacts. c The posed and masked multi-view AFM images are used to train a neural network representing a signed distance field in space by the differentiable volume rendering technique (Supplementary Fig. 5). d The 3D surface model extracted from the signed distance field, and corresponding topography images without artifacts.

Full size image

The step of Multi-view AFM Scanning (Fig. 1a) captures multi-view AFM images of 3D micro- and nanostructures, providing essential geometric information for the subsequent reconstruction process. Previous tilting methods19,20,21,22,23,24,25 acquire complete geometric information with only two AFM scans towards each sidewall of the grating. However, in nanotechnology research, the prior knowledge of the sample’s shape and orientation is often unknown. To address this, we design a multi-view AFM scanning process that is independent of the sample’s shape. This process aims to comprehensively acquire the surface geometric information of unknown and complex-shaped structures. We use the sample-tilting approach to collect multi-view data, thus avoiding modifications to the mechanics of conventional AFM. For this purpose, we designed a rotatable stage with a tilt angle (Supplementary Fig. 2). In our experiment, the tilt angle is set to 30°, the largest tilt angle that prevents the cantilever of the AFM probe from hitting the sample (see Methods for further details). We carefully designed the size of the whole stage so that it can be used in the limited activity space of a commercial AFM without any collision. The sample is placed on a turntable in the center of the stage so that multi-view scans around the sample can be acquired as it rotates to different directions. We performed eight tilt scans for each sample, and these scan directions are evenly distributed around 360° of the sample’s surroundings. Together with a scan without sample tilt, a total of nine multi-view scans are acquired per sample.

In the step of Data Alignment and Mask Solving (Fig. 1b), we iteratively align multi-view AFM data to a unified coordinate system and remove the artifacts in AFM images. A critical process in the tilting method involves establishing the spatial relationship among multi-view data. This requires determining a coordinate transformation, denoted as pose T, to align data from different views within the same coordinate system (Supplementary Fig. 3). To achieve this, some methods19,22,23 employ designs with high-cost components to enable precise control of probe scanning direction, which allows direct access to T. Others20,24,25 utilize the Iterative Closest Point (ICP) algorithm50 to solve T by minimizing the distance between AFM data points. The ICP algorithm relies heavily on data free from artifacts that do not represent the actual sample shape. Consequently, these previous methods20,24,25 manually remove highly recognizable artifacts from AFM data of simple structures before using the ICP algorithm. However, for multi-view images obtained by a conventional AFM system on intricate structures, there are two challenges: eliminating artifacts and solving for the pose. Here, we define the label of whether each data point is an artifact as a latent variable, mask M. It determines whether the measurement from each AFM pixel is utilized in the reconstruction process. To simultaneously solve for T and M, we propose an iterative EM-like algorithm51,52. Initially, we consider all AFM data artifact-free, i.e., M0 is all zeros, and directly apply the ICP algorithm to obtain a set of coarse poses, T0. As ICP also considers AFM artifacts when minimizing the point set distance between multi-view data, T0 is coarse and contains many errors. In the E-step, we project multi-view data onto each other using Ti−1 from the previous iteration i−1. We then conduct cross-validation of the projected data to identify areas of inconsistency in multi-view data. These regions are then labeled as ones, and we obtain the updated Mi. The motivation for the cross-validation is that artifacts vary with the probe-sample angle, so they are inconsistent in multi-view data. In contrast, the sample’s geometric surface remains consistent across different views, regardless of the probe scanning directions. In the M-step, we erase the artifact through Mi and apply the ICP algorithm again to compute the updated Ti. Iterating the EM steps, the data filtered out of most artifacts by M yields a more precise T. The improved T also makes the cross-validation accurate. Two steps are iteratively performed to enhance each other. After k iterations, we obtain the accurate pose Tk and the artifact mask Mk for each viewpoint of the AFM data.

The step of Neural Implicit Surface Reconstruction (Fig. 1c) utilizes the aligned and masked AFM data to train an implicit function represented by a neural network and extract the final 3D surface model of micro- and nanostructures from the network. Specifically, we follow previous work53 and model the geometry surface of the sample as a neural network encoded Signed Distance Field (SDF): \(s(x;\theta ):{{\mathbb{R}}}^{\!3}\to {\mathbb{R}}\), where x denotes a 3D position and θ is the parameters of a Multilayer Perceptron (MLP). The SDF defines a scalar field where each point in space is associated with the shortest distance to a surface. This distance is positive if the point is outside the surface and negative if it is inside. Previously, the neural implicit surface reconstruction methods42 were developed for posed images from cameras in the macroscopic world, not for nanotechnology and AFM data applications. To adapt this method to our reconstruction process, we convert AFM images into depth maps as captured by virtual orthogonal cameras (Supplementary Fig. 4). Each pixel in AFM images transformed by pose T and filtered by mask M represents a sample ray. The loss function is the disparity between the AFM data and the depth value derived from differentiable volume rendering along the ray. We then optimize the MLP network parameters θ through back propagation54. Moreover, we also use the multiresolution hash encoding technique55 to accelerate the training process. Upon completing the training, we can query the SDF value of any spatial point by inferring the network. Based on the fact that the zero set of SDF represents the structure surface, the Marching Cubes algorithm56 is finally utilized to extract the 3D surface model of the micro- and nanostructures (Fig. 1d).

Reconstruction of two-photon lithography structures

In this section, we evaluate the proposed MVN-AFM on microstructures printed by TPL technology. The TPL technology, which focuses a femtosecond laser into tiny voxels in a photosensitive resist, enables 3D printing of a given computer-aided design (CAD) model with sub-100 nm resolution through the two-photon polymerization (TPP) process57. To fully demonstrate the performance of MVN-AFM on complex 3D microstructures, we printed a set of samples with different geometrical features. Specifically, we printed six structures (Supplementary Fig. 6): cylinder, undercut, spiral, gear, monkey, and house. For centrosymmetric structures, we incorporated three small cones around each microstructure to indicate their orientation, as depicted in the first four rows of Fig. 2. This step is unnecessary for non-centrosymmetric structures, such as the monkey and the house. The height of these microstructures varies between 2 μm and 3.5 μm. We performed AFM scans in tapping mode, with a scan size of 10 μm × 10 μm and 256 lines of 256 points for each AFM image, corresponding to a pixel size of 39 nm. Because the minimum feature size of structures printed by the commercial photoresist in our experiment is 160 nm (see “Methods” section for further printing details), the pixel density we choose is sufficient to capture the geometry feature of the TPL samples.

Fig. 2: MVN-AFM reconstructs the surface models of two-photon lithography (TPL) microstructures.
figure 2

a–f Scanning Electron Microscope (SEM) images of TPL microstructures. g–l 3D models of TPL microstructures’ conventional AFM scanning data. m–r 3D models of TPL microstructures reconstructed by MVN-AFM. g, h, m, n Include the cross-section profiles in the x-z plane. i–l, o–r Include the cross-section profiles in the x-y plane. More visualizations can be found in Supplementary Fig. 7 and Supplementary Movie 1.

Full size image

The cylinder (Fig. 2a) is a representative structure that challenges conventional AFM scanning58 and previous tilting methods. Unlike grating structures, the vertical annular sidewall cannot be divided into distinct left and right sections. Next, the undercut (Fig. 2b) is a prevalent structural feature in semiconductor manufacturing59. This structure differs from the cylinder by having a sloped sidewall. We further constructed the gear (Fig. 2c), a mechanical structure frequently encountered in Micro-Electro-Mechanical systems (MEMS)60. The spiral (Fig. 2d) is distinguished by an intricate array of rotating curved concave and convex structures on its sidewall. Furthermore, we also conducted tests using the Suzanne Monkey (Fig. 2e), a standard model in computer graphics61. Unlike the previous columnar structures, this model poses unique challenges due to its curved features and the indistinct boundary between its top surface and sidewalls. We finally designed a house structure (Fig. 2f) that included shapes with both planar and curved features, along with detailed elements like grooves on the sidewalls to represent doors and windows.

In conventional AFM scanning, the results are a mixture of incomplete surface geometry and artifacts, which do not accurately represent the sample surface. As illustrated in Fig. 2g and h, despite the vast difference in sidewall geometry, the scanning result of the undercut is indistinguishable from that of the cylinder model. Some detailed features are also virtually invisible, such as the doors and windows in the house model (Fig. 2l). The cross-sectional profiles reveal considerable distortion of these scanning results, which may lead researchers to misjudge the actual shape of these samples. Moreover, it is obvious that manually separating artifacts from the AFM scans of these intricate structures is almost impractical.

In contrast, the proposed MVN-AFM framework effectively eliminates artifacts while precisely merging geometric information from multi-view AFM scanning into accurate and comprehensive 3D models. These reconstructed models align consistently with SEM images and demonstrate the surface of these samples. These models clearly differentiate between the cylinder (Fig. 2m) and undercut (Fig. 2n) structure, precisely reconstruct the gear’s teeth (Fig. 2o), and capture the correct orientation of the spiral threads (Fig. 2p) and the monkey’s subtly inward-curving side faces (Fig. 2q). Even the minutely detailed grooves (Fig. 2r) on the house sidewalls are observable.

Reconstruction of nanoparticles

To further demonstrate the generalization of MVN-AFM on structures with smaller sizes and different geometry features, we selected some widely used nanoparticles, including PMMA nanospheres and ZIF-67 nanocrystals.

Nanospheres are a common type of nanoparticle with a wide range of applications62. The characteristics of nanospheres depend greatly on their size and surface structure6, making accurate 3D reconstruction valuable for their research. To evaluate the effectiveness of our proposed method on spherical structure, we chose PMMA43,44,45 with a diameter of about 500 nm, a widely used type of polymeric nanosphere. In Fig. 3e, it is evident that the artifacts in the conventional AFM scanning data are seamlessly connected with the top curved surface of the nanospheres, and the overall shape does not exhibit a spherical appearance. Under these circumstances, manually distinguishing artifact boundaries in AFM scanning as in previous methods becomes unachievable, and the details on the sides of the nanospheres are entirely lost, posing a challenge for researchers to accurately determine the size and structure of these nanospheres. In contrast, MVN-AFM demonstrates its advanced capabilities by accurately reconstructing several adherent nanospheres, each mirroring the shape observed in SEM photographs (Fig. 3c). The reconstruction result of our method is the deepest penetration made by the tip. The multi-view scanning enables the tip to approach the sample’s surface closely, reducing the errors in the results. As shown in the cross-sections in Fig. 3e, g, the model of MVN-AFM has a center width very close to the height, which is consistent with the characteristics of a sphere. However, it is impossible to get the geometric information of structures that the tip cannot touch, such as the region near the bottom of the nanosphere that is tangential to the substrate plane. Therefore, the bottom width of the reconstructed model by MVN-AFM is smaller than the conventional AFM result but not close to zero.

Fig. 3: MVN-AFM reconstructs the surface models of poly(methyl methacrylate) (PMMA) nanospheres and zeolitic imidazolate framework-67 (ZIF-67) nanocrystals.
figure 3

a, b SEM images of nanoparticles in the overhead view. c, d SEM images of nanoparticles in the tilt view. e, f 3D models and cross-sections of nanoparticles’ conventional AFM scanning data. g, h 3D models and cross-sections of nanoparticles reconstructed by MVN-AFM. More visualizations can be found in Supplementary Movie 2.

Full size image

Next, we selected ZIF-67, a cubic symmetric nanocrystal, as a representative crystal-like nanoparticle to assess the effectiveness of our method. ZIF-6746 and its derivatives exhibit various excellent properties, leading to their extensive attention and research47,48,49. The morphological characteristics and size of nanocrystals can be tailored by manipulating experimental conditions during synthesis, leading to variations in their properties63,64,65. Therefore, obtaining accurate 3D surface models of nanocrystals is of paramount importance. In the SEM images (Fig. 3d), the ZIF-67 nanocrystals exhibit a distinct polyhedral shape, ranging in size from about 100 nm to 500 nm. However, conventional AFM results (Fig. 3f) only partially demonstrate the top surface of the crystals, resulting in an overall blurred representation of the particles’ shape. Furthermore, in scenarios where multiple crystals aggregate, as illustrated in our example, only the uppermost crystal in the stack is visible in the conventional AFM scanning (Fig. 3f, the arrow in cross-section), with the underlying crystal completely obscured by the top crystal and associated probe artifacts. On the contrary, the surface model reconstructed by our method (Fig. 3h) accurately captures the polyhedral shape of the ZIF-67 crystals, delineating their side planes and edges with precision. Even in cases where the particles are stacked up, the MVN-AFM method successfully reveals the bottom crystal (Fig. 3h, the arrow in cross-section), typically obscured in conventional scans, and accurately represents the arrangement of the particles in the stack, aligning with the SEM photograph (Fig. 3d).

In our nanoparticle experiments, PMMA nanospheres and ZIF-67 nanocrystals differ markedly from the previous TPL microstructures in terms of material compositions, geometric features, and particle sizes. MVN-AFM precisely reconstructs these diverse samples by the exact same procedure and parameters, showcasing its generalizability and potential applicability in a broad spectrum of micro- and nanostructure research.

Evaluation of simulated data

To complement the previous qualitative comparisons on real experimental data, we embarked on quantitative evaluations using a set of simulated AFM data. We generated these data based on the CAD models of structures in the TPL experiment. The simulation environment allows for the precise determination of the spatial relationships between multi-view AFM data and access to an accurate surface model of the sample, a feat challenging to achieve in real-world experiments. To ensure that the simulated data closely mimics real AFM scanning conditions, we developed a simulated probe model. This model is based on the quadrilateral pyramid probe (Fig. 4a) utilized in our TPL experiments. Considering the nanoscale curvature of the actual AFM probe is negligible compared to the microscale dimensions of the TPL samples, we simplified the probe representation into a pyramid shape (Fig. 4b). The simulation of AFM scanning was then carried out by modeling the rigid body collision66 between the probe and the sample models. As depicted in Fig. 4c, d, the simulated data exhibit a high degree of similarity to the real AFM data in terms of the overall shape and the presence of artifacts.

Fig. 4: The consistency between our simulated AFM data and the real experiment data.
figure 4

a SEM images of the real AFM probe used in the two-photon lithography experiment. b The probe model in the simulation environment. c The real AFM images captured by the multi-view tilt scanning of the monkey structure in the two-photon lithography experiment. d Simulated multi-view AFM images constructed by simulating the collision between the probe and the structure surface.

Full size image

First, we focus on showcasing the enhancements MVN-AFM brings to the alignment accuracy of multi-view AFM data. In the alignment process, we estimate the pose Test for each viewpoint. We also acquire the corresponding accurate pose Tgt in our simulation environment. Here, we respectively calculate the absolute pose error (APE) for the rotation and translation components between the estimated and ground truth poses of multi-view AFM data (see Methods for further details of error calculation). As illustrated in Fig. 5, we present a comparative analysis between the alignment method in MVN-AFM and the direct ICP alignment of raw AFM data, which includes artifacts. The analysis reveals that MVN-AFM achieves a substantial improvement in alignment accuracy, evidenced by an impressive average reduction of 46% in rotation errors and 27% in translation errors. These results not only demonstrate the negative impact of artifacts present in AFM data on the precision of data alignment but also highlight the efficacy of MVN-AFM in mitigating these challenges.

Fig. 5: Evaluation of MVN-AFM’s improvement in multi-view data alignment accuracy.
figure 5

Comparison of the absolute pose error, that is, rotation and translation error (lower is better), of direct alignment of raw AFM data and the alignment method of MVN-AFM.

Full size image

In the subsequent analysis, we compare the models reconstructed by two prominent multi-view depth fusion techniques: the neural implicit method and Truncated Signed Distance Function (TSDF) Fusion31. Our method interprets AFM images as depth images from virtual orthogonal cameras, framing the challenge as the depth fusion problem in computer vision. Depth fusion techniques are categorized into traditional31,32,33 and neural implicit methods34,35,36,41,42. The TSDF Fusion is a widely used traditional method that efficiently fuses multi-view depth data by dividing the 3D space into weighted discrete voxels and updating these weights according to the depth information along the pixel ray. However, multi-view AFM scanning of micro- and nanostructures presents unique challenges, particularly the uneven sampling density (Supplementary Fig. 9a) due to restricted tilt angles and limited viewpoints during the scanning process. This limitation often leads to regions with sparse sampling, such as the sidewall grooves of the spiral model (Fig. 6c). In the context of TSDF Fusion (Supplementary Fig. 9b), unintersected voxels in sparsely sampled regions demonstrate as voids in the reconstructed model, a limitation evident in Fig. 6a. Conversely, the neural implicit method, which represents the 3D model as a continuous neural network, exhibits its ability to construct a smooth and complete surface model, even with limited sample points, as depicted in Fig. 6b. This capability of the neural implicit method to effectively handle sparse data and reconstruct intricate surfaces makes it more suitable for the 3D reconstruction of multi-view AFM data in the MVN-AFM framework.

Fig. 6: Evaluation of MVN-AFM’s improvement in reconstructed 3D models.
figure 6

a Given a set of masked and posed multi-view AFM images, the spiral model reconstructed by the Truncated Signed Distance Function (TSDF) Fusion method. The local zoom shows the voids on the reconstructed model. b Given the same set of masked and posed multi-view AFM images, the spiral model reconstructed by the neural implicit surface reconstruction. c The ground truth model of the spiral. d The images of the monkey and house models in the given viewpoints. e The simulated AFM images and corresponding height error maps. f The topography images of 3D models reconstructed by MVN-AFM and corresponding height error maps. More visualizations can be found in Supplementary Fig. 8.

Full size image

Finally, we evaluated the accuracy of the topography in the 3D surface models reconstructed by MVN-AFM. Our simulation environment enables the capture of precise surface topography unaffected by the probe’s shape. The difference between accurate surface topography and the AFM images reveals substantial artifacts (Fig. 6e), particularly around the edges and at the sharper geometric features of the structure in the raw AFM data. These results underscore the complexity of artifacts in AFM images of intricate structures and highlight the challenges associated with their manual removal. The visualization of the difference between the topography images from 3D models of MVN-AFM and the accurate topography images (Fig. 6f) clearly indicates that MVN-AFM is highly effective in eliminating the artifacts present in the AFM data. Moreover, it successfully integrates accurate surface geometric information from various viewpoints, significantly diminishing the surface topography error. To quantify these improvements, we calculated the average of the absolute pixel error values across multiple viewpoints for each model. As summarized in Table 1, this analysis reveals that MVN-AFM achieved an exceptional average reduction of 94% in topography error for each structure, affirming the high accuracy of the 3D models reconstructed by MVN-AFM.

Table 1 The error comparison of conventional AFM images and MVN-AFM
Full size table

Discussion

In this work, we introduce MVN-AFM, a framework for 3D surface reconstruction of intricate micro- and nanostructures using multi-view AFM scanning data. We propose an iterative optimization method to simultaneously align the multi-view data and remove artifacts in the AFM image, achieving higher alignment accuracy. Moreover, we utilize the neural implicit surface reconstruction technique to the field of nanotechnology, which enables fusing spatially overlapping multi-view AFM data into an accurate 3D model. MVN-AFM shows considerable practical value. Extensive experiments demonstrate the superior capability of MVN-AFM on diverse micro- and nanostructures, including microstructures printed by TPL, PMMA nanospheres, and ZIF-67 nanocrystals. The 3D models reconstructed by MVN-AFM provide researchers with a more comprehensive representation of micro- and nanostructures than what is achievable with conventional AFM scanning and 2D SEM images. The success of MVN-AFM across these varied samples, each with distinct geometries, types, and sizes, robustly affirms its effectiveness and broad applicability in nanofabrication, nanoparticles, and many other fields. Importantly, MVN-AFM only requires a conventional AFM system and a standard AFM probe to achieve these results. This aspect makes MVN-AFM a more accessible and cost-effective option for researchers to analyze intricate 3D micro- and nanostructures.

Our framework is efficient and flexible. While multi-view AFM data provides more surface information, it also increases the time for AFM scanning. We tested the effect of reconstruction using different numbers of tilted AFM data(Supplementary Fig. 10 and Supplementary Table 1). We found that for the structures in the TPL experiment, the reconstruction quality converged with only eight tilt scans. The scanning time for one AFM image is about 4.5 minutes, and considering the time required to switch to different scanning directions, multi-view scanning takes about 2 hours for a single structure. Next, given a set of multi-view AFM data, our framework takes about 10 minutes to complete the 3D reconstruction. Notably, the basis of our algorithm is the multi-view consistency of the accurate surface topography and the multi-view inconsistency of image artifacts in multi-view AFM data. Because our algorithm does not take parameters such as scanning number, tilt angle, and probe shape as prior information, users have the flexibility to adjust these parameters according to their requirements. Here are some guidelines for the setting of these parameters. Firstly, users can choose the scanning number by starting with a small number of scans and then gradually increasing it until the reconstruction results converge. Eight tilt scans performed well in our experiments. Secondly, users should maximize the tilt angle while preventing the probe cantilever from hitting the sample, as discussed in detail in Methods. Lastly, our method does not require specially shaped probes. However, if available, using probes with high aspect ratios benefits obtaining more geometric information during multi-view scanning.

Here, our study underscores the immense potential of integrating nanotechnology with neural implicit representations67,68,69, an emerging and rapidly evolving field in computer vision. Specifically, we employ neural implicit surface reconstruction methods, where a neural network effectively represents a continuous SDF in space. Because of the continuous nature of neural networks, it is more suitable for representing geometric surface models that are inherently continuous in space than traditional discrete methods, as demonstrated in numerous recent works34,35,36. Our research further reveals the successful application of this technique in the reconstruction of 3D micro- and nanostructures with multi-view AFM data.

Our methodology’s foundational assumption is that the sample remains static during the multi-view AFM scanning process because the multi-view AFM data alignment step depends on the consistency of geometric features across different views. Therefore, our method is unsuitable for dynamic samples, such as living cells or samples prone to deformation during scanning. Precisely reconstructing the deformation process of nanostructures is widely demanded in many research, which points to a promising direction for future work. One possible solution is applying our method to high-speed atomic force microscopy (HS-AFM)70, which allows observing the dynamic action of nano-objects.

Methods

Hardware and software requirements of MVN-AFM

In our experiments, all the code of MVN-AFM was run on a computer with an Intel i9-13900KF CPU, an Nvidia RTX4090 GPU, 64 gigabytes of RAM, and a Linux operation system with a 5.15.0 kernel version, which is a typical configuration of the current lab workstation computer. In order to run the code of MVN-AFM properly, it requires at least one graphics card with memory larger than 12 gigabytes. We use Open3D71 0.17.0, an open-source Python library, to handle the 3D data. Our implementation of neural implicit surface reconstruction is based on an open-source repository72 of hash encoding55 and NeuS42, and the network is built on the deep learning framework PyTorch73 1.13.1.

Multi-view AFM scanning

All of the multi-view AFM images in our experiments were acquired through a commercial AFM (Dimension ICON, Bruker) operated in ‘Tapping Mode’ in the air, with 256 lines of 256 points, at a scan rate of 1 Hz, using a TESPA-V2 probe (Bruker) with a resonance frequency of 320 kHz and nominal spring constant of 37 Nm−1. The setpoint amplitude was set to  ~70% of the free air amplitude. The AFM probe has a height of 15 μm, an overall shape of a quadrilateral pyramid, a front angle of 25°, a back angle of 17.5°, and a side angle of 20°. The stage has a 24 mm × 24 mm square bottom, and the height is 16 mm with a 30° tilt angle. The turntable can hold a 4 mm × 4 mm sample. The whole stage can be placed directly into a commercial AFM and does not collide with any part of the AFM during scanning. In our experiments, we rotate 45° each time between two adjacent scans and obtain eight tilt scans around the sample. Together with a conventional scan without sample tilt, nine scans are acquired per sample (Supplementary Movie 3).

We set the tilt angle to 30° to ensure that the AFM cantilever does not touch the sample surface during the scanning process while making the tilt angle as large as possible. For many types of AFM probes, the AFM tip is not located at the very end of the cantilever. As shown in Fig. 7a, the length of line segment OC of the TESPA-V2 probe used in our experiments is not negligible relative to the height of the tip, line segment OT. If the tilt angle is too large, point C may contact the sample before point T during the AFM probe engagement, causing the scan to fail. As depicted in Fig. 7b, to ensure that the tip contacts the sample, the tilt angle σ must satisfy σ < ∠OCT − β, where β is the inherent angle of the AFM holder, which is 11°. For our probe TESPA-V2, the angle ∠OCT is approximately 45°. As the tilt angle increases, the AFM probe can access more structures inaccessible to conventional scanning without sample tilt, such as undercut sidewalls, providing more comprehensive surface geometry information for better 3D reconstruction. Therefore, we chose a 30° tilt angle in all our experiments to achieve the largest possible tilt angle while satisfying the constraints. We also included a slight margin in the tilt angle to ensure the cantilever would not touch the sample.

Fig. 7: The limitation of tilt angle.
figure 7

a The SEM image of the TESPA-V2 AFM probe in our experiment. b The image of AFM tilt scanning.

Full size image

Precisely localizing the identical region across multi-view AFM scans is a critical step in the data capture process. The methodologies for achieving this localization are diverse and can be tailored to the unique characteristics of the experiment sample. In our TPL experiments, we utilized polymer grid markers printed around the sample to assist in localization by the optical microscope in the AFM system. For experiments of nanoparticles, we constructed scored markers on mica bases. These are just examples of the various strategies that can be adopted for localization, with other available methods including the use of a Transmission Electron Microscopy (TEM) index grid74,75 or the creation of noticeable artificial markers76. The common destination of these techniques is to ensure that the specific structure for 3D reconstruction can be precisely and efficiently located within the AFM system.

Data alignment and mask solving

First, we claim some basic concepts in this step. Each AFM image is equivalent to a set of 3D points under an AFM coordinate system (Supplementary Fig. 3), with the z-axis being the position feedback direction of AFM and the x-y plane being the probe scanning plane. We define the AFM coordinate system of the scan result without sample tilt as the destinated sample coordinate system. Moreover, we define a corresponding virtual orthogonal depth camera for each AFM image (Supplementary Fig. 4). With a given set of raw AFM data, we convert the AFM height information h into a depth value d for a virtual orthogonal camera parallel to the x-y plane, d = α − h, where α is the assumed height of the camera. The value of α is simply ensured all d to be positive. Each AFM data point is treated as a ray \(r=o+d \vec{v}\), originating from the pixel position o on the imaging plane and extending along the direction \(\vec{v}\) of the camera to the depth d.

In the initialization and the M-step, we applied the point-to-plane ICP algorithm50 to align the data points filtered by mask M of AFM images and get a set of transformations T. In the E-step, We compute the artifact mask M of each AFM image by a cross-validation method. In detail, we first transform each camera ray r to the sample coordinate system to obtain the ray \({r}^{{\prime} }\) by the currently solved \(T=\{(R,t)| R\in {{\mathbb{R}}}^{3\times 3},\,t\in {{\mathbb{R}}}^{3}\}\), where \({r}^{{\prime} }={o}^{{\prime} }+d{\vec{v}}^{{\prime} },{o}^{{\prime} }=Ro+t\), and \({\vec{v}}^{{\prime} }=R\vec{v}\). Subsequently, we generate n sets of meshes by connecting spatial points corresponding to neighboring pixels in n AFM images within the sample coordinate system. Next, we compute the intersection of each ray with these meshes and obtain n depth values, D = {d1, d2, . . . , dn}. Due to a basic fact, the artifacts of tip-sample convolution cause an expansion of the overall topography11, resulting in the height value of AFM scanning being larger than the actual sample height, equivalent to the smaller depth value. Therefore, we consider a pixel as an artifact when \({D}_{\max }-d \, > \,\phi\), where d is the measured depth of each pixel, and ϕ is set to 3% of the AFM scan size initially and linearly reduced to 1% with iteration, a value determined experimentally and applied consistently across all our experiments. The tiny threshold is set to make the algorithm robust to noises in the AFM data and inaccurate T during the iteration process. These iterative EM-steps reinforce each other. After a fixed number of iterations, five in our experiments, the resolved poses T, and masks M are saved for subsequent steps. For a set of multi-view AFM data, this process takes about 2 minutes.

Neural implicit surface reconstruction

In the neural implicit surface reconstruction step, we train a multi-resolution hash table with learnable parameters and an MLP neural network named the SDF network by the aligned and masked multi-view AFM data (Supplementary Fig. 5). In the training process, we sample 3D points along the ray \({r}^{{\prime} }\) of pixels filtered by mask M. First, the 3D point coordinate is encoded by multiresolution hash technology55. Here, we use 16 resolution levels, each obtaining a 2-dimensional feature vector. Concatenating the hash encoding and the 3D coordinate, we get a 35-dimensional feature as input of the SDF network. The SDF network is a one-layer MLP network with 64 hidden sizes and ReLU activation, which maps the input feature to an SDF value at that 3D point. The SDF value of each point is converted to a density value through the unbiased and occlusion-aware weight function proposed by NeuS42. Then, the density values of sample points along the ray are accumulated by the differentiable volume rendering method to obtain the depth value \(\widehat {\!d}\) of that ray. The loss function L consists of a depth error term Ldepth and a regularization term Lreg:

$${L}_{{{\rm{depth}}}}=\frac{1}{b}\sum _{p}^{b}{\left(\widehat{{d}_{p}}-{d}_{p}\right)}^{\!2},$$
(1)
$${L}_{{{\rm{reg}}}}=\frac{1}{bm}\sum _{p,q}^{b,m}{(\parallel {{{\bf{n}}}}_{pq}\parallel -1)}^{2}.$$
(2)

Ldepth is the mean square error (MSE) between the rendering depth value of each pixel and the AFM data supervision, where b is the batch size and p is the index. The regularization term53 is used to constrain the SDF field represented by the network, where n is the normal of the sample point, m is the number of sample points along a ray, and q is the index. Lreg facilitates a smooth and natural surface, commonly used in SDF-based neural implicit surface reconstruction methods.

$$L={L}_{{{\rm{depth}}}}+\lambda {L}_{{{\rm{reg}}}}.$$
(3)

The weight λ is 0.1 in our experiment. During the network training, the Adam optimizer updates the network parameters with a learning rate of 0.001 to minimize the loss and perform 20,000 iterations. In one iteration, we randomly select 256 rays with 1024 sample points along the ray. Notably, Each set of network parameters can only represent a 3D model of one structure, so multi-view AFM images for different samples need to be trained from scratch. The whole training time is about 8 minutes on an Nvidia RTX4090 GPU. To visualize the 3D model, we divide the space into 256 × 256 × 256 voxels. Subsequently, the SDF values for each voxel are obtained through neural network inference, followed by the extraction of meshes using the Marching Cube algorithm56. Unlike the traditional discrete voxel-based representation31, which requires the prior determination of a voxel division resolution, neural implicit surface representations do not have a resolution limitation. The network can infer SDF values at any ___location, enabling the generation of a mesh representation with arbitrary resolution.

Constructing the two-photon lithography structures

We used a commercial photoresist IP-Dip2 (Nanoscribe GmbH) as our material. The IP-Dip2 was dropped on a glass substrate with a thickness of 170–190 μm (Borosilicate substrates, Nanoscribe GmbH) for fabricating the structures. We used A commercial Direct Laser Writing setup (Photonic Professional GT2, Nanoscribe GmbH) equipped with a 780 nm femtosecond laser (a repetition rate of 80 MHz, a pulse duration of 80-100 fs) and a 63×, numerical aperture (NA) = 1.4 oil immersion objective to print the microstructures. We imported the STL files into Describe 2.7 (Nanoscribe GmbH) to generate the executable job files. We set the slice and hatching distances to 0.1 μm for microstructures, the highest accuracy this machine can achieve. These distances were set to 0.3 μm for grid markers because they are only used for optical microscope localization, which has no high requirement for printing accuracy. The printing parameters were set to 30 mW of laser power and 10,000 μm/s scanning speed. Then, we imported the executable job files to Nanowrite 1.8 (Nanoscribe GmbH) to start the job. After the printing process, the printed structures were developed with propylene glycol methyl ether acetate (PGMEA) for 20 minutes and isopropyl alcohol (IPA) for 5 minutes to wash out the unpolymerized resists at room temperature and leave the microstructures on the substrate.

Constructing the nanoparticle samples

PMMA nanosphere dispersion (500 nm) was purchased from the Jiangsu Zhichuan Technology Co., Ltd (China). ZIF-67 powder (300 nm) was purchased from the Nanjing Xianfeng Nano Co., Ltd (China). We used ethanol to dilute these nanoparticles, sonicated them for 10 minutes, and then deposited the suspension onto a 4 mm × 4 mm mica base. We performed multi-view localization of the particles of interest based on the markers of the mica surface around the region. We used the same view number, tilt angle, AFM scanning mode, and AFM probe as in the TPL experiment. The AFM scan size was 2 μm × 2 μm for PMMA nanospheres and 1.5 μm × 1.5 μm for ZIF-67 nanocrystals. We obtained SEM images of these micro- and nanostructures by sputter-coating samples with platinum by sputtering apparatus (MCIOO, Hitachi) and then observing them with a field-emission scanning electron microscope (GeminiSEM 300, ZEISS).

Constructing the simulated data

The simulated data was generated using the 3D design software Blender61 3.3. Within Blender, we constructed models of the structures as well as the AFM probe. To mimic the conditions of our real-world multi-view AFM scanning, we set up orthogonal cameras within the software positioned to align with the scanning directions in our real experiment. Furthermore, to replicate the real-world experimental setup more accurately, we rotated the probe model by 11°. This adjustment accounts for the inherent angle between the working cantilever of the AFM holder and the scanning plane in a real AFM system11,58. Next, we orthogonally projected the surface model onto these cameras and performed a convolution of the probe shape to generate simulated AFM images. In the quantitative evaluation of the accuracy of solved poses, we extracted the precise poses of these cameras directly from Blender and evaluated the absolute pose error (APE) by an open-source tool EVO77. The camera pose T at each viewpoint consists of a rotation matrix R and a translation vector t:

$$T=\left[\begin{array}{cc}R&t\\ 0&1\end{array}\right]\in {{\mathbb{R}}}^{4\times 4},R\in {{\mathbb{R}}}^{3\times 3},t\in {{\mathbb{R}}}^{3}.$$
(4)

The relative pose Ei between the estimated camera pose Test,i and the ground truth pose Tgt,i for viewpoint i is calculated:

$${E}_{i}={T}_{{{\rm{est}}},i}^{-1}{T}_{{{\rm{gt}}},i}\in {{\mathbb{R}}}^{4\times 4}.$$
(5)

For n viewpoints, the translation and rotation components of the relative pose E are extracted and averaged respectively to obtain the translation and rotation errors:

$${{{\rm{APE}}}}_{{{\rm{translation}}}}=\frac{1}{n}\mathop{\sum }\limits_{i=1}^{n}\parallel \!\!{{\rm{trans}}}({E}_{i})\!\!\parallel ,$$
(6)
$${{{\rm{APE}}}}_{{{\rm{rotation}}}}=\frac{1}{n}\mathop{\sum }\limits_{i=1}^{n}| {{\rm{angle}}}(\log ({{\rm{rot}}}({E}_{i})))| ,$$
(7)

where trans ( ⋅ ) extracts the translation vector from the pose matrix, and the final translation error is in units of micrometers, rot ( ⋅ ) extracts the rotation matrix from the pose matrix, log ( ⋅ ) is the inverse operation of Rodrigues’ rotation formula78, transforming the rotation matrix into a rotation vector, and angle ( ⋅ ) then calculates the rotation angle from the rotation vector, in units of degrees. We implemented the TSDF Fusion method based on an open source repository79 and added support for the orthogonal camera projection model that allows for the fusion of multi-view AFM data. We divided the space into 256 × 256 × 256 voxels to keep consistent with the setup of the mesh model extraction step in the neural implicit surface reconstruction. When it comes to evaluating the topography images of the 3D models reconstructed by MVN-AFM, we employ the Mean Absolute Error (MAE) as our metric:

$${{\rm{MAE}}}=\frac{1}{mn}\mathop{\sum }\limits_{i=1}^{m}\mathop{\sum }\limits_{j=1}^{n}\left\vert {\bar{h}}_{ij}-{h}_{ij}\right\vert ,$$
(8)

where n denotes the number of multi-view images, m is the pixel number in each image, \(\bar{h}\) is the accurate height value of a pixel, and h denotes the value of pixels in raw AFM images or topography images from MVN-AFM.