Data Science for Health Image Alignment: A User-Friendly Open-Source ImageJ/Fiji Plugin for Aligning Multimodality/Immunohistochemistry/Immunofluorescence 2D Microscopy Images
Abstract
:1. Introduction
2. Available Tools for Multimodality 2D Image Registration
AIBLROI | Not available. |
BigWarp | Bogovic, J. A., Hanslovsky, P., Wong, A., & Saalfeld, S. (2016, April). Robust registration of calcium images by learned contrast synthesis. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) (pp. 1123–1126). IEEE. [23] |
Correlia | Rohde, F., BRAUMANN, U. D., & Schmidt, M. (2020). Correlia: an ImageJ plug-in to co-register and visualise multimodal correlative micrographs. Journal of Microscopy, 280(1), 3–11. [9] |
ec-CLEM | Paul-Gilloteaux, P., Heiligenstein, X., Belle, M., Domart, M. C., Larijani, B., Collinson, L., … & Salamero, J. (2017). eC-CLEM: flexible multidimensional registration software for correlative microscopies. Nature methods, 14(2), 102–103. [25] |
elastix | Klein, S., Staring, M., Murphy, K., Viergever, M. A., & Pluim, J. P. (2009). Elastix: a toolbox for intensity-based medical image registration. IEEE transactions on medical imaging, 29(1), 196–205. [8] |
ITK | McCormick, M. M., Liu, X., Ibanez, L., Jomier, J., & Marion, C. (2014). ITK: enabling reproducible research and open science. Frontiers in neuroinformatics, 8, 13. [27] |
LSAWSIFT | Not available. |
RVSS | Not available. |
StackReg | Thevenaz, P., Ruttimann, U. E., & Unser, M. (1998). A pyramid approach to subpixel registration based on intensity. IEEE transactions on image processing, 7(1), 27–41. [33] |
TrakEM2 | Cardona, A., Saalfeld, S., Schindelin, J., Arganda-Carreras, I., Preibisch, S., Longair, M., … & Douglas, R. J. (2012). TrakEM2 software for neural circuit reconstruction. PloS one, 7(6), e38011. |
DS4H-IA | Piccinini, F., Duma, M.E. Tazzari, M., Pyun, J-C, Martinelli, G., Castellani, G., Carbonaro, A. (2022). DS4H Image Alignment: an user-friendly open-source ImageJ/Fiji plugin for aligning multimodality/IHC/IF 2D microscopy images. Submitted to Sensors. |
3. Data Science for Health Image Alignment (DS4H-IA)
3.1. Registration—Via Corner Points
3.2. Registration—Automatic Modality
4. Experiments
4.1. DS4H-IA Validation with Real-World Images
4.2. DS4H-IA Validation with Synthetically Generated Images
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Klein, A.; Andersson, J.; Ardekani, B.A.; Ashburner, J.; Avants, B.; Chiang, M.-C.; Christensen, G.E.; Collins, D.L.; Gee, J.; Hellier, P.; et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. NeuroImage 2009, 46, 786–802. [Google Scholar] [CrossRef] [PubMed]
- Musumeci, G. Past, present and future: Overview on histology and histopathology. J. Histol. Histopathol. 2014, 1, 5. [Google Scholar] [CrossRef]
- Bruni, D.; Angell, H.K.; Galon, J. The immune contexture and Immunoscore in cancer prognosis and therapeutic efficacy. Nat. Rev. Cancer 2020, 20, 662–680. [Google Scholar] [CrossRef] [PubMed]
- Caplan, J.; Niethammer, M.; Taylor, R.M., II; Czymmek, K.J. The power of correlative microscopy: Multi-modal, multi-scale, multi-dimensional. Curr. Opin. Struct. Biol. 2011, 21, 686–693. [Google Scholar] [CrossRef]
- Srivastava, R.; Varner, J. Emerging technologies: Systems biology. Biotechnol. Prog. 2007, 23, 24–27. [Google Scholar] [CrossRef]
- Bashiri, F.S.; Baghaie, A.; Rostami, R.; Yu, Z.; D’souza, R.M. Multi-modal medical image registration with full or partial data: A manifold learning approach. J. Imaging 2018, 5, 5. [Google Scholar] [CrossRef] [PubMed]
- Zhao, S. Multi-sensor Image Registration for Precisely Locating Time-sensitive Objects. In Proceedings of the 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC), Xiamen, China, 5–7 July 2019; pp. 144–148. [Google Scholar]
- Klein, S.; Staring, M.; Murphy, K.; Viergever, M.A.; Pluim, J.P.W. elastix: A toolbox for intensity-based medical image registration. IEEE Trans. Med. Imaging 2009, 29, 196–205. [Google Scholar] [CrossRef]
- Rohde, F.; Braumann, U.D.; Schmidt, M. Correlia: An ImageJ plug-in to co-register and visualise multimodal correlative micrographs. J. Microsc. 2020, 280, 3–11. [Google Scholar] [CrossRef] [PubMed]
- Fu, Y.; Lei, Y.; Wang, T.; Curran, W.J.; Liu, T.; Yang, X. Deep learning in medical image registration: A review. Phys. Med. Biol. 2020, 65, 20TR01. [Google Scholar] [CrossRef]
- Slomka, P.J.; Baum, R.P. Multimodality image registration with software: State-of-the-art. Eur. J. Nucl. Med. 2008, 36, 44–55. [Google Scholar] [CrossRef]
- Kiebel, S.J.; Ashburner, J.; Poline, J.B.; Friston, K.J. MRI and PET coregistration—A cross validation of statistical parametric mapping and automated image registration. Neuroimage 1997, 5, 271–279. [Google Scholar] [CrossRef] [PubMed]
- Sjollema, K.A.; Schnell, U.; Kuipers, J.; Kalicharan, R.; Giepmans, B.N. Correlated light microscopy and electron mi-croscopy. Methods Cell Biol. 2012, 111, 157–173. [Google Scholar]
- Piccinini, F.; Bevilacqua, A.; Lucarelli, E. Automated image mosaics by non-automated light microscopes: The MicroMos software tool. J. Microsc. 2013, 252, 226–250. [Google Scholar] [CrossRef]
- Brand, M.V.D.; Hoevenaars, B.M.; Sigmans, J.H.M.; Meijer, J.W.R.; van Cleef, P.H.J.; Groenen, P.J.; Hebeda, K.M.; van Krieken, J.H.J.M. Sequential immunohistochemistry: A promising new tool for the pathology laboratory. Histopathology 2014, 65, 651–657. [Google Scholar] [CrossRef]
- Tan, W.C.C.; Nerurkar, S.N.; Cai, H.Y.; Ng, H.H.M.; Wu, D.; Wee, Y.T.F.; Lim, J.C.T.; Yeong, J.; Lim, T.K.H. Overview of multiplex immunohistochemistry/immunofluorescence techniques in the era of cancer immunotherapy. Cancer Commun. 2020, 40, 135–153. [Google Scholar] [CrossRef]
- Abràmoff, M.D.; Magalhães, P.J.; Ram, S.J. Image processing with ImageJ. Biophotonics Int. 2004, 11, 36–42. [Google Scholar]
- Schindelin, J.; Arganda-Carreras, I.; Frise, E.; Kaynig, V.; Longair, M.; Pietzsch, T.; Preibisch, S.; Rueden, C.; Saalfeld, S.; Schmid, B.; et al. Fiji: An open-source platform for biological-image analysis. Nat. Methods 2012, 9, 676–682. [Google Scholar] [CrossRef] [PubMed]
- Bulgarelli, J.; Tazzari, M.; Granato, A.M.; Ridolfi, L.; Maiocchi, S.; de Rosa, F.; Petrini, M.; Pancisi, E.; Gentili, G.; Vergani, B.; et al. Dendritic cell vaccination in metastatic melanoma turns “non-T cell inflamed” into “T-cell inflamed” tumors. Front. Immunol. 2019, 10, 2353. [Google Scholar] [CrossRef]
- Linkert, M.; Rueden, C.T.; Allan, C.; Burel, J.-M.; Moore, W.; Patterson, A.; Loranger, B.; Moore, J.; Neves, C.; MacDonald, D.; et al. Metadata matters: Access to image data in the real world. J. Cell Biol. 2010, 189, 777–782. [Google Scholar] [CrossRef]
- Arganda-Carreras, I.; Sorzano, C.O.; Marabini, R.; Carazo, J.M.; Ortiz-de-Solorzano, C.; Kybic, J. Consistent and elastic registration of histological sections using vector-spline regularization. In Proceedings of the International Workshop on Computer Vision Ap-proaches to Medical Image Analysis, Graz, Austria, 12 May 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 85–95. [Google Scholar]
- Tumedei, M.M.; Piccinini, F.; Azzali, I.; Pirini, F.; Bravaccini, S.; De Matteis, S.; Agostinelli, C.; Castellani, G.; Zanoni, M.; Cortesi, M.; et al. Follicular Lymphoma Microenvironment Traits Associated with Event-Free Survival. Int. J. Mol. Sci. 2023, 24, 9909. [Google Scholar] [CrossRef] [PubMed]
- Bogovic, J.A.; Hanslovsky, P.; Wong, A.; Saalfeld, S. Robust registration of calcium images by learned contrast synthesis. In Proceedings of the 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), Prague, Czech Republic, 13–16 April 2016; pp. 1123–1126. [Google Scholar]
- Schmidt, M.; Rohde, F.; Braumann, U.D. Visualization and co-registration of correlative microscopy data with the ImageJ plug-in Correlia. Methods Cell Biol. 2021, 162, 353–388. [Google Scholar] [PubMed]
- Paul-Gilloteaux, P.; Heiligenstein, X.; Belle, M.; Domart, M.C.; Larijani, B.; Collinson, L.; Raposo, G.; Salamero, J. eC-CLEM: Flexible multidimensional registration software for correlative microscopies. Nat. Methods 2017, 14, 102–103. [Google Scholar] [CrossRef]
- de Chaumont, F.; Dallongeville, S.; Chenouard, N.; Hervé, N.; Pop, S.; Provoost, T.; Meas-Yedid, V.; Pankajakshan, P.; Lecomte, T.; Le Montagner, Y.; et al. Icy: An open bioimage informatics platform for extended reproducible research. Nat. Methods 2012, 9, 690–696. [Google Scholar] [CrossRef] [PubMed]
- McCormick, M.; Liu, X.; Jomier, J.; Marion, C.; Ibanez, L. ITK: Enabling reproducible research and open science. Front. Neurosci. 2014, 8, 13. [Google Scholar] [CrossRef]
- Pieper, S.; Halle, M.; Kikinis, R. 3D Slicer. In Proceedings of the 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro, Arlington, VA, USA, 15–18 April 2004; pp. 632–635. [Google Scholar]
- Marstal, K.; Berendsen, F.; Staring, M.; Klein, S. SimpleElastix: A user-friendly, multi-lingual library for medical image registration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, USA, 27–30 June 2016; pp. 134–142. [Google Scholar]
- Lowekamp, B.C.; Chen, D.T.; Ibáñez, L.; Blezek, D. The design of SimpleITK. Front. Neuroinformat. 2013, 7, 45. [Google Scholar] [CrossRef]
- Lowe, D.G. Distinctive Image Features from Scale-Invariant Keypoints. Int. J. Comput. Vis. 2004, 60, 91–110. [Google Scholar] [CrossRef]
- Cardona, A.; Saalfeld, S.; Schindelin, J.; Arganda-Carreras, I.; Preibisch, S.; Longair, M.; Tomancak, P.; Hartenstein, V.; Douglas, R.J. TrakEM2 Software for Neural Circuit Reconstruction. PLoS ONE 2012, 7, e38011. [Google Scholar] [CrossRef]
- Thevenaz, P.; Ruttimann, U.; Unser, M. A pyramid approach to subpixel registration based on intensity. IEEE Trans. Image Process. 1998, 7, 27–41. [Google Scholar] [CrossRef]
- Saalfeld, S.; Fetter, R.; Cardona, A.; Tomancak, P. Elastic volume reconstruction from series of ultra-thin microscopy sections. Nat. Methods 2012, 9, 717–720. [Google Scholar] [CrossRef]
- Levin, D. The approximation power of moving least-squares. Math. Comput. 1998, 67, 1517–1531. [Google Scholar] [CrossRef]
- Fischler, M.A.; Bolles, R.C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 1981, 24, 381–395. [Google Scholar] [CrossRef]
- Nealen, A. An As-Short-As-Possible Introduction to the Least Squares, Weighted Least Squares and Moving Least Squares Methods for Scattered Data Approximation and Interpolation. 2004. Available online: http://www.nealen.com/projects (accessed on 9 May 2023).
- Schaefer, S.; McPhail, T.; Warren, J. Image deformation using moving least squares. ACM Trans. Graph. 2006, 25, 533–540. [Google Scholar] [CrossRef]
- Bay, H.; Tuytelaars, T.; Van Gool, L. Surf: Speeded up robust features. In Proceedings of the Computer Vision—ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, 7–13 May 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 404–417. [Google Scholar]
- Bradski, G. The openCV library. Dr. Dobb’s J. Softw. Tools Prof. Program. 2000, 25, 120–123. [Google Scholar]
AIBLROI | BigWarp | Correlia | ec-CLEM | elastix | ITK | LSAWSIFT | RVSS | StackReg | TrakEM2 | DS4H-IA | |
VERSION | |||||||||||
Year of first release | 2006 | 2016 | 2020 | 2017 | 2010 | 1999 | 2008 | 2009 | 2010 | 2005 | 2022 |
Current version | O | 7.0.5 | 1.0 | 1.0.1.5 | 5.0.1 | 5.2.1 | 28 October 2018 | 3.0.7 | 7 July 2011 | 1.3.6 | 1.0 |
DOCUMENTATION | |||||||||||
User guide | X | X | X | X | X | X | O | X | X | X | X |
Website | X | X | X | X | X | X | O | X | X | X | X |
Video tutorial | O | X | O | X | X | X | O | O | O | X | X |
Sample dataset | O | O | X | O | X | X | O | O | X | X | X |
Open source | X | X | X | X | X | X | X | X | X | X | X |
Implementation language | Java | Java | Java | Java | C++ | C++ | Java | Java | Java | Java | Java |
USABILITY | |||||||||||
Input image format | All common | All common | All common | All common | All common | All common | All common | All common | All common | All common | All common |
No programming experience required | X | X | X | X | O | O | X | X | X | X | X |
User-friendly GUI | X | X | O | O | O | O | O | X | X | O | X |
Intuitive visualisation settings | X | O | O | O | O | O | O | O | X | O | X |
No commercial licences required | X | X | X | X | X | X | X | X | X | X | X |
Portability on Win/Linux/Mac | X | X | X | X | X | X | X | X | X | X | X |
FUNCTIONALITY | |||||||||||
Manual registration | X | X | X | X | O | X | O | O | X | X | X |
Automatic registration | O | O | X | X | X | X | X | X | X | X | X |
Image scale correction | X | X | X | X | X | X | O | X | X | X | X |
Image rotation correction | X | X | X | X | X | X | O | X | X | X | X |
Elastic correction | O | X | X | X | X | X | O | X | O | X | O |
Multiple image handling | O | O | X | O | O | X | X | X | X | X | X |
Multichannel/RGB image handling | O | X | X | X | X | X | X | X | X | X | X |
OUTPUT | |||||||||||
Resized aligned images | X | X | X | X | X | X | X | X | X | X | X |
Full-sized aligned images | O | O | O | O | O | O | O | X | O | O | X |
Registration parameters | O | X | X | X | X | X | O | X | X | X | X |
Editable result | O | X | X | X | X | X | O | X | O | X | X |
PositionA | PositionB | PositionC | PositionD | PositionE | |||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | ||
DatasetA - PositionA | Cy3 | X | X | X | X | X | X | O | X | X | X | O | X | X | X | X | X | X | X | X | X |
FITC | X | X | X | X | X | X | O | X | X | X | O | X | X | X | X | O | X | X | X | X | |
DIC | O | O | X | O | O | O | O | O | O | O | X | O | O | O | X | O | O | O | X | O | |
DAPI | X | O | X | X | X | O | O | X | X | O | O | X | X | O | X | X | X | O | X | X | |
DatasetB - PositionA | Cy3 | X | X | O | O | X | O | O | O | X | X | O | O | O | O | O | O | X | X | O | O |
FITC | X | X | O | O | O | X | O | O | O | O | O | X | X | O | O | O | X | X | O | O | |
DIC | O | O | X | O | O | O | X | O | O | O | X | O | O | O | O | O | O | O | X | O | |
Cy5 | O | O | O | X | O | O | O | O | O | O | O | X | O | O | O | X | O | O | O | O | |
DatasetC - PositionA | Cy3 | X | X | O | X | X | X | O | O | X | X | O | O | X | X | O | X | X | X | O | X |
FITC | X | X | O | O | X | X | O | X | X | X | O | X | X | X | O | X | X | X | O | O | |
DIC | O | O | X | O | O | O | X | O | O | O | O | O | O | O | O | O | O | O | O | X | |
Cy5 | O | O | O | X | O | O | O | X | O | X | O | X | O | O | O | X | X | X | O | O |
PositionA | PositionB | PositionC | PositionD | PositionE | |||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | Cy3 | FITC | DIC | Cy5/DAPI | ||
DatasetA—PositionA | Cy3 | X | X | O | X | X | X | O | X | X | X | O | X | X | X | O | X | X | X | O | X |
FITC | X | X | O | X | X | X | O | O | X | X | O | X | X | X | O | O | X | X | O | X | |
DIC | O | O | X | O | O | O | O | O | O | O | O | O | O | O | O | O | O | O | O | O | |
DAPI | X | O | O | X | X | O | O | X | X | O | O | X | X | O | O | X | X | O | O | X | |
DatasetB—PositionA | Cy3 | X | X | O | O | X | O | X | O | X | O | O | O | X | O | X | O | X | X | O | O |
FITC | X | X | X | O | X | X | O | O | O | X | X | O | O | X | X | O | X | X | O | O | |
DIC | O | X | X | O | O | O | X | O | O | O | X | O | O | O | X | O | O | O | X | O | |
Cy5 | O | O | O | X | O | O | O | X | O | O | O | X | O | O | O | X | O | O | O | X | |
DatasetC—PositionA | Cy3 | X | X | O | O | X | X | O | O | X | X | O | O | X | X | O | O | X | X | O | O |
FITC | X | X | O | O | X | X | O | O | X | X | O | O | X | X | O | O | X | X | O | O | |
DIC | O | X | X | O | O | O | X | O | O | O | X | O | O | O | X | O | O | O | X | O | |
Cy5 | O | O | O | X | O | O | O | X | O | O | O | X | O | O | O | X | O | O | O | X |
75% Overlap (Mean ± Std) | 50% Overlap | 25% Overlap | ||||
---|---|---|---|---|---|---|
SIFT | SURF | SIFT | SURF | SIFT | SURF | |
DatasetD | 0.01 ± 0.4 | 0.01 ± 0.4 | 0.02 ± 0.4 | 0.02 ± 0.4 | 0.03 ± 0.4 | 0.03 ± 0.4 |
DatasetE | 0.02 ± 0.5 | 0.01 ± 0.5 | 0.03 ± 0.5 | 0.03 ± 0.5 | 0.04 ± 0.5 | 0.05 ± 0.5 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Piccinini, F.; Tazzari, M.; Tumedei, M.M.; Stellato, M.; Remondini, D.; Giampieri, E.; Martinelli, G.; Castellani, G.; Carbonaro, A. Data Science for Health Image Alignment: A User-Friendly Open-Source ImageJ/Fiji Plugin for Aligning Multimodality/Immunohistochemistry/Immunofluorescence 2D Microscopy Images. Sensors 2024, 24, 451. https://doi.org/10.3390/s24020451
Piccinini F, Tazzari M, Tumedei MM, Stellato M, Remondini D, Giampieri E, Martinelli G, Castellani G, Carbonaro A. Data Science for Health Image Alignment: A User-Friendly Open-Source ImageJ/Fiji Plugin for Aligning Multimodality/Immunohistochemistry/Immunofluorescence 2D Microscopy Images. Sensors. 2024; 24(2):451. https://doi.org/10.3390/s24020451
Chicago/Turabian StylePiccinini, Filippo, Marcella Tazzari, Maria Maddalena Tumedei, Mariachiara Stellato, Daniel Remondini, Enrico Giampieri, Giovanni Martinelli, Gastone Castellani, and Antonella Carbonaro. 2024. "Data Science for Health Image Alignment: A User-Friendly Open-Source ImageJ/Fiji Plugin for Aligning Multimodality/Immunohistochemistry/Immunofluorescence 2D Microscopy Images" Sensors 24, no. 2: 451. https://doi.org/10.3390/s24020451