Next Article in Journal
ZnO Functional Nanomaterial in Green Microalgae Growth Advancement
Previous Article in Journal
Application of Primary Network Analysis in Real Water Distribution Systems
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Proceeding Paper

Chip-Sized Microscopy for Continuous Monitoring: Application in White Wine Fermentation and Yeast Cell Counting via Deep Learning †

Electronic and Biomedical Engineering Department, University of Barcelona, C/Martí Franquès, 1, 08028 Barcelona, Spain
*
Author to whom correspondence should be addressed.
Presented at the 1st International Conference on AI Sensors & the 10th International Symposium on Sensor Science, Singapore, 1–4 August 2024.
Eng. Proc. 2024, 78(1), 1; https://doi.org/10.3390/engproc2024078001
Published: 8 October 2024

Abstract

Nowadays, continuous monitoring is a difficult issue in microscopy. A chip-sized microscope was developed, composed only of microelectronic components, with high optical resolution and a wide field of view. Due to its miniaturized size, it can be placed on or attached to the sample for continuous monitoring in the sample environment. An example of an application of this microscope for the food and beverage industry is described, referring to the study of the fermentation process of white wine. The comparison of the images acquired with conventional optical microscopy reveals similar results. To automatically count yeast cells, the traditional image postprocessing is compared with deep learning. Neural networks achieve similar cell recognition characteristics but with an ~100× speed improvement, by directly processing the obtained holograms.

1. Introduction

Over the years, microscopy has undergone a remarkable evolution, driven by the current needs and technological advances in this field. In current practice, researchers follow set steps for measuring samples in laboratories. Initially, they collect the fractions of a sample to be analyzed. The samples are then prepared according to protocols to ensure that they are in the necessary conditions for a detailed analysis. The next step is to fix the prepared samples to a plate, to be observed under a microscope in the laboratory. With the help of the microscope, researchers can extract the data and information relevant to their research. However, continuous monitoring is impossible with this procedure.
Recently, we developed a chip-sized microscope, which is based on microelectronic technology, with high optical resolution and a wide field of view [1,2,3,4]. Due to its miniaturized components, this device is compact and low-cost compared to other microscopy technologies on the market, making it suitable for mass production. Moreover, this microscope has a unique feature compared to others, as it can be placed directly on the sample to be examined, allowing for continuous and parallel analyses. This capability transforms the conventional paradigm of “collecting fractions of samples to be observed in a laboratory with a microscope”, to a new scenario where you can “bring the instrument to the sample in the field and observe it thoroughly”.
In this paper, we present different applications where this microscope can be of interest. Then, we discuss the applicability of the microscope to studying the fermentation process of wine, including a preliminary solution to account for the fermentation status based on artificial intelligence.

2. Background

Microscopes are used extensively to analyze biological and geological specimens. They have several characteristics in common, e.g., microscopes are big instruments and often quite expensive. Moreover, every microscope is operated in a similar way, i.e., the samples are taken from their experimental environment and placed on the sample holder of the microscope for observation. So, we need to interrupt the normal procedure of the experiment in order to make every observation. Such a mode of observation has several problems. The instrument can only be used to observe one sample at a time. However, to assess how an experiment evolves, observations are required at different times. This makes the process of observation unscalable. Additionally, microscopes are managed by humans, and we are prone to errors during the measurement and during the interpretation of the measurements. Also, the samples are often damaged during observation. Moreover, most of the time, sample observation requires a specific sample conditioning. All these issues imply additional costs. Finally, preparing the sample and observing it under the microscope is a time-consuming activity.

3. Purpose of the Research

We propose a compact microscope that can be placed surrounding the sample, grasping it. Then, the images acquired can be sent to the cloud, to be processed by artificial intelligence. Developing machine learning algorithms for every experiment will help in the decision-making process related to that experiment, eliminating possible human errors. Having a microscope in every experiment and for every sample can assist in the advancement of several scientific and commercial fields. We identified some of them, mainly consisting of microfluidic devices and cell counters.
In the first field, we can identify cell and tissue cultures. These cultures are commonly developed after many attempts, which need to be assessed. In traditional microscopy, the sample must be removed from the incubator for imaging, potentially disrupting its environment. However, with the miniature microscope we have developed, samples can remain in the incubator, allowing for continuous monitoring without disturbing the conditions. During the study, the microscope captures images of the sample in its natural environment, eliminating the need to repeatedly take the sample out and place it on a microscope plate. This continuous, non-invasive imaging helps to avoid potential damage to the specimen, although reagents such as culture media may still be needed to maintain its viability. The advantages of using our miniature microscope include the ability to observe the sample in situ within the incubator (Figure 1), continuous monitoring through of the experiment, parallelizing measurements by applying a microscope to each sample, and collecting large amounts of real-time data during the process, among other benefits.
In the second group, we can identify many applications in the heath sector and food and beverage industry, such as the continuous monitoring of wine making that can be carried out at different stages of production (Figure 2). The art of winemaking is fully based on the fermentation process, which involves yeast converting sugars into alcohol [5,6]. A good yeast must be selected first, free of any contaminant. Then, the fermentation needs to be controlled until the sugar is completely transformed into alcohol, and this can be observed by measuring the increase in the number of yeast cells. The usual procedure for cellars involves periodically taking samples of wine and sending them to a specific center that uses microscopy to analyze the density of yeast cells in the samples. The rate at which the number of cells increases is related to the quality of the final wine. For example, a slow fermentation can be the result of the presence of a toxin [7]. The purpose of the next section is to explain our experience with our compact microscope when analyzing the number of yeast cells in the process, compared with standard microscopy techniques. In the future, the microscope can be used after grape harvest and pressing (Figure 2a), integrated into the tank (Figure 2b) or the bottle (Figure 2c).

4. Results—Case Study Determination of Cells in Partially Fermented Samples of Wine

The example presented next refers to a white wine that was produced with the yeast of Saccharomyces Cerevisiae from a strain in Catalonia, Spain (S. cerevisiae P29, Catalan Institute of Vine and Wine—INCAVI, Vilafranca del Penedes, Catalonia, Spain). We have compared images obtained with a conventional microscope with the images captured with our prototype. The prototype is presented in Figure 3. It consists of a LED microdisplay (Jade Bird Display, Shanghai, China, JBD 0.13” display, 2 μm pixel size, 4 μm pixel pitch, 640 × 480 LEDs) in the bottom, a CMOS camera (TIS-DMM 37UX178-ML, Iberoptics Sistemas Ópticos, Madrid, Spain, with 2.4 μm pixel size, 8.92 mm diagonal, 3096 × 2080 pixels) on the top and the sample in between both components. The microscope is operated by switching on one LED and acquiring a hologram with the CMOS sensor. To increase the field of view, several LEDs are switched on sequentially, and then the corresponding holograms are stitched by using the well-known SIFT algorithm. Images are obtained after stitching by applying the light backpropagation and twin-image-removal algorithms.
The liquid sample is inserted through the inlet of a microslide (μ-Slide 0.2 Luer Glass Bottom from Ibidi, Inycom, Zaragoza, Spain), shown in Figure 3. This microslide has a glass at the bottom with a thickness of ~170 μm. It is important, as with our microscope, that the maximum resolution (2 μm) is obtained with the minimum distance of the sample to the sourcing LED light [4], and this is achieved thanks to the state-of-the-art microdisplay used, with micrometer sized LEDs. The microfluidic channel is 250 μm thick. A wide field of ~5 mm2 is obtained by stitching images gathered by the separate LEDs from the microdisplay.
In Figure 4, the comparison of images acquired with a conventional confocal microscope (Figure 4a) and our microscope (Figure 4b) is presented. As can be observed, we obtain a hologram corresponding to the specimen inserted in the microscope, as we did not include any lens. Such a hologram is a superposition of stitched images from 25 different LEDs. After applying a light backpropagation algorithm and twin image removal to this hologram, the image of Figure 4c, corresponding to the plane of the image, is obtained. In this image, we can measure the density of cells by applying an automated method. For example, by applying Hough Transform, the complete process, up to counting the cells, including the described procedure, is roughly 4 min. Comparing with conventional microscopy, the detection range is very similar to conventional microscopes, from 0.5 to 5 million cells per mL.
In our future research, we want to dynamically quantify yeast cell density by measuring cells flowing in a microfluidic channel attached to the wine tank. With this, we expect to improve the detection limit for this experiment. However, to pursue direct real-time observation, it is necessary to decrease the acquisition and processing time of the images captured by the microscope. Another approach to measure the density of cells, without having to use the computationally expensive backpropagation of the holograms, is to use specialized neural networks and clusterization. For that purpose, we extract relevant data directly form the holograms. Figure 5a presents a hologram captured with our microscope. The clusters reconstructed with a neural network are presented in Figure 5b. Figure 5c presents the projected boxes over the backpropagated image. In the resulting image, we can identify 48 clusters in just 400 ms instead of the four minutes required by the stitching, hologram backpropagation, twin image removal and automatic identification by Hought Transform. The network used was a deep neural network based on mobilenet [8]. We required a simple neural network to be able to translate it to hardware in the near future. In order to train the network, we generated more than 14,000 synthetic images. These images were obtained considering cells in the channel that were propagated to the plane at which we have the camera to get the holograms.

5. Conclusions

The application of a compact microscope composed only of a GaN-LED microdisplay, a CMOS camera and a package was demonstrated by studying different fermentation stages of a white wine sample with a Saccharomyces Cerevisiae yeast. Images with a large FOV (~500 × 400 μm2) were obtained by stitching and postprocessing holograms, with a high-enough resolution to observe the yeast cells (diameter ~5–10 μm) in samples with a population from 0.5 to 5 million yeast cells per mL. As a method for faster postprocessing, a mobilenet-based deep neural network was used to extract the same information directly from the holograms, achieving a speed improvement from 4 min to 0.4 s. In the near future, and for this application, we plan to implement the neural network in hardware and perform measurements directly in the microchannel to increase the lower detection limit beyond conventional methods.

6. Patents

Optical detection device, its method for operating and computer program, WO2017005792A1 PCT 2017, Europe: EP3320567B1 Active, USA: US10201054B2 Active, Priority 2015/07/07. University of Barcelona and Technical University of Braunschweig.
Methods and device for obtaining a microscopic digital images, WO2023/280920A2 PCT, Priority 2021/07/06. University of Barcelona.

Author Contributions

Conceptualization, Á.D., J.D.P. and A.V.; Methodology Á.D., S.M., S.M.-M., J.C., I.B.-A., J.D. and A.V.; Software I.B.-A., J.D. and O.C.; Validation S.M., J.C., S.M.-M., I.B.-A. and Á.D.; Supervision Á.D., J.D.P. and A.V.; Draft preparation Á.D., J.D. and A.V.; Writing and revision Á.D. and A.V.; Funding acquisition Á.D., J.D. and A.V. All authors have read and agreed to the published version of the manuscript.

Funding

This work has received funding from Grants PID2019-105714RB-I00, PID2022-136833OB-C21 and PDC2023-145805-I00 funded by MICIU/AEI/10.13039/501100011033 and from the European Regional Development Fund and from the European Union NextGenerationEU/PRTR. This work also received funding from Grant PREP2022-000341 funded by MCIN and by “ESF Investing in your future”. Finally, this work has also received funding from FEADER and Departament d’Acció Climàtica, Alimentació i Agenda Rural (DACC) of the Generalitat de Catalunya with grant 56 30123 2021 2A form the budget AG06D/482000100/5741/0000.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Franch, N.; Canals, J.; Moro, V.; Vilá, A.; Romano-Rodríguez, A.; Prades, J.D.; Gülink, J.; Bezshlyakh, D.; Waag, A.; Kluczyk-Korch, K.; et al. Nano Illumination Microscopy: A Technique Based on Scanning with an Array of Individually Addressable NanoLEDs. Opt. Express 2020, 28, 19044. [Google Scholar] [CrossRef] [PubMed]
  2. Moreno, S.; Canals, J.; Moro, V.; Franch, N.; Vilà, A.; Romano-Rodriguez, A.; Prades, J.D.; Bezshlyakh, D.D.; Waag, A.; Kluczyk-Korch, K.; et al. Pursuing the Diffraction Limit with Nano-LED Scanning Transmission Optical Microscopy. Sensors 2021, 21, 3305. [Google Scholar] [CrossRef]
  3. Vilà, A.; Moreno, S.; Canals, J.; Diéguez, A. A Compact Raster Lensless Microscope Based on a Microdisplay. Sensors 2021, 21, 5941. [Google Scholar] [CrossRef]
  4. Vilà, A.; Moreno, S.; Diéguez, A. 24-1: Can Light Microscopes Really Be Chip-Sized? SID Symp. Dig. Tech. Pap. 2022, 53, 271–274. [Google Scholar] [CrossRef]
  5. Chambers, P.J.; Pretorius, I.S. Fermenting Knowledge: The History of Winemaking, Science and Yeast Research. EMBO Rep. 2010, 11, 914–920. [Google Scholar] [CrossRef] [PubMed]
  6. Alba-Lois, L.; Segal-Kischinevzky, C. Yeast Fermentation and the Making of Beer and Wine. Nat. Educ. 2010, 3, 17. [Google Scholar]
  7. Bisson, L.F. Stuck and Sluggish Fermentations. Am. J. Enol. Vitic. 1999, 50, 107–119. [Google Scholar] [CrossRef]
  8. Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv 2017, arXiv:1704.04861. [Google Scholar]
Figure 1. Concept for testing hundreds of organ-on-a-chip devices inside the incubator by using a microscope for every experiment. The proposed compact microscope prototype is shown in the inset, and multiple microscopes can be placed inside the incubator.
Figure 1. Concept for testing hundreds of organ-on-a-chip devices inside the incubator by using a microscope for every experiment. The proposed compact microscope prototype is shown in the inset, and multiple microscopes can be placed inside the incubator.
Engproc 78 00001 g001
Figure 2. Concept for analyzing the fermentation process in different stages of production with a compact microscope: (a) after harvesting, crushing and pressing, (b) during the first fermentation of red/white wines, and (c) analyzing the microbiological quality of bottled wine or for the second fermentation in the case of champagne/cava or spirit wines.
Figure 2. Concept for analyzing the fermentation process in different stages of production with a compact microscope: (a) after harvesting, crushing and pressing, (b) during the first fermentation of red/white wines, and (c) analyzing the microbiological quality of bottled wine or for the second fermentation in the case of champagne/cava or spirit wines.
Engproc 78 00001 g002
Figure 3. (a) Microscope drawing and (b) microscope prototype and procedure for wine sample insertion.
Figure 3. (a) Microscope drawing and (b) microscope prototype and procedure for wine sample insertion.
Engproc 78 00001 g003
Figure 4. Images of the intermediate stage of the wine fermentation: (a) confocal microscopy image; (b) hologram obtained by our microscope; (c) image of the sample after image processing of (b).
Figure 4. Images of the intermediate stage of the wine fermentation: (a) confocal microscopy image; (b) hologram obtained by our microscope; (c) image of the sample after image processing of (b).
Engproc 78 00001 g004
Figure 5. An alternative reconstruction by using an artificial neural network directly from holograms. (a) Original hologram. (b) Produced segmentation mask, as a representation of an inferred backpropagated image. (c) Projected boxes over the backpropagated image. The recall with synthetic images was ~0.9.
Figure 5. An alternative reconstruction by using an artificial neural network directly from holograms. (a) Original hologram. (b) Produced segmentation mask, as a representation of an inferred backpropagated image. (c) Projected boxes over the backpropagated image. The recall with synthetic images was ~0.9.
Engproc 78 00001 g005
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Diéguez, Á.; Moreno, S.; Moncada-Madrazo, S.; Caravaca, O.; Diéguez, J.; Canals, J.; Benito-Altamirano, I.; Prades, J.D.; Vilà, A. Chip-Sized Microscopy for Continuous Monitoring: Application in White Wine Fermentation and Yeast Cell Counting via Deep Learning. Eng. Proc. 2024, 78, 1. https://doi.org/10.3390/engproc2024078001

AMA Style

Diéguez Á, Moreno S, Moncada-Madrazo S, Caravaca O, Diéguez J, Canals J, Benito-Altamirano I, Prades JD, Vilà A. Chip-Sized Microscopy for Continuous Monitoring: Application in White Wine Fermentation and Yeast Cell Counting via Deep Learning. Engineering Proceedings. 2024; 78(1):1. https://doi.org/10.3390/engproc2024078001

Chicago/Turabian Style

Diéguez, Ángel, Sergio Moreno, Sofía Moncada-Madrazo, Oriol Caravaca, Joel Diéguez, Joan Canals, Ismael Benito-Altamirano, Juan Daniel Prades, and Anna Vilà. 2024. "Chip-Sized Microscopy for Continuous Monitoring: Application in White Wine Fermentation and Yeast Cell Counting via Deep Learning" Engineering Proceedings 78, no. 1: 1. https://doi.org/10.3390/engproc2024078001

APA Style

Diéguez, Á., Moreno, S., Moncada-Madrazo, S., Caravaca, O., Diéguez, J., Canals, J., Benito-Altamirano, I., Prades, J. D., & Vilà, A. (2024). Chip-Sized Microscopy for Continuous Monitoring: Application in White Wine Fermentation and Yeast Cell Counting via Deep Learning. Engineering Proceedings, 78(1), 1. https://doi.org/10.3390/engproc2024078001

Article Metrics

Back to TopTop