Next Article in Journal
Experiences of Diabetes Self-Management: A Focus Group Study among the Middle-Aged Population of Rural Pakistan with Type 2 Diabetes
Previous Article in Journal
Urinary N-Acetyl-β-d-glucosaminidase (uNAG) as an Indicative Biomarker of Early Diabetic Nephropathy in Patients with Diabetes Mellitus (T1DM, T2DM): A Systematic Review and Meta-Analysis
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Study Protocol

Implementation and Evaluation of a Mobile Retinal Image Acquisition System for Screening Diabetic Retinopathy: Study Protocol

Fraunhofer Portugal Research Center for Assistive Information and Communication Solutions (Fraunhofer AICOS), Rua Alfredo Allen 455-461, 4200-135 Porto, Portugal
Faculty of Medicine, University of Porto, Alameda Prof. Hernâni Monteiro, 4200-319 Porto, Portugal
Department of Community Medicine, Health Information and Decision, Faculty of Medicine, University of Porto, Rua Dr. Plácido da Costa, 4200-450 Porto, Portugal
Center for Health Technology and Services Research, Faculty of Medicine of the University of Porto, Rua Dr. Plácido da Costa, 4200-450 Porto, Portugal
Portuguese Red Cross School of Health Lisbon, Jardim 9 de Abril, 1-5, 1249-083 Lisbon, Portugal
Department of Ophthalmology, Central Lisbon Hospital Center, Rua José António Serrano, 1150-199 Lisbon, Portugal
NOVA Medical School, Universidade NOVA de Lisboa, Campo dos Mártires da Pátria 130, 1169-056 Lisbon, Portugal
Portuguese Diabetes Association (APDP), Rua Rodrigo da Fonseca 1, 1250-189 Lisbon, Portugal
Portuguese Retina Institute, Av. das Nações Unidas 29, 1600-531 Lisbon, Portugal
Chronic Diseases Research Center (CEDOC), NOVA Medical School, Universidade NOVA de Lisboa, Campo dos Mártires da Pátria 130, 1169-056 Lisbon, Portugal
Author to whom correspondence should be addressed.
Diabetology 2022, 3(1), 1-16;
Received: 17 November 2021 / Revised: 10 December 2021 / Accepted: 24 December 2021 / Published: 4 January 2022


Screening diabetic retinopathy, a major cause of blindness, is time-consuming for ophthalmologists and has some constrains in achieving full coverage and attendance. The handheld fundus camera EyeFundusScope was recently developed to expand the scale of screening, drawing on images acquired in primary care and telescreening made by ophthalmologists or a computer-aided diagnosis (CADx) system. This study aims to assess the diagnostic accuracy of the interpretation of images captured using EyeFundusScope and perform its technical evaluation, including image quality, functionality, usability, and acceptance in a real-world clinical setting. Physicians and nurses without training in ophthalmology will use EyeFundusScope to take pictures of the retinas of patients with diabetes and the images will be classified for the presence or absence of diabetic retinopathy and image quality by a panel of ophthalmologists. A subgroup of patients will also be examined with the reference standard tabletop fundus camera. Screening results provided by the CADx system on images taken with EyeFundusScope will be compared against the ophthalmologists’ analysis of images taken with the tabletop fundus camera. Diagnostic accuracy measures with 95% confidence intervals (CIs) will be calculated for positive and negative test results. Proportion of each category of image quality will be presented. Usability and acceptance results will be presented qualitatively.

1. Introduction

1.1. Background and Rationale

Diabetic retinopathy affects 34.6% of patients with diabetes and is the leading cause of blindness worldwide [1,2,3]. The disease is asymptomatic in early stages [4,5], so it is important to screen individuals at risk of developing this condition. Retinal image acquisition is highly technical and requires specialized technicians managing tabletop retinal cameras—large devices that need to be fixed and calibrated onto a table in a healthcare setting [6,7]. Then, all images are analysed by an ophthalmologist, and of those, only 11% show referable diabetic retinopathy, meaning that almost 90% of screened patients have mild or no signs of diabetic retinopathy and no diabetic macular oedema [8].
Despite the implemented screening programmes in different countries [9,10,11,12], less than half of individuals with diabetes receive retinal screenings [5,13,14,15]. Constrains related to lack of healthcare professionals, infrastructure, and costs have been reported [16]. Moreover, patients’ attendance rates range between 61% and 88.9% [17], for reasons such as lack of awareness or misinformation regarding the need for eye screening, not receiving screening invitation, and being unable to drive because of the effect of eye drops for pupil dilation [17,18]. This represents an important loss of opportunity to diagnose early and treat diabetic retinopathy [18], which can, ultimately, be responsible for patients’ vision loss and blindness [18]. The World Health Organization reinforces the need for screening of all individuals with diabetes [19]. As more and more patients are diagnosed with diabetes and as the human lifespan increases, healthcare services throughout the world need to adapt quickly to ensure that screenings are delivered [20].
In many diabetic retinopathy screening programmes, fundus images are acquired with tabletop fundus cameras and transferred electronically to a reading centre to be analysed by an ophthalmologist [6]. This telemedicine approach enables to overcome ophthalmologists’ shortage, as eye technicians—not ophthalmologists—perform image acquisition. However, this screening modality relies on patients’ uptake to screening appointments. Opportunistic screenings when patients visit their doctors may be an alternative to increase patient uptake. Moreover, evidence from research points out that diabetic retinopathy screening intervals could be personalized to each patient, based on an individual risk assessment [21,22,23]. Rather than inviting all patients with diabetes to an annual screening, intervals can be extended for some patients [21,22,23]. A reorganization of screening programmes will be needed, and a tabletop fundus camera located in a centralized healthcare unit for rigid scheduled screenings may not suit the new screening model; instead, a handheld fundus camera, permanently available in each primary healthcare unit and possible to operate by non-ophthalmologists and non-eye technicians, may be more appropriate. These healthcare professionals would be able to screen patients with diabetes when they attend a regular medical appointment, reaching the patients who would fail to attend a scheduled screening. Handheld fundus cameras—smaller and easier to transport than tabletop fundus cameras [6,23]—would in principle enable screenings at multiple settings: offices of family physicians, general practitioners, endocrinologists, and nurses, and at patients’ homes or community settings [8,24,25,26,27,28]. CellScope Retina (University of California Berkeley, Berkeley, CA, USA), Remidio Fundus on Phone (Remidio, Karnataka, India), D-EYE (Si14 SpA, Padova, Italy), VistaView (Volk Optical, Mentor, OH, USA), and EyeFundusScope (Fraunhofer AICOS, Porto, Portugal) are handheld fundus cameras that integrate a smartphone [29,30,31,32,33]. Other handheld fundus cameras, for example, PiCTOR Prestige (Volk Optical, Mentor, OH, USA), do not integrate a smartphone; instead, they have dedicated touchscreens and computing units [34].
Whether the images are taken with tabletop fundus cameras or handheld fundus cameras, the ophthalmologists’ workload is a challenge. Artificial intelligence (AI)-based software for diabetic retinopathy screening has the potential to automate image analysis and quickly identify pathologic features (microaneurysms, haemorrhages, and neovascularization) [34], reducing ophthalmologists’ workload. In recent years, many AI algorithms for diabetic retinopathy screening have been developed [35]. Retmarker (Retmarker Ltd., Coimbra, Portugal), iGradingM (EMIS Leeds, UK), EyeArt (Eyenuk Inc., Woodland Hills, CA, USA), and IDx-DR (IDx, LLC, Coralville, IA, USA) obtained a Conformité Européenne (CE) Mark [8], and only the latter also obtained approval from the Food and Drug Administration (FDA, Silver Spring, MD, USA) [34]. All of them are indicated for use with conventional tabletop retinal cameras [8,35]. Medios DR AI was developed to analyse images taken with a smartphone-based nonmydriatic retinal camera—Remidio Fundus on Phone (Remidio, Karnataka, India) [36]. However, its accuracy study was conducted using pupil dilation drops in 90.6% of patients [37]. Increasing the size of the pupils with eye drops increases the portion of the retina that can be seen in fundus photography, and hence increases the diagnostic accuracy for diabetic retinopathy [38,39,40]. However, ensuring good accuracy with a smartphone-based fundus camera without pupil dilation induced by eye drops is essential, since it potentially increases the coverage of the screening: eye drops is contraindicated in patients with narrow angle glaucoma, narrow filtration angle, and known hypersensitivity [41]; in addition, the effect of that medication in prohibiting driving is one of the main barriers to screening attendance [42].
The computer-aided diagnosis (CADx) system for diabetic retinopathy screening based on deep learning integrated with EyeFundusScope (Fraunhofer AICOS, Porto, Portugal) smartphone-based retinal camera was assessed on images acquired with EyeFundusScope [43] in a pilot study (in this study a commercial ophthalmoscope was connected to a smartphone) and showed high sensitivity and specificity (67% and 95% respectively) without pupil dilation drops [44]. The study was conducted with patients with known diabetic retinopathy at an ophthalmology outpatient clinic, using indirect ophthalmoscopy as the comparator. In another study, the same AI software was used to classify retinal images taken with tabletop fundus cameras [45]. Therefore, a comparison with the reference standard for screening (tabletop fundus camera) in a generalizable sample of individuals with diabetes, to assess the ability to discriminate patients with diabetic retinopathy from those without it [46] is lacking. Moreover, in that study, EyeFundusScope was operated by a highly trained technical professional [44]. However, since image acquisition with smartphone-based fundus cameras can be challenging for beginners, we need to assess the quality of images and the diagnostic accuracy of EyeFundusScope when operated by their potential future users—nurses and physicians without training in ophthalmology.
The usability of EyeFundusScope was not yet assessed among physicians and nurses without training in ophthalmology. In a previous study [47], physicians and nurses performed fundus image acquisition, and although usability measures were not collected, as usability was not the focus of that work, users expressed that EyeFundusScope was easy to use. However, subjects were healthy and young adults; in patients with difficulty to remain upright and follow instructions, the use of EyeFundusScope (or other retinal cameras) may be challenging.
Because the quality of retinal images is essential to enable proper analysis, it is important to assess whether the focus and field of view are adequate, as well as vessel visibility. Image acquisition in older people and with cataracts is difficult [48]. In the 80+ age group, technical failure rate is 41.6% without pupil dilation [48]. Furthermore, the proportion of ungradable images in people with central cataracts accounts for 57% of ungradable images, and the more the years after diagnosis of diabetes, the higher is the proportion of ungradable images [48]. Moreover, the skill of each operator influences the quality of images, and although we expect it to vary for each operator, we need to assess if these differences affect the AI classification for diabetic retinopathy [49].
New medical devices receive market authorization approval after provision of scientific evidence of safety and effectiveness. This information is generated in clinical research and is presented to the regulators acting in each country. With this study, we aim to provide that evidence for EyeFundusScope and its integrated CADx system. We will consider two scenarios of clinical use: (1) EyeFundusScope to acquire retinal images that are then classified for diabetic retinopathy by an ophthalmologist; and (2) images acquired with EyeFundusScope that are classified for diabetic retinopathy by the CADx system (Figure 1). Considerations for clinical implementation, namely real-world technical functionality of EyeFundusScope, its usability, acceptance, and integration in clinical workflow were not assessed yet and will be assessed in this study.

1.2. Objectives

1.2.1. Primary Objectives

The primary objective of this project is:
  • To assess the diagnostic accuracy of using EyeFundusScope alone and along with the CADx system for diabetic retinopathy classification.

1.2.2. Secondary Objectives

The secondary objectives of this project are as follows.
  • To assess the image quality and technological functioning of EyeFundusScope in real clinical settings.
  • To analyse the association of patients’ clinical characteristics with test accuracy and the quality of images acquired with EyeFundusScope.
  • To assess the inter and intraoperator agreement and reliability of EyeFundusScope for diabetic retinopathy classification.
  • To assess the usability and acceptance of EyeFundusScope.
  • To optimize the ground truth for training an AI software for diabetic retinopathy and image quality automated classification.

2. Methods

2.1. Study Design

This project comprises the following studies:
  • A diagnostic test accuracy cross-sectional study, in which a sample of individuals with diabetes will be prospectively included and screened for diabetic retinopathy by EyeFundusScope (index test) and a tabletop fundus camera (reference standard test); and
  • A pilot study using a handheld nonmydriatic fundus camera operated by physicians and nurses not specialists in ophthalmology in another sample of individuals—this study includes image quality, screening results agreement with these images, usability, and acceptance.

2.2. Eligibility Criteria, Study Setting, Recruitment, and Sample Size

A consecutive sample of individuals diagnosed with diabetes, aged 18–90 years, will be eligible to participate, after receiving information about the study and providing written informed consent form. Clinical researchers will approach eligible participants when they present at Hospital CUF Tejo (Lisbon, Portugal) for routine medical appointments or community assessments, and at ACeS Gondomar—ARS Norte (Porto, Portugal)—a primary healthcare unit—for diabetic retinopathy screenings. People with diabetes with the following conditions will be excluded from the study: blindness, strabismus, retinal detachment, blepharospasm, or no autonomy to remain seated—retinal imaging may be more difficult if these conditions are present: sensitivity to light due to medication, recent photodynamic therapy, or other—to avoid eye pain or discomfort—and pregnant women or breastfeeding, for ethical reasons. All eligible participants will undergo both the index test and the reference standard test. The Standards for Reporting of Diagnostic Accuracy Studies (STARD) diagram [50] will be presented, showing the number of patients assessed for eligibility and the number of non-consenting patients, as well as participants excluded, along with the respective reasons, and the number of patients screened with the index test and with the reference standard test.
The sample size (n = 578) was calculated on the basis of 95% CIs, 10% margin of error, expected sensitivity and specificity—81% and 96%, respectively [45]—and the prevalence of DR among people with diabetes of 13% [51]. Then, we inflated the sample size by an additional 25% (n = 116, total n = 462) to take account of ungradable images. Sample size calculation was performed at the subject level, since both eyes are required to obtain the reference standard result, considering the worst eye for the result for each subject.

2.3. Diagnostic Accuracy Study

2.3.1. Index Test

The index test consists in screening each participant for diabetic retinopathy using EyeFundusScope. Physicians and nurses without training in ophthalmology will complete a training session before they can start using EyeFundusScope; the study protocol and the EyeFundusScope instruction manual will be available to clinician participants during the study. The photography protocol consists of acquiring at least two retinal images of each eye—one centred on the macula and one centred on the optic disc. Acquiring images of both eyes is expected to take five minutes or less, with minimal impact on clinical workflow. Drops for pupil dilation will not be used. EyeFundusScope images will be classified for diabetic retinopathy by a panel of ophthalmologists blinded to each other. In addition, a subsample of participants will be photographed at least by two clinicians.

2.3.2. Reference Test

Tabletop fundus images will be used as the reference standard test for diabetic retinopathy, as this is the screening test in use in clinical practice and one of recommended tests by the World Health Organization [52]. The classification of images by ophthalmologists is insufficiently reliable [53,54,55], and this could bias the accuracy measures of the diagnostic test under study. To address this challenge, we will use a panel diagnosis as the reference standard: at least two ophthalmologists will independently classify images acquired with EyeFundusScope regarding diabetic retinopathy classification, blinded to the results of the index test. The ophthalmologists will also be masked to the clinical status and characteristics of participants (Appendix A, Table A1). Discordances will be solved based on consensus; otherwise, a third ophthalmologist will classify those images. In this study, we will adopt a modified version of the International Clinical Disease Severity Scale (ICDSS), which is in use at the study sites and with which ophthalmologists are familiar with (Table 1) [56].

2.4. Inter and Intraoperator Agreement and Reliability

Operating a smartphone-based fundus camera requires some skills that non-specialists in ophthalmology may lack. Variability between operators of a medical device may compromise screening results [49]. Considering 3 EyeFundusScope operators, k of 0.8, k maximum amplitude of 0.2, diabetic retinopathy prevalence of 20% and 95% confidence intervals, the sample size estimate for reproducibility calculations is 200 individuals with diabetes for the study of interoperator reproducibility, and 200 individuals with diabetes for the study of intraoperator reproducibility. To assess how similar the classification of images collected by different users are to each other, at least one physician and one nurse without training in ophthalmology will acquire retinal images in a subsample of patients with diabetes (n = 200) using EyeFundusScope. Moreover, the patient may move the head or the eye during acquisition, potentially affecting the classification of diabetic retinopathy. We think it is important to assess if physicians and nurses without training in ophthalmology can acquire images in a reliable way. Variability between different image acquisitions by the same user will be measured, with the same clinician participant photographing the retina of the same patient at least three times in another subsample.

2.5. Image Quality and Technical Performance

Images acquired with EyeFundusScope will be classified by a panel of ophthalmologists regarding their quality: insufficient, usable, good, and excellent (Table 2 and Appendix B, Figure A1). This classification will be compared with the one provided automatically by a software for retinal image quality assessment [57]. This evaluation is to ensure that the images contain relevant information for observation by ophthalmologists. The use of EyeFundusScope in real clinical settings can throw up unexpected challenges; all potential difficulties will be reported.

2.6. Usability

As retinal imaging can be technically challenging, the ease of use of EyeFundusScope by physicians and nurses without training in ophthalmology for retinal image acquisition on patients with diabetes will be evaluated in this study. The number of attempts and time to make an acceptable image will be recorded. Additionally, after all retinal image acquisitions have been performed, users will fill out the “System Usability Scale”, an instrument with 10 questions answered with 5-point Likert scale (1 = strongly disagree, 5 = strongly agree) [58]—we will use the version previously translated, culturally adapted, and validated to the Portuguese population [59]; qualitative observations (at least 50) of users operating EyeFundusScope will complement the usability study.

2.7. Technology Acceptance

A questionnaire will be conducted with physicians and nurses after their participation in image acquisition to assess their perceptions about the use of EyeFundusScope as a medical device for diabetic retinopathy screening: which errors happened, its integration in clinical workflow, and potential barriers to its implementation (Appendix C). Because a suitable test is such that is accepted by the population—Wilson and Jungner’s 5th and 6th principles of screening [52]—we will assess EyeFundusScope’s acceptance by conducting interviews, which will include the questions presented in Appendix D with 50 patients, observations of retinal image acquisitions, and taking written notes of relevant comments of patients participants during image acquisition about eventual concerns regarding this new technology (n = 20). Qualitative findings will be analysed using thematic analysis [60].

2.8. Data Collection Methods and Management

2.8.1. Data Collection Methods

Retinal images collected with EyeFundusScope will be coded and stored using a dedicated mobile application that will automatically upload the images to a server; images will then be displayed on a web platform for image classification by ophthalmologists. The time required for the clinician to acquire all images of each patient and to insert the images in the information system will be recorded. Then, each ophthalmologist will record an image score for each retinal image during its interpretation and classification for diabetic retinopathy and image quality. The time required for the ophthalmologist to interpret and classify each retinal image will be recorded automatically in the web platform and will be considered from the time the ophthalmologist starts reading the image until the time the ophthalmologist finishes documenting the classification.
Demographics and clinical and laboratory data will be collected from patients and electronic health records using a paper data collection form. Researchers will conduct in-loco observations of healthcare professionals using EyeFundusScope. All data will finally be entered into a Microsoft Excel® datasheet.

2.8.2. Data Management

All the data collected will be encoded. Information about each participant, introduced in the mobile app, as well as images of the retina acquired with EyeFundusScope, will be sent through encrypted communication to protected servers, where it will be stored for later analysis by ophthalmologists. Data will be hosted on Fraunhofer AICOS (Porto, Portugal) servers, or the CUF Tejo hospital. As soon as the information is entered, including images of the retina, with associated ID, introduced in the mobile app, it will be accessible to researchers. Ophthalmologists will classify retinal images using a web platform that will anonymously display fundus images and some clinical information of the patient. This platform is only accessible through a login, which each ophthalmologist will receive and will require encrypted communication. This process complies with all the requirements of the General Data Protection Regulation and was subject to evaluation by the Data Protection Officer (DPO) from CUF and the CUF Department of Information Systems (DPI). Paper data entry forms and paper consent forms will be enclosed in opaque envelopes and stored in restricted access areas inside the Sponsor’s facilities. Researchers are committed to maintaining secrecy and confidentiality in all their activities. Analysis of the study data, as well as its presentation in reports, scientific publications, or other forms of publication and dissemination, will preferably be done in an aggregated form and will not allow the identification of study participants.

2.9. Monitoring

2.9.1. Data Monitoring

A data monitoring committee is not needed in this study, because the study will be performed in an adult population with a non-critical or life-threatening disease, and the medical device under investigation is well characterized and known for not harming patients.

2.9.2. Harms

The clinical investigators will collect information about adverse events of study procedures—device failures and device-related adverse events—during the study and will assess them for intensity, causality, and expectancy; then, adverse events will be reported to the Sponsor (Fraunhofer AICOS, Porto, Portugal) according to the study protocol. Additionally, the Study Monitor—as an independent representative of the Sponsor—will monitor the study to ensure that it is carried out in accordance with the protocol, the applicable regulations, and the Clinical Investigation of Medical Devices for Human Subjects—Good Clinical Practice (ISO 14155:2020) [61], performing visits to research centres.

2.9.3. Auditing

Independent regulatory authorities, namely the National Clinical Research Ethics Committee—CEIC (Lisbon, Portugal) and the National Authority of Medicines and Health Products (INFARMED I.P., Lisbon, Portugal), will eventually perform external audits. The Investigators and the Sponsor (Fraunhofer AICOS, Porto, Portugal) will collaborate on such audits, facilitating the Auditors’ access to documents, facilities, records, and other elements related to the study. Internal audits will not be performed due to close monitoring of the Study Monitor, and the low risk of damage from any deviations from the planned and/or established study protocol.

2.10. Ethics and Dissemination

The study received institutional and ethical approval from ARS Norte and Hospital Infante Santo (now Hospital CUF Tejo, Lisbon, Portugal); approval from CEIC is also expected. Important eventual protocol modifications will be communicated to research ethics committees and other relevant parties. Clinical researchers will obtain written informed consent from participants before the study. The study will be carried out in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). The privacy rights of human subjects will be observed in all phases of the study. Information about participants will be stored using data encryption techniques. Identified data will only be available for researchers involved in data collection and will be deidentified prior to the results analysis and dissemination. CADx’s results will not be displayed on the smartphone screen, so that the nurse or the physician operator will not be influenced in their clinical decision. The participants will receive adequate care for adverse events and insurance coverage. The authors will publish the study results in scientific journals, conferences, and presentations to stakeholders, and on social media. The study protocol will be publicly available on the ISRCTN Registry.

2.11. Statistical Analysis

Demographic, clinical, and laboratory characteristics of the study population will be reported using descriptive statistics using absolute frequencies (n) and relative frequencies (%) for categorical variables. Median and percentiles or mean and standard deviation will be used for continuous variables when appropriate.
Results of the screening for diabetic retinopathy with EyeFundusScope (index test) will be compared to those of the tabletop fundus camera (reference standard) by calculating sensitivity, specificity, predictive values, and likelihood ratios. A cross-tabulation of results of index test by the results of the reference test will be reported, as well as non-classifiable images due to insufficient image quality. The reliability and interobserver agreement between ophthalmologists (reference standard) will be calculated to address some expected variability in the ophthalmologists’ classification using kappa statistics and overall and specific observed agreement.
The association between the classification for diabetic retinopathy and image quality using retinal images acquired with EyeFundusScope will be presented using association measures, namely odds ratios with 95% confidence intervals (OR [95% CI]). Exploratory association between image quality classified by ophthalmologists and the software for automatic image quality classification will be calculated in each image and fixation point obtained using descriptive statistics and association measures. Association between diabetic retinopathy classification by ophthalmologists with image quality automatic classification will be conducted using descriptive statistics and association measures.
Overall measure of retinal image quality based on standard quality assessment protocol of images acquired with EyeFundusScope, and image quality of EyeFundusScope as a function of time and number of images acquired by each operator, using descriptive statistics and association measures will be reported. Inter and intraoperator agreement for diabetic retinopathy classification results will be assessed using descriptive statistics, overall and specific agreement proportion for each category; reliability will be assessed using Cohen’ s kappa. Agreement related to image quality between the software for image quality classification and the ratings from ophthalmologists will be assessed using descriptive statistics, overall observed agreement, and Cohen’ s kappa.
We will calculate and present CIs for all statistical measures.
Proportion of individuals who are willing to undergo further annual screening with the new test will be assessed and qualitative analyses of interviews and observations will be presented. The proportion of healthcare professionals who report ease of use, the score of the “System Usability Scale” [58], qualitative analyses of the observation, the mean time required to perform retinal image acquisition with EyeFundusScope of each eye and of each patient, number of eyes in which it was not possible to acquire images, and number of technical failures of both EyeFundusScope and the information system.
Data will be analysed using IBM ® SPSS ® Statistics and/or R Studio ® and the significance level will be 5%.

3. Discussion

The current study will contribute to inform the potential for implementation of diabetic retinopathy screening with EyeFundusScope and EyeFundusScope + CADx system in clinical practice. If the results of this clinical validation are found to be satisfactory, it will be possible to obtain the CE Mark as a medical device, which opens the door for its use as a mobile eye fundus camera and an automated diagnostic system based on deep learning. This new screening technology will support physicians acquiring retinal images opportunistically and that does not require the intervention of an ophthalmologist to identify referrable diabetic retinopathy. Therefore, it can increase screening coverage, and save ophthalmologists’ valuable time to in person observation and treatment of patients screened positive.
Good image quality is essential for good accuracy of screening results and, ultimately, the clinical value of the tools that we will assess in this study. A few studies have assessed the quality of images of handheld fundus cameras, with which we could compare our results. Darma and colleagues [62] reported image quality obtained with fundus images acquired without pupillary dilation with a smartphone-based fundus camera (Panoptic 11,820 (Welch Allyn, Skaneateles Falls, NY, USA) + iPhone 4 (Apple, Cupertino, CA, USA) and a dedicated handheld fundus camera (Smartscope M5 + EY3 lens; Optomed, Oulu, Finland): 2% and 50%, respectively, of images of quality of 4 (using a 5-point scale ranging from 1 = “can’t see anything” to 5 = “subtle details visible”). Those images were taken from retinas of healthy people with a mean age of 26 years old, in which fundus photograph is easier than in older people and patients with other eye diseases, such as cataracts. However, the study was conducted in 2014, and since then, smartphone technology developed significantly; therefore, we expect our results for image quality to be higher. A recent study [63] described that the fundus images acquired with a handheld fundus camera (Horus Eye-Fundus Camera; Medimaging Integrated Solution Inc., Hsinchu, Taiwan) without pupillary dilation had good (55.7%) or excellent (22.7%) image quality. The average age of the participants was 62 years old, and they had diverse retinal pathology. A direct comparison of results of other studies will not be possible because different scales for image quality are used.
Piyasena and colleagues showed that in the 80+ age group, the technical failure rates reduced from 41.6% to 16.9% following mydriasis [48]. This study concluded that the odds of having one eye ungradable increases by 2.6% (95% CI 1.6–3.7%) for each extra year since the diagnosis of diabetes, with central cataract (57%) being the major cause of ungradability [48].
Different smartphone-based retinal cameras have shown high diagnostic accuracy, with pooled sensitivity and specificity of 87% (95% CI 74–94%) and 94% (95% CI 81–98%), respectively, for any diabetic retinopathy, and 91% (95% CI 86–94%) and 89% (95% CI 56–98%) for referrable diabetic retinopathy [27]. When AI was used to classify images acquired with smartphone-based cameras, pooled sensitivity and specificity were 91% (95% CI 84–96%) and 50% (95% CI 38–62%) [27]. A direct comparison will be difficult, as in some studies images were acquired by ophthalmologists or retina specialists and/or used different grading criteria of diabetic retinopathy. However, our study will focus on enabling the validation of EyeFundusScope and for EyeFundusScope + CADx system as medical devices for diabetic retinopathy screening. We will interpret the results of our study according to the levels that would comply with the accepted standards of established national-level screening programmes: sensitivity ≥80% and specificity ≥95%) [64].

4. Conclusions

Evidence from this study of clinical validation will be useful for paving the way of obtaining a CE mark for EyeFundusScope and for EyeFundusScope + CADx system for diabetic retinopathy. Furthermore, the results of this study will inform the decision makers about the implementation of opportunistic diabetic retinopathy screenings, as well as of extended screening programmes to areas underserved by clinic infrastructure and to people with limitations in mobility. A medical device that is safe, accurate, and easy to use by physicians and nurses not specialists in ophthalmology has the potential to bring confidence to medical teams to adopt the device in their healthcare institutions.
Moreover, with this study, we anticipate challenges to local implementation and integration of these new point-of-care diabetic retinopathy screening tools, both EyeFundusScope and EyeFundusScope + CADx system. This translation from the laboratory to real clinical settings and use conditions within the target population will facilitate its future implementation into clinical practice. At the end of the study, we will have a user-ready and clinically validated medical device. Therefore, the results of this study will be relevant for patients, health professionals and decision makers.

Author Contributions

Conceptualization, S.R., M.M.-S., F.S., and F.N.; methodology, S.R., M.M.-S., M.D.-M., F.S., C.C.D., and F.N.; writing—original draft preparation, S.R.; writing—review and editing, M.M.-S., M.D.-M., F.S., C.C.D., and F.N.; project administration, F.S. and F.N.; funding acquisition, F.N.; resources, S.R., M.M.-S., M.D.-M., F.S., C.C.D., and F.N.; validation, M.M.-S., M.D.-M., F.S., C.C.D., and F.N.; supervision, M.M.-S., M.D.-M., F.S., C.C.D., and F.N. All authors have read and agreed to the published version of the manuscript.


This work was supported by EyeFundusScopeNEO: Demonstration of EyeFundusScope with Non-Expert Ophthalmology Users, cofounded by Portugal 2020, framed under the Operational Program Competitiveness and Internationalization (COMPETE 2020) and European Regional Development Fund from European Union, with operation code POCI-01-0247-FEDER-038400.

Institutional Review Board Statement

The study will be conducted according to the guidelines of the Declaration of Helsinki and was approved by the Ethics Committee of ARS NORTE on 16/08/2018 (Decision no. 115/2018) and CUF on 21 December 2019.

Informed Consent Statement

Informed consent will be obtained from all subjects involved in the study.

Data Availability Statement

Data will not be publicly available.


The authors thank Telmo Barbosa from Fraunhofer AICOS (Porto, Portugal) for the development and management of the web annotation tool for classification of retinal images by ophthalmologists.

Conflicts of Interest

The authors declare the following conflicts of interest: S.R., F.N. and F.S. are employees of Fraunhofer AICOS (Porto, Portugal), and this institution is developing EyeFundusScope, a smartphone-based fundus camera for diabetic retinopathy screening, and a computer-aided diagnosis (CADx) system for diabetic retinopathy screening based on deep learning. M.M.-S., M.D.-M. and C.C.D. are supervisors of the doctoral student S.R., but they have no financial involvement with Fraunhofer AICOS. No other relationships or activities could appear to have influenced the submitted work. The funders had no role in the design of the study or in the writing of the manuscript.

Appendix A

Sociodemographic characteristics and clinical and laboratory data.
Table A1. Sociodemographic characteristics and clinical and laboratory data.
Table A1. Sociodemographic characteristics and clinical and laboratory data.
ClinicalType of diabetes
Duration of diabetes
Other eye diseases: cataract, glaucoma, age-related macular degeneration, myopia (number of diopters), other
Current diabetic retinopathy classification (reference standard)

Appendix B

Example images according to the classification for quality.
Figure A1. Classification of image quality: (a) excellent, (b) good, (c) usable, (d) inadequate.
Figure A1. Classification of image quality: (a) excellent, (b) good, (c) usable, (d) inadequate.
Diabetology 03 00001 g0a1

Appendix C

Usability interview for physicians and nurses.
  • Age________
  • Sex: Feminine____ Masculine____
  • Profession: _______________________________
  • Number of years of professional experience ______________
  • Do you have a smartphone or have experience using smartphones? Yes____ No____
  • Could you give a good and a bad example of using the system?
  • Could you tell us what errors happened while you were using the system? Please describe the context and the problems that occurred.
  • Do you think using the system is simple? Yes____ No____
  • If you think not, which aspects in your opinion could be improved?
  • Do you think that this system could be used as part of a regular consultation?
    Yes____ No____
  • Would you recommend to your colleagues the use of this system in their queries?
    Yes____ No____
  • In your opinion, which factors are favourable to the use of this system in the implementation of screening for diabetic retinopathy in the context of a regular consultation?
  • What factors can make it difficult? ________________________________________________________________
  • Did you notice some discomfort on users regarding the use of the equipment under study?
    Yes____ No____
    If yes, what is the estimated percentage of cases in which this occurred?
    0%____ 1 a 25%____ 26 a 50%____ 51 a 75%____ 76 a 99%____ 100%____
    If yes, what did they verbalize?
  • Did you notice some distrust on users regarding the use of the equipment under study?
    Yes____ No____
    If so, what is the estimate of cases in which this occurred?
    0%____ 1 a 25%____ 26 a 50%____ 51 a 75%____ 76 a 99%____ 100%____
    If yes, what did they verbalize?
  • Overall, is there some aspect of the system that you think could be improved?
    Yes____ No____
    What are these aspects?

Appendix D

Technology acceptance questionnaire to patients.
  • Did you feel some discomfort during the exam with this (new) equipment?
    Yes____ No____
    If yes, which? __________________________________________________________________________
  • In the next screening, would you do this exam with this (new) equipment again?
    Yes____ No____
  • At the next screening, would you prefer this (new) device over the other?
    Yes____ No____
  • At the next screening, if only one of the exams was available, which would you prefer?
  • Conventional equipment____ New equipment____


  1. Kollias, A.N.; Ulbig, M.W. Diabetic retinopathy: Early diagnosis and effective treatment. Dtsch. Arzteblatt Int. 2010, 107, 75–83. [Google Scholar] [CrossRef]
  2. Duh, E.J.; Sun, J.K.; Stitt, A.W. Diabetic retinopathy: Current understanding, mechanisms, and treatment strategies. JCI Insight 2017, 2, e93751. [Google Scholar] [CrossRef]
  3. Annual Report of the National Diabetes Observatory—2016. Available online: (accessed on 1 November 2021).
  4. Chua, J.; Lim, C.X.Y.; Wong, T.Y.; Sabanayagam, C. Diabetic Retinopathy in the Asia-Pacific. Asia-Pac. J. Ophthalmol. 2018, 7, 3–16. [Google Scholar]
  5. Pieczyński, J.; Grzybowski, A. Review of Diabetic Retinopathy Screening Methods and Programmes Adopted in Different Parts of the World. Eur. Ophthalmic Rev. 2015, 9, 49–55. [Google Scholar] [CrossRef][Green Version]
  6. Panwar, N.; Huang, P.; Lee, J.; Keane, P.A.; Chuan, T.S.; Richhariya, A.; Teoh, S.; Lim, T.H.; Agrawal, R. Fundus Photography in the 21st Century—A Review of Recent Technological Advances and Their Implications for Worldwide Healthcare. Telemed. J. E Health 2016, 22, 198–208. [Google Scholar] [CrossRef]
  7. Carl Zeiss Meditec Clarus 700 Instructions for Use. Available online: (accessed on 3 December 2021).
  8. Tufail, A.; Kapetanakis, V.V.; Salas-Vega, S.; Egan, C.; Rudisill, C.; Owen, C.G.; Lee, A.; Louw, V.; Anderson, J.; Liew, G.; et al. An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost-effectiveness. In Health Technol Assess; NIHR Journals Library: Southampton, UK, 2016; Volume 20, pp. 1–72. [Google Scholar]
  9. Tozer, K.; Woodward, M.A.; Newman-Casey, P.A. Telemedicine and Diabetic Retinopathy: Review of Published Screening Programs. J. Endocrinol. Diabetes 2015, 2, 1–10. [Google Scholar]
  10. Massin, P.; Chabouis, A.; Erginay, A.; Viens-Bitker, C.; Lecleire-Collet, A.; Meas, T.; Denormandie, P.; Choupot, G.; André, B. OPHDIAT: A Telemedical Network Screening System for Diabetic Retinopathy in the Ile-de-France. Diabetes Metab. 2008, 34, 227–234. [Google Scholar] [CrossRef] [PubMed]
  11. Indian Health Service—The Federal Health Program for American Indians and Alaska Natives. Teleophthalmology. Available online: (accessed on 2 May 2020).
  12. Diabetic Retinopathy Screening–Standard no. 016/2018. Available online: (accessed on 2 November 2021).
  13. Hazin, R.; Barazi, M.K.; Summerfield, M. Challenges to establishing nationwide diabetic retinopathy screening programs. Curr. Opin. Ophthalmol. 2011, 22, 174–179. [Google Scholar] [CrossRef]
  14. AlHargan, M.H.; AlBaker, K.M.; AlFadhel, A.A.; AlGhamdi, M.A.; AlMuammar, S.M.; AlDawood, H.A. Awareness, knowledge, and practices related to diabetic retinopathy among diabetic patients in primary healthcare centers at Riyadh, Saudi Arabia. J. Fam. Med. Prim. Care 2019, 8, 373–377. [Google Scholar] [CrossRef]
  15. Glasson, N.M.; Larkins, S.L.; Crossland, L.J. What do patients with diabetes and providers think of an innovative Australian model of remote diabetic retinopathy screening? A qualitative study. BMC Health Serv. Res. 2017, 17, 158. [Google Scholar] [CrossRef][Green Version]
  16. Piyasena, M.M.P.N.; Murthy, G.V.S.; Yip, J.L.Y.; Gilbert, C.; Zuurmond, M.; Peto, T.; Gordon, I.; Hewage, S.; Kamalakannan, S. Systematic review on barriers and enablers for access to diabetic retinopathy screening services in different income settings. PLoS ONE 2019, 14, e0198979. [Google Scholar] [CrossRef]
  17. Kashim, R.M.; Newton, P.; Ojo, O. Diabetic Retinopathy Screening: A Systematic Review on Patients’ Non-Attendance. Int. J. Environ. Res. Public Health 2018, 15, 157. [Google Scholar] [CrossRef] [PubMed][Green Version]
  18. Zimmer-Galler, I.E.; Kimura, A.E.; Gupta, S. Diabetic retinopathy screening and the use of telemedicine. Curr. Opin. Ophthalmol. 2015, 26, 167–172. [Google Scholar] [CrossRef] [PubMed]
  19. World Health Organization. Prevention of Blindness from Diabetes Mellitus. Available online: (accessed on 1 May 2020).
  20. Guariguata, L.; Whiting, D.R.; Hambleton, I.; Beagley, J.; Linnenkamp, U.; Shaw, J.E. Global estimates of diabetes prevalence for 2013 and projections for 2035. Diabetes Res. Clin. Pract. 2014, 103, 137–149. [Google Scholar] [CrossRef] [PubMed]
  21. Scanlon, P.H. Screening Intervals for Diabetic Retinopathy and Implications for Care. Curr. Diab. Rep. 2017, 17, 96. [Google Scholar] [CrossRef] [PubMed]
  22. DCCT/EDIC Research Group; Nathan, D.M.; Bebu, I.; Hainsworth, D.; Klein, R.; Tamborlane, W.; Lorenzi, G.; Gubitosi-Klug, R.; Lachin, J.M. Frequency of Evidence-Based Screening for Retinopathy in Type 1 Diabetes. N. Engl. J. Med. 2017, 376, 1507–1516. [Google Scholar]
  23. Ichhpujani, P.; Thakur, S. Apps and Social Networking Pages for Vitreoretinal and Uveal Diseases. In Smart Resources in Ophthalmology: Applications and Social Networking; Springer: Singapore, 2018; pp. 97–140. [Google Scholar]
  24. Hong, K.; Collon, S.; Chang, D.; Thakalli, S.; Welling, J.; Oliva, M.; Peralta, E.; Gurung, R.; Ruit, S.; Tabin, G.; et al. Teleophthalmology through handheld mobile devices: A pilot study in rural Nepal. J. Mob. Technol. Med. 2019, 8. [Google Scholar] [CrossRef] [PubMed]
  25. Giardini, M.E.; Livingstone, I.A.T.; Jordan, S.; Bolster, N.M.; Peto, T.; Burton, M.; Bastawrous, A. A smartphone based ophthalmoscope. In Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA, 26–30 August 2014; IEEE: Piscataway, NJ, USA, 2014; pp. 2177–2180. [Google Scholar]
  26. Jin, K.; Lu, H.; Su, Z.; Cheng, C.; Ye, J.; Qian, D. Telemedicine screening of retinal diseases with a handheld portable non-mydriatic fundus camera. BMC Ophthalmol. 2017, 17, 89. [Google Scholar] [CrossRef]
  27. Tan, C.H.; Kyaw, B.M.; Smith, H.; Tan, C.S.; Tudor Car, L. Use of Smartphones to Detect Diabetic Retinopathy: Scoping Review and Meta-Analysis of Diagnostic Test Accuracy Studies. J. Med. Internet Res. 2020, 22, e16658. [Google Scholar] [CrossRef]
  28. Micheletti, J.M.; Hendrick, A.M.; Khan, F.N.; Ziemer, D.C.; Pasquel, F.J. Current and Next Generation Portable Screening Devices for Diabetic Retinopathy. J. Diabetes Sci. Technol. 2016, 10, 295–300. [Google Scholar] [CrossRef] [PubMed]
  29. Ocular CellScope. Available online: (accessed on 2 November 2021).
  30. Fundus on Phone. Available online: (accessed on 2 November 2021).
  31. D-Eye Portable Ophthalmoscope. Available online: (accessed on 2 November 2021).
  32. Volk VistaView. Available online: (accessed on 2 November 2021).
  33. Ophtha: Decentralised Screening in Ophthalmology. Available online: (accessed on 2 November 2021).
  34. Pictor Prestige Portable Fundus Camera. Available online: (accessed on 2 November 2021).
  35. van der Heijden, A.A.; Abramoff, M.D.; Verbraak, F.; an Hecke, M.V.; Liem, A.; Nijpels, G. Validation of automated screening for referable diabetic reti-nopathy with the IDx-DR device in the Hoorn Diabetes Care System. Acta Ophthalmol. 2018, 96, 63–68. [Google Scholar] [CrossRef] [PubMed]
  36. Medios DR Remidio. Available online: (accessed on 2 November 2021).
  37. Natarajan, S.; Jain, A.; Krishnan, R.; Rogye, A.; Sivaprasad, S. Diagnostic Accuracy of Community-Based Diabetic Retinopathy Screening With an Offline Artificial Intelligence System on a Smartphone. JAMA Ophthalmol. 2019, 137, 1182–1188. [Google Scholar] [CrossRef] [PubMed]
  38. The Dilated Eye Exam: Why It’s So Important. Available online: (accessed on 2 December 2021).
  39. Karakaya, M.; Hacisoftaoglu, R.E. Comparison of smartphone-based retinal imaging systems for diabetic retinopathy detection using deep learning. BMC Bioinform. 2020, 21, 259. [Google Scholar] [CrossRef]
  40. Goh, J.K.H.; Cheung, C.Y.; Sim, S.S.; Tan, P.C.; Tan, G.S.W.; Wong, T.Y. Retinal Imaging Techniques for Diabetic Retinopathy Screening. J. Diabetes Sci. Technol. 2016, 10, 282–294. [Google Scholar] [CrossRef][Green Version]
  41. Minims Tropicamide 0.5% w/v, Eye Drops Solution-Summary of Product Characteristics. Available online: (accessed on 8 December 2021).
  42. Dervan, E.; Lillis, D.; Flynn, L.; Staines, A.; O’Shea, D. Factors that influence the patient uptake of diabetic retinopathy screening. Ir. J. Med. Sci. 2008, 177, 303–308. [Google Scholar] [CrossRef]
  43. Nunes, F.; Madureira, P.; Rêgo, S.; Braga, C.; Moutinho, R.; Oliveira, T.; Soares, F. A Mobile Tele-Ophthalmology System for Planned and Opportunistic Screening of Diabetic Retinopathy in Primary Care. IEEE Access 2021, 9, 83740–83750. [Google Scholar] [CrossRef]
  44. Felgueiras, S.; Costa, J.; Gonçalves, J.; Soares, F. Mobile-based Risk Assessment of Diabetic Retinopathy using a Smartphone and Adapted Ophthalmoscope. In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies; SCITEPRESS-Science and Technology Publications, Funchal, Madeira, Portugal, 19–21 January 2018; pp. 168–175. [Google Scholar]
  45. Rêgo, S.; Dutra-Medeiros, M.; Soares, F.; Monteiro-Soares, M. Screening for Diabetic Retinopathy Using an Automated Diagnostic System Based on Deep Learning: Diagnostic Accuracy Assessment. Ophthalmologica 2021, 244, 250–257. [Google Scholar] [CrossRef]
  46. Design Considerations for Pivotal Clinical Investigations for Medical Devices. Available online: (accessed on 2 November 2021).
  47. Braga, C.; Rêgo, S.; Nunes, F. Clinicians’ Perspectives on Using Mobile Eye Fundus Cameras to Screen Diabetic Retinopathy in Primary Care. In Proceedings of the 2020 IEEE International Conference on Healthcare Informatics (ICHI), Oldenburg, Germany, 30 November–3 December 2020; pp. 1–7. [Google Scholar]
  48. Piyasena, M.M.P.N.; Gudlavalleti, V.S.M.; Gilbert, C.; Yip, J.L.; Peto, T.; MacLeod, D.; Fonseka, C.; Kulatunga, A.; Bandutilake, B.; Dhanapala, M.; et al. Development and Validation of a Diabetic Retinopathy Screening Modality Using a Hand-Held Nonmydriatic Digital Retinal Camera by Physician Graders at a Tertiary-Level Medical Clinic: Protocol for a Validation Study. JMIR Res. Protoc. 2018, 7, e10900. [Google Scholar] [CrossRef]
  49. Shen, Z.; Fu, H.; Shen, J.; Shao, L. Modeling and Enhancing Low-Quality Retinal Fundus Images. IEEE Trans. Med. Imaging 2021, 40, 996–1006. [Google Scholar] [CrossRef]
  50. Cohen, J.F.; Korevaar, D.A.; Altman, D.G.; Bruns, D.E.; Gatsonis, C.A.; Hooft, L.; Irwig, L.; Levine, D.; Reitsma, J.B.; de Vet, H.C.W.; et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: Explanation and elaboration. BMJ Open 2016, 6, e012799. [Google Scholar] [CrossRef] [PubMed]
  51. Retinopatia Diabética-Patologias. Available online: (accessed on 2 November 2021).
  52. Diabetic Retinopathy Screening: A Short Guide. Increase Effectiveness, Maximize Benefits and Minimize Harm. Available online: (accessed on 2 November 2021).
  53. Ruamviboonsuk, P.; Teerasuwanajak, K.; Tiensuwan, M.; Yuttitham, K. Thai Screening for Diabetic Retinopathy Study Group Interobserver agreement in the interpretation of single-field digital fundus images for diabetic retinopathy screening. Ophthalmology 2006, 113, 826–832. [Google Scholar] [CrossRef] [PubMed]
  54. Gegundez-Arias, M.E.; Ortega, C.; Garrido, J.; Ponte, B.; Alvarez, F.; Marin, D. Inter-observer Reliability and Agreement Study on Early Diagnosis of Diabetic Retinopathy and Diabetic Macular Edema Risk. In Proceedings of the International Conference on Bioinformatics and Biomedical Engineering, Granada, Spain, 20–22 April 2016; pp. 369–379. [Google Scholar]
  55. Rêgo, S.; Dutra-Medeiros, M.; Bacelar-Silva, G.M.; Borges, T.; Soares, F.; Monteiro-Soares, M. Reliability of Classification by Ophthalmologists with Telescreening Fundus Images for Diabetic Retinopathy and Image Quality. J. Diabetes Sci. Technol. 2021, 15, 710–712. [Google Scholar] [CrossRef] [PubMed]
  56. Northern Region Diabetic Retinopathy Screening Procedures Manual. Available online: (accessed on 2 November 2021).
  57. Simões, B.R.B.A. Image Quality Improvement of Medical Images Using Deep Learning for Computer-Aided Diagnosis. Master’s Thesis, NOVA University Lisbon, Lisbon, Portugal, 2021. [Google Scholar]
  58. Brooke, J. SUS: A quick and dirty usability scale. In Usability Evaluation in Industry, 1st ed.; Jordan, P.W., Thomas, B., McClelland, I.L., Weerdmeester, B., Eds.; CRC Press: London, UK, 1996. [Google Scholar]
  59. Martins, A.I.; Rosa, A.F.; Queirós, A.; Silva, A.; Rocha, N.P. European Portuguese Validation of the System Usability Scale (SUS). Procedia Comput. Sci. 2015, 67, 293–300. [Google Scholar] [CrossRef][Green Version]
  60. Braun, V.; Clarke, V. Thematic analysis. In APA Handbook of Research Methods in Psychology; Volume 2: Research Designs: Quantitative, Qualitative, Neuropsychological, and Biologica; American Psychological Association: Washington, DC, USA, 2012; pp. 57–71. [Google Scholar]
  61. ISO 14155: 2020 Clinical Investigation of Medical Devices for Human Subjects—Good Clinical Practice. Available online: (accessed on 2 November 2021).
  62. Darma, S.; Zantvoord, F.; Verbraak, F.D. The quality and usability of smartphone and hand-held fundus photography, compared to standard fundus photography. Acta Ophthalmol. 2015, 93, e310–e311. [Google Scholar] [CrossRef]
  63. Lin, T.-C.; Chiang, Y.-H.; Hsu, C.-L.; Liao, L.-S.; Chen, Y.-Y.; Chen, S.-J. Image quality and diagnostic accuracy of a handheld nonmydriatic fundus camera: Feasibility of a telemedical approach in screening retinal diseases. J. Chin. Med. Assoc. 2020, 83, 962–966. [Google Scholar] [CrossRef] [PubMed]
  64. Scanlon, P.H. The English National Screening Programme for Diabetic Retinopathy 2003–2016. Acta Diabetol. 2017, 54, 515–525. [Google Scholar] [CrossRef] [PubMed][Green Version]
Figure 1. Two scenarios of use of EyeFundusScope: (a) Images acquired with EyeFundusScope are sent to a reading centre to be classified for diabetic retinopathy by an ophthalmologist. (b) Images acquired with EyeFundusScope are classified for diabetic retinopathy by the CADx system.
Figure 1. Two scenarios of use of EyeFundusScope: (a) Images acquired with EyeFundusScope are sent to a reading centre to be classified for diabetic retinopathy by an ophthalmologist. (b) Images acquired with EyeFundusScope are classified for diabetic retinopathy by the CADx system.
Diabetology 03 00001 g001
Table 1. Severity levels of diabetic retinopathy adapted from [56].
Table 1. Severity levels of diabetic retinopathy adapted from [56].
Severity LevelsCharacteristics
R0No retinopathy: any lesion related to DR 1
R1Mild non-proliferative retinopathy: microaneurysms, retinal haemorrhages with or without any exudate excluding the definition of DME 2
R2Moderate or severe pre-proliferative: “rosary” veins, venous loops and duplications, intra-retinal microvascular anomalies, multiple, deep, cottony exudates
R3Proliferative: disc neo-vessels, retinal neo-vessels, vitreous or pre-retinal haemorrhage, pre-retinal fibrosis with or without traditional retinal detachment
M1Maculopathy: Presence of exudates less than 1 DD 3 from the centre of the fovea, or circinate or grouped exudates in the macular area, or retinal thickening to less than 1 DD 3 from the centre of the fovea, or any microaneurysm or haemorrhage less than 1 DD 3 from the centre of the fovea, if associated with an AV 4 < 0.5
P0Photocoagulated retina not needing more treatment
P1Photocoagulated retina needing more treatment
Not classifiableNon-informative image
1 Diabetic retinopathy; 2 diabetic macular oedema; 3 disc diameter; 4 venous anomaly.
Table 2. Criteria for classification of image quality by ophthalmologists.
Table 2. Criteria for classification of image quality by ophthalmologists.
Image Quality LevelCharacteristics
  • Small vessels are clearly visible and sharp
  • Absolute edge-to-edge sharpness in the optical circle
  • The optic disc and the macula are visible, and the temporal arches are complete
  • Small vessels are clearly visible, but not sharp
  • Without low quality factors, besides the lack of absolute sharpness from end to end in the optical circle
  • The optic disc and the macula are visible, and the temporal arches are complete
  • The field of view is partially hidden
  • Irregular lighting with the legible region of the background image greater than 80% of the area seen
  • Optic disc or macula are visible, and some of the vessels or retinal lesions (identified by ophthalmologists) are also visible, but not always in good contrast or with some blur
  • Large or nearly complete light/dark regions, making it impossible to observe any retinal regions
  • Optic disc or macula are not visible
  • Blur or totally out of focus
  • Large artefacts (occupying more than 25% of the retinal area)
  • Totally undiagnosed, or the image has serious quality problems and cannot be used to provide a reliable diagnosis by ophthalmologists
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Rêgo, S.; Monteiro-Soares, M.; Dutra-Medeiros, M.; Soares, F.; Dias, C.C.; Nunes, F. Implementation and Evaluation of a Mobile Retinal Image Acquisition System for Screening Diabetic Retinopathy: Study Protocol. Diabetology 2022, 3, 1-16.

AMA Style

Rêgo S, Monteiro-Soares M, Dutra-Medeiros M, Soares F, Dias CC, Nunes F. Implementation and Evaluation of a Mobile Retinal Image Acquisition System for Screening Diabetic Retinopathy: Study Protocol. Diabetology. 2022; 3(1):1-16.

Chicago/Turabian Style

Rêgo, Sílvia, Matilde Monteiro-Soares, Marco Dutra-Medeiros, Filipe Soares, Cláudia Camila Dias, and Francisco Nunes. 2022. "Implementation and Evaluation of a Mobile Retinal Image Acquisition System for Screening Diabetic Retinopathy: Study Protocol" Diabetology 3, no. 1: 1-16.

Article Metrics

Back to TopTop