Remote Moderated Usability Testing of a Mobile Phone App for Remote Monitoring of Pregnant Women at High Risk of Preeclampsia in Karachi, Pakistan

Abstract

This study assessed the usability of the smartphone app, named “Raabta” from the perspective of pregnant women at high risk of preeclampsia to improve the Raabta app for future implementation. Think-aloud and task-completion techniques were used with a purposive sample of 14 pregnant women at high risk of preeclampsia. The sessions were audio-recorded and later professionally transcribed for thematic analysis. The study generated learnings associated with four themes: improving the clarity of instructions, messaging, and terminology; accessibility for non-tech savvy and illiterate Urdu users; enhancing visuals and icons for user engagement; and simplifying navigation and functionality. Overall, user feedback emphasized the importance of enhancing the clarity of instructions, messaging, and terminology within the Raabta app. Voice messages and visuals were valued by users, particularly among the non-tech savvy and illiterate Urdu users, as they enhance accessibility and enable independent monitoring. Suggestions were made to enhance user engagement through visual improvements such as enhanced graphics and culturally aligned color schemes. Lastly, users highlighted the need for improved navigation both between screens and within screens to enhance the overall user experience. The Raabta app prototype will be modified based on the feedback of the users to address the unique needs of diverse groups.

1. Introduction

Pakistan’s Maternal Mortality Ratio is 186 deaths per 100,000 live births [1], and one-third of maternal deaths in tertiary hospitals are linked to preeclampsia or eclampsia [2]. Preeclampsia is a hypertensive disorder of pregnancy that is characterized by high blood pressure and protein in the urine after 20 weeks of gestation. It is associated with high maternal mortality, preterm birth, pregnancy loss, and stillbirth [3]. High maternal and neonatal deaths from preeclampsia result from the lack of early risk detection, diagnosis, and treatment of pregnant women at high risk of preeclampsia [3]. A potential tool to support pregnant women at high risk of preeclampsia is telemonitoring, which refers to the use of information technology to monitor patients at a distance [4]. Women at high risk for preeclampsia would take home blood pressure readings and record self-reported symptoms related to preeclampsia/eclampsia, and the data would be sent to clinicians in real-time. Telemonitoring is a promising tool for improving pregnant women’s health outcomes from preeclampsia, as well as quality and coordination of care in a cost-effective and scalable manner [4,5,6,7,8]. Through telemonitoring, clinical alerts can be generated based on the patient’s blood pressure values and symptoms as appropriate, and patients can be sent tailored self-care messages. This could lead to the earliest possible detection of alarming increases in blood pressure, resulting in an early intervention such as through medications, hospitalization or delivery of the baby. In addition, telemonitoring may provide pregnant women at high risk of preeclampsia with the reassurance of clinical oversight and self-care support that may reduce anxiety and stress related to pregnancy and hospitalization [7,9,10,11,12].

Given the paucity of evidence regarding telemonitoring needs and interventions for supporting pregnant women at high risk of preeclampsia in low- to middle-income countries (LMICs) [9], our research team sought to develop and assess the feasibility of a smartphone–based telemonitoring program to support pregnant women at high risk of preeclampsia in Karachi, Pakistan [12]. Karachi, the largest city in Pakistan with a population of over 14 million, has low-income areas and a generally low socioeconomic status among its residents. This leads to poor access to healthcare and education resources, a low literacy rate, and a poor quality of life for its residents. A smartphone app, named “Raabta” which means ‘a connection’ in Urdu, has been developed at the Centre for Digital Therapeutics, University Health Network in Toronto, Canada. Our vision for the Raabta app was to create a remote monitoring program to enable pregnant women at high risk of preeclampsia to monitor their blood pressure and symptoms at home for real-time monitoring by providers. In developing the Raabta app, we leveraged the insights from the telemonitoring needs assessment study previously conducted by our team with pregnant women, their caregivers, and providers to design a user-centered telemonitoring program [10,11]. Several features were included in the app that considered the local context, including the use of the local language (Urdu), illustrations when appropriate to support women with low literacy, an easy-to-understand green–yellow–red alert system that uses the well-known traffic lights analogy, and voice message alerts in conjunction with the text.

Prior to implementing the Raabta app as part of the larger feasibility study, we aimed to conduct usability testing of the Raabta app to better inform the design, content, and functionality of the Raabta app. Researchers have emphasized the significance of conducting usability testing for new healthcare technologies to ensure the new technology meets the needs of the intended users and to enable the optimum impact on healthcare services and outcomes [13,14,15,16]. The usability testing of the Raabta app was carried out with the aim of soliciting end-user feedback to determine ease of use and where gaps in design, content, or functionality exist. In particular, usability testing involved the assessment of the user interface, user experience, satisfaction with content, and functionality of the app, given that these factors influence the frequency of use and understanding of the technology [13,17].

Our objective was to conduct a formative usability testing during the development process of an early prototype of the Raabta app from the perspective of pregnant women at high risk of preeclampsia in Karachi, Pakistan. We aimed to use the remote usability approach to understand the user experience and identify usability issues that would help improve the final version of the Raabta app to be used in a feasibility study.

Traditional usability testing involves in-person sessions, leading to recruitment challenges related to time constraints and social distancing, especially during the COVID-19 pandemic. The pandemic accelerated the adoption of remote usability testing, offering convenience, and flexibility, and overcoming geographic limitations, promoting ongoing research and participant engagement [16]. Since then, remote usability testing has become the default approach to maintain advancements in technology evaluation or research [16]. Remote usability testing has been explored in some studies involving older adults and individuals with limited mobility, particularly during the COVID-19 pandemic, when technology became pivotal for their remote access to services and social interaction [18,19,20]. Nevertheless, there is a scarcity of evidence regarding the utilization of remote usability techniques in countries like Pakistan and other low- to middle-income nations. As technology continues to advance, the field of remote usability testing is expected to gain greater recognition in the literature. The findings of this study will contribute to the body of growing evidence concerning remote moderated usability testing in resource-limited environments.

2. Materials and Methods

2.1. Study Design and Setting

A qualitative remote usability testing approach [21] with semi-structured, audio-taped interviews and observations by a trained observer was undertaken to determine the usability of an early prototype of the Raabta app, with the aim of refining the prototype for future implementation. According to Nielsen Norman’s approach [13,22,23], remote moderated usability testing is similar to an in-person study, with the testing conducted through video-conferencing platform. The key distinction is that the researcher and participant are not physically present in the same location. In our study, the remote moderated usability testing was conducted via Zoom, as a suitable local research analyst with expertise in usability testing was not available for this study. The study was conducted between November and December 2022 in the outpatient area of the Jinnah Post Graduate Medical Center (JPMC) Obstetrics and Gynecology department, whose vast majority of patients are low-income women with high-risk pregnancies. The study was approved by the Aga Khan University (AKU) ethical review committee (2020-2153-8519), the JPMC institutional review board (44379), and the University of Toronto research ethics board (30635).

2.2. Raabta App

The Raabta app was designed to support Pakistani pregnant women at high risk of preeclampsia, many of whom face several sociocultural and financial constraints [10,11]. Hosting of the Raabta app and storage of the study data will be at the Centre for Digital Therapeutics.

The patient-facing technology includes the Raabta smartphone application (app) and Bluetooth-enabled blood pressure device that is validated for use in pregnancy. The app enables pregnant women to take daily blood pressure readings at home with a provided Bluetooth-enabled home blood pressure monitor and to answer one yes/no symptom question. The app provides automated alerts (i.e., self-care messages) to women based on the entered data. The app uses a rules-based algorithm, which was developed using NICE guidelines [24] and through consultations with expert clinicians. The algorithm includes settings to trigger alerts and self-care messages for high-risk pregnant women with or without antihypertensives and symptoms of preeclampsia such as taking additional blood pressure readings, calling the clinician (medical officer and/or midwife/research nurse), visiting the OB-GYN emergency department, etc. Other features of the Raabta app include the use of the local language (Urdu), illustrations for daily symptom monitoring questions (for pregnant women who cannot read), an easy-to-understand green–yellow–red alert system that uses the well-known traffic lights analogy, and voice message alerts in conjunction with the text. All these features and functionalities were incorporated to ensure that the Raabta app is culturally aligned and relevant. The features of the technology have been informed by our previous needs assessment study with pregnant women at HRPE and their caregivers [10,11]. Figure 1 displays various screens from the Raabta application.

The clinician-facing technology includes the web-based Raabta dashboard. To support clinical decision making, the medical officer and research nurse situated at the JPMC OB-GYN outpatient department will receive alerts from the telemonitoring system if the patient’s blood pressure trends are out of the target and/or if the patient is reporting symptoms. The medical officer will act as a central point person to communicate with the patients (via phone calls or using the asynchronous telemonitoring system chat feature) and with the rest of the participant’s care team as needed.

The medical officer and research nurse will train recruited pregnant women at HRPE and their caregivers, if appropriate, on the use of the telemonitoring system at the same antenatal visit at which they have consented to participate in the study. The training session for pregnant women and caregivers will last approximately 1 h. Supplementary A offers a comprehensive guide on utilizing the Raabta program. This guide will be used to train women during enrollment. Additionally, we have developed a short animated video in the Urdu language that will be used to train women on the use of the Raabta program. If the participant owns a compatible smartphone (i.e., an Android phone operating a minimum of Android V.5.1.1.), the app will be downloaded onto their phone. Otherwise, a mobile phone will be provided for the duration of the study. Technical support will be available to all participants by the research nurse and the medical officer via a provided phone number. Once at home, participants will be asked to take their blood pressure every morning and answer symptom questions using the app to receive auto-alerts and self-care messages (Figure 1). For instance, if the participant develops hypertension (systolic between 140 and 154 mm Hg inclusive and diastolic between 90 and 104 mm Hg inclusive) with no preeclampsia symptoms, they will be prompted by the app to take a second blood pressure measurement in 4–6 h. If the participant develops severe hypertension (systolic ≥155 mm Hg and diastolic ≥105 mm Hg) and/or preeclampsia symptoms, they will be prompted by the app to immediately visit the emergency department.

2.3. Participant Recruitment

In planning the usability testing protocol, our research team recognized that the intended users for the Raabta app may represent diverse user groups in terms of Urdu language literacy and technological literacy [25,26]. To ensure representative participants in our usability testing [12], we recruited a sample that was diverse in terms of smartphone use and Urdu language literacy to ensure that the included participants are reflective of the broader population at JPMC. Participants were required to meet the NICE criteria [24] (

Table 1. Definition of Pregnant Women at HRPE as per NICE Guideline.

NICE Guidelines define pregnant women at HRPE as those who have one high-risk factor or more than one moderate-risk factor for PE.

High-risk factors include: Hypertensive disease in a previous pregnancy; Chronic kidney disease; Autoimmune diseases, such as systemic lupus erythematosus or antiphospholipid syndrome; Type 1 or type 2 diabetes; Chronic hypertension.

Moderate risk factors include: First pregnancy; Age 40 years or older; Pregnancy interval of more than 10 years; Body mass index (BMI) of 35 kg/m2 or more at the first visit; Family history of PE; Multi-fetal pregnancy.

) for being at high risk of preeclampsia and were required to speak the Urdu language, the official and primary language in Pakistan, as the Raabta app prototype was developed in Urdu. The participants were purposively sampled into four target user groups [25,26] based on their ability to read Urdu and use smartphones: (1) pregnant women at high risk of preeclampsia who are able to read Urdu and experienced with using smartphones; (2) pregnant women at high risk of preeclampsia with low to no literacy in Urdu and without experience using smartphone; (3) pregnant women at high risk of preeclampsia who are able to read Urdu and without experience using smartphones; and (4) pregnant women at high risk of preeclampsia with low to no literacy in Urdu but are experienced with using smartphones. We planned to recruit at least 5 participants in each target user group for usability testing [13] and ceased recruitment once data saturation was reached. The on-site facilitator coordinated with the on-site nurse to identify and recruit participants from the JPMC outpatient department. Once a potential participant was identified, informed consent was obtained from the participant by the on-site facilitator.

2.4. Procedure

After obtaining consent from the participant, the on-site facilitator [22] and participant moved to a quiet area within the JPMC outpatient department to complete a questionnaire aimed at gathering sociodemographic and disease-related data, as well as information on access to basic mobile phones and smartphones, comfort level with smartphones, and ability to read Urdu to ensure diverse user groups for the usability testing. The usability testing was conducted immediately after recruitment, while the participants were waiting to see their doctor for a routine clinic visit. Due to time and travel constraints, the usability testing was conducted by a research analyst, who was located in Toronto, Canada, through a moderated remote usability testing approach [22]. The research analyst, who had experience conducting research in Pakistan and was familiar with the culture, population, and language, conducted the testing remotely via Zoom, with the on-site facilitator handling logistics and physical tasks. The moderated remote testing allowed the research analyst and participants to be in the same virtual space, facilitating observation and interaction during the testing [22].

During the usability session, the research analyst shared the computer screen to allow the participants to see the Raabta prototype. Participants were instructed and prompted to verbalize what they were looking at and what they were doing or trying to do (e.g., clicking on a button, scrolling up and down on the app, looking at content, reading, entering text) so that the research analyst was able to mimic their actions and follow along on the computer screen. The research analyst followed a script (Supplementary B) that was specifically developed to guide the usability testing of the Raabta app, and the script evolved during the course of testing based on user feedback and unexpected usability issues. Figure 2 illustrates the sequence of research activities.

Two usability techniques were used for the testing, which were the think-aloud [23] and task completion [27] techniques. For the think-aloud technique [23], participants are asked to look at the screens of the Raabta app and verbalize what they were thinking and feeling on each screen. For instance, participants were prompted to verbally express their initial impressions of the home screen, thoughts about the language, graphics, and features on the home screen, what they liked or disliked about the home screen, and what they would like to see improved. The research analyst encouraged the participants to communicate their actions and expectations of each screen. For the task complete technique, participants are asked to perform several tasks such as measuring blood pressure, completing symptom assessment, manually entering blood pressure readings, calling the doctor, and navigating previous readings. The research analyst observed the participant’s facial expressions and actions via the video throughout testing to assess task completion field [27]. The research analyst provided prompts only when the participant reached a roadblock or when the participant remained silent [27]. These techniques enabled the research analyst to determine the participants’ misinterpretations of design elements, which would be used to inform actionable redesign recommendations.

2.5. Data Analysis

All the usability testing sessions were audiotaped and later professionally transcribed and translated to English for analysis. The research analyst also made intuitive notes on the interaction of participants with the Raabta app screens. Interview transcripts and memos from researchers were uploaded to NVivo software (Version 12, released 2017, QSR International Pty Ltd., City of Manningham, Australia) for data analysis. The data analysis was primarily thematic [28], given that the intention was to uncover the major issues from usability testing and to understand user experiences with the Raabta app. Thematic analysis is a systematic method of breaking down and organizing rich data from qualitative research by tagging individual observations and quotations with appropriate codes, to facilitate the discovery of significant themes [28]. The thematic analysis involved several steps, including familiarization with the data, generating initial codes, searching for themes, reviewing and refining themes, and finally, producing a comprehensive analysis.

3. Results

3.1. Sample Selection and Participant Characteristics

A total of 35 pregnant women at high risk of preeclampsia were approached for usability testing, of which 14 (40%) individuals participated in the remote moderated usability testing. Reasons for nonparticipation included patients and accompanying caregivers being hesitant to leave the appointment line for testing, due to their concern about losing their place and potentially having to wait longer, and participants generally not wishing to participate in the study.

The participants ranged in age from 19 to 42 years (mean 29 years; SD ± 6.2). Half of the participants (n = 7) had access to a smartphone, of which four women had personal access to a smartphone. Half of the participants (n = 7) indicated that they are comfortable with smartphone use and have internet access. When asked about Urdu language literacy, half of the participants (n = 7) indicated that they are able to read the Urdu language. The detailed demographic information for all the patient participants is displayed in

Table 2. Characteristics of patient participants (n = 14).

Characteristics of Pregnant Women at HRPE	Category	N (%) or Mean ± SD
Gender	Female	14 (100%)
Maternal Age (years)		28.7 ± 6.2
Educational Level	No Education	2 (14%)
	Primary Education	4 (29%)
	Secondary Education	5 (36%)
	University Degree	3 (21%)
Occupation	Housewife	9 (64%)
	Professional	5 (36%)
History of Preeclampsia	Yes	5 (36%)
	No	9 (64%)
Gestational Age (weeks)		29.6 ± 6.9
Gravida (total number of pregnancies)	Primigravida	3 (21%)
	Multigravida	11 (79%)
Parity (livebirths and stillbirths)	Nulli-parity	3 (21%)
	Multiparity	11 (79%)
Access to a personal home blood pressure machine	Yes	4 (29%)
	No	10 (71%)
Personal or shared access to a mobile phone	Personal access	6 (43%)
	Shared access	7 (50%)
Able to use smartphone	Yes	7 (50%)
	No	7 (50%)
Access to the internet	Yes	7 (50%)
	No	7 (50%)
Able to read Urdu language	Yes	7 (50%)
	No	7 (50%)

.

Following the guidance from Nielsen J on the suggested sample size for usability testing [13], a total of 14 participants were recruited in usability testing sessions to ensure approximately 5 participants in each group. Among these groups, five participants were able to both read Urdu and were experienced with using a smartphone, five individuals were not able to read Urdu and did not have experience with using a smartphone, two participants could not read Urdu but had experience using a smartphone, and the remaining two participants were able to read Urdu but did not have experience using a smartphone. Usability testing sessions averaged 70 min (range 45–75 min). The thematic analysis uncovered four themes: improving the clarity of instructions, messaging, and terminology; accessibility for the non-tech savvy and illiterate Urdu users; enhancing visuals and icons for user engagement; and simplifying navigation and functionality.

3.2. Theme 1: Improving Clarity of Instructions, Messaging, and Terminology

Users expressed that the instructions from the app were sometimes unclear. For example, some users did not understand how to take a blood pressure measurement when the Raabta app instructed them to do so. Users suggested that the instructions should explicitly state that the provided blood pressure machine is to be used to take the reading.

“I got confused when it told me to measure my blood pressure. I didn’t know which device to use until someone explained it to me. They should make it clear that we should use the blood pressure machine they provided.”

(User 05, 27 years old)

Furthermore, users encountered difficulties in understanding the terminology used within the Raabta app. During usability testing when users were instructed to call a doctor, it was observed that they mistakenly clicked on the Raabta button (home button) located at the bottom right of the screen, assuming it would initiate the call instead of using the designated ‘call button’ positioned at the bottom left. This confusion stemmed from the literal meaning of the word “Raabta”, which means “connection”. To address this issue, users recommended changing the name of the home button to “homepage” in Urdu, to enhance comprehension and functionality.

“I thought ‘Raabta’ meant ‘call’ because of the ‘connection’ meaning. I clicked on it to call the doctor, but it didn’t work. They should change the name to ‘homepage’ to make it clear.”

(User 13, 32 years old)

The majority of the users were able to understand the symptom assessment question and found it easy to answer one yes/no symptom assessment question. However, some users attempted to select specific symptom graphics, mistakenly thinking they could indicate their specific symptoms. For example, one woman persistently tried to select the headache symptom by clicking on the graphic, intending to inform the doctor about her headache. To improve the messaging of the symptom assessment task, the users suggested that the messaging for the symptom assessment task should be amended to ensure that users understand the task accurately. Users suggested that the symptom question should say ‘Do you have any of the following symptoms? Select yes or no’ instead of just saying ‘Do you have any of the following symptoms?’.

“I tried to click the symptom pictures to tell the doctor about my headache, but it didn’t work. They should change the message to say ‘Select yes or no’ so we know what to do.”

(User 02, 19 years old)

3.3. Theme 2: Accessibility for Non-Tech Savvy and Illiterate Urdu Users

Users who could not read Urdu or use smartphones due to limited technology literacy valued voice messages and visual depictions. Voice messages and visuals enabled users to monitor their blood pressure and answer symptom-related questions independently. For instance, women who were not able to read Urdu understood the blood pressure measurement task more easily compared to the group of women who were able to read Urdu given that these women solely relied on the voice messages. Users even requested the option to repeat voice messages, highlighting its importance for comprehension.

“You know, I couldn’t read Urdu, but I got the hang of that symptom assessment thing with the voice messages and pictures. It was like someone was guiding me. And we need the option to hear those messages again, it really helps us understand.”

(User 08, 40 years old)

The majority of users successfully completed the symptom assessment, with Urdu readers comprehending the written symptom question and providing accurate responses. Another user group, consisting of individuals who were either unable to read Urdu and/or operate a smartphone, utilized the voice message feature and symptom illustrations to complete symptom assessment.

3.4. Theme 3: Enhancing Visuals and Icons for User Engagement

Users indicated the significance of visual enhancements to improve user engagement and understanding. For instance, users provided valuable suggestions to improve the visuals for the symptom assessment task to ensure an accurate understanding. Specifically, users advised changing the visuals for symptoms like blurred vision and upper abdominal pain to facilitate recognition for illiterate Urdu users. For instance, using a graphic of a woman holding her upper abdomen could represent upper abdominal pain.

“I really felt that better visuals could help us understand things more clearly. I suggested they change the pictures, especially for things like blurred vision and upper abdominal pain. You know, like using a picture of a woman holding her upper abdomen for upper abdominal pain.”

(User 11, 24 years old)

In some instances, users suggested modifying the color scheme to enhance user engagement and ensure cultural alignment. For example, they recommended changing the color scheme for the “yes” and “no” buttons of the symptom card to blue and red, respectively, instead of purple. This suggestion aimed to improve the visibility and clarity of the buttons, making them more intuitive and culturally familiar to the users.

Participants also highlighted the importance of enhancing icon recognition to ensure a more intuitive understanding and improved user experience. Specifically, users expressed confusion regarding the voice message icon in the Raabta app, as it closely resembled the icon used in WhatsApp. This confusion particularly affected non-tech-savvy users. To address this issue, participants proposed alternative icons that better represent the voice messaging feature. Some suggestions included using a cartoon character holding a microphone or an old-fashioned audio cassette tape. Furthermore, users appreciated the green, yellow, and red alerts during the usability testing. Most were able to take appropriate actions as indicated in the alerts. Tech-savvy and Urdu language-proficient users were able to understand the written text accompanying the alerts. To improve comprehension and clarity of the alert messages, users recommended incorporating icons alongside written alerts. In particular, users suggested symbols such as a checkmark for green alerts, a cross for red alerts, an ambulance for emergency alerts, and a clock for alerts requiring attention.

“You can add some symbols with red, yellow, and green colors. Like a checkmark for green, maybe a cross for red, or an ambulance for emergencies. It would help everyone.”

(User 04, 28 years old)

3.5. Theme 4: Streamlining User Navigation/Simplifying Navigation and Functionality

Users faced challenges with navigation both between screens and within screens, highlighting the need for enhanced navigation within the Raabta app to enhance the overall user experience. A specific example of this was the difficulty users encountered when attempting to navigate back to the current screen using the designated ‘back to today’ button. Instead of relying on this button, users resorted to using the scroll function to access previous readings and eventually return to the present screen. This particular navigation issue posed a significant challenge, especially for non-tech-savvy users with limited Urdu language literacy.

“Using the app was a bit tough, especially when I tried to go back to the current screen. There was this ‘back to today’ button, but it didn’t make sense to me. I just scrolled back to get to where I wanted. I can imagine it being even harder for those who aren’t good with phones or don’t read Urdu.”

(User 10, 31 years old)

Another common challenge was encountered when users attempted to manually enter their blood pressure readings, regardless of their level of familiarity with smartphones. The majority of users faced obstacles while navigating through the required steps for manual entry, particularly when the readings did not automatically synchronize with the paired app. To address this challenge and enhance the accessibility of the manual entry feature, users suggested incorporating a dedicated manual blood pressure entry button on the home screen.

4. Discussion

A formative usability testing was carried out to help ensure the Raabta app is efficient and easy to use for pregnant women at high risk of preeclampsia in Pakistan. This study provides an in-depth investigation into the usability issues in the Raabta app and offers insights to improve the Raabta app. The study leveraged think-aloud [23] and task completion [27] methodologies to identify usability issues in the early stage of the Raabta app development. The Raabta app is unique in that it was designed for diverse end-user groups with respect to the Urdu language and technological literacy [10,11,12]. User engagement has been a key focus of this study, aligning with the principles advocated by previous research [24]. Engaging end-users in the development process is a well-documented approach that has been shown to bridge the gap between existing technology products and populations who would most benefit [14]. By actively engaging end-users, we aimed to gain a deeper understanding of the intuitive user interface of the Raabta app and ensure its suitability for the target population.

In our study, participants emphasized the importance of enhancing the clarity of instructions, messaging, and terminology within the Raabta app. In particular, users recommended clearer instructions for blood pressure measurement tasks and suggested amending the symptom assessment question to ensure an accurate understanding. Non-tech savvy and illiterate Urdu users highly valued accessible features like voice messages and visuals, enabling independent monitoring of blood pressure and symptom assessment. Visual enhancements, including improved graphics and culturally aligned color schemes, were recommended to enhance user engagement. In a study conducted by Reinecke et al., participants using the culturally adapted interface performed tasks 22% faster, needed fewer clicks, and made fewer errors. These findings show that interfaces that adapt to cultural preferences can immensely increase the user experience [29]. Similar to other studies, simplifying the navigation and functionality within the app was deemed crucial, with users facing difficulties in navigating between screens and within screens.

The study also uncovered the Importance of identifying user subgroups based on traits such as technological literacy, language literacy, and demographic and disease characteristics [25] for capturing the usability issues by user subgroups. As such, our study offers an approach for conducting formative evaluations of mobile applications to assess their usability and suitability for diverse user subgroups during the development process. Barnum et al. suggest that researchers need to agree on the user subgroups as well as define the characteristics of each subgroup in the planning stage for effective usability testing [26]. In our study, the user subgroup with competency in using smartphones was able to complete most of the tasks in the Raabta app, whereas the non-tech-savvy users had difficulties in understanding some of the design elements. The insights gained from this study regarding user subgroup identification could be useful for researchers who are creating digital health interventions for various subgroups of users. Further research is needed to investigate how user characteristics influence the usability of mobile-based telemonitoring programs.

This study emphasizes the importance of conducting usability testing before implementing a product, to ensure that it is culturally appropriate, usable, and acceptable to the end users—in this case, pregnant women at high risk of preeclampsia in Pakistan. While there is increased attention toward providing culturally competent care, very little is known about how to design culturally competent health interventions delivered through mobile-based telemonitoring programs [30]. Available evidence in this field proposed the incorporation of culturally specific symbols, colors, and textures within the app to enable users to understand and interpret tasks, regardless of their level of familiarity with technology [31,32]. In our study, we designed a culturally appropriate prototype for Raabta by taking into consideration the needs of high-risk pregnant women [10] and their caregivers [11], as well as their lifestyles, cultural practices, socioeconomic status, social ties, and feedback from local clinicians. Through usability testing of a culturally appropriate mobile-based telemonitoring program that included the local language (Urdu), culturally appropriate written and recorded self-care messages, graphics, and other interactive features, this study has contributed to the field of culturally competent care. However, further research is needed to understand the key characteristics of a culturally competent mobile-based telemonitoring program and how to evaluate these aspects in usability evaluations.

Finally, this study highlights the importance of conducting remotely moderated usability testing in settings with budget and time constraints and a scarcity of human factor expertise [22]. In our study, the remotely moderated usability testing enabled the researcher to conduct the testing from her own location without having to travel. However, this kind of testing requires considerable advanced preparation as well as support from an on-site facilitator to help with logistics and physical tasks [22]. Researchers can employ this technique when UX-research resources are limited, participants are geographically distributed, and the facilitator and the participant cannot be in the same physical location. Future research should call on human factors, human–computer interaction, and digital health communities to make remote usability testing more accessible, efficient, and effective for diverse populations, including those with lower technological literacy or language proficiency [18].

4.1. Next Steps

Findings from the usability testing will be used to carry out the final development of the Raabta application for the purpose of conducting a feasibility trial. The feasibility study will consist of providing the Raabta app to 25–50 pregnant women at high risk of preeclampsia. The trial will help assess the feasibility of implementing the Raabta app for high-risk pregnant women, as well as inform additional revisions before rolling out the Raabta app for a future large effectiveness trial. The intent is for the Raabta app to be expanded for use in other health systems where pregnant women at high risk of preeclampsia face socio-cultural and financial constraints for receiving timely care during pregnancy, after similar usability testing is conducted to understand the required modifications for use within the different communities.

4.2. Limitations

One limitation of this study was that only one round of usability testing was conducted, mainly due to time constraints. However, no major issues were identified during the initial round that would have warranted the need for a second round of testing. Second, this usability study was conducted with patients in a single setting (JPMC) which represents diverse user groups in terms of Urdu language literacy and technological literacy; therefore, the findings may differ regionally and nationally. Third, the remote moderated usability testing might have influenced the way in which participants perceived and engaged with the Raabta app due to the absence of direct in-person interaction with the app. The absence of direct in-person interaction with the app, which is typical in traditional usability testing, could have introduced a different dynamic to the user experience. Participants might not have felt the same level of personal connection or real-world context that in-person testing provides. This could potentially affect how they navigated and interacted with the app, as the testing environment differed from their usual circumstances. Therefore, while the remote approach allowed for a broader geographic reach and convenience, it is essential to consider that it might not fully capture the nuances of how users would engage with the Raabta app in their daily lives. Fourth, two of the user groups had only two participants for the usability testing, which could limit the transferability of the findings for individuals who are either unable to read Urdu or not at ease with using smartphones.

5. Conclusions

The study evaluated the Raabta app’s usability, highlighting the need for clearer instructions, messaging, and terminology. The non-tech savvy and illiterate Urdu users valued accessible features like voice messages and visuals. Recommendations included visual enhancements and simplified navigation to enhance user engagement and address navigation challenges in the app. The study also emphasized the significance of identifying user subgroups based on technological literacy, language proficiency, and demographic and disease characteristics, providing an approach for formative evaluations of mobile applications. Culturally competent care was considered in the design of the Raabta app, but further research is needed to evaluate and refine culturally competent mobile-based telemonitoring programs. This study also provided valuable lessons for researchers interested in conducting remotely moderated usability testing in resource-constrained environments. The findings will inform the final development of the Raabta app and a future feasibility trial of the Raabta program to support high-risk pregnant women in Pakistan.