Next Article in Journal
Respiratory Symptoms and Skin Sick Building Syndrome among Office Workers at University Hospital, Chiang Mai, Thailand: Associations with Indoor Air Quality, AIRMED Project
Previous Article in Journal
The Appraisal of Self-Care Agency Scale-Revised (ASAS-R): Reliability and Validity among Portuguese Medical Students
Previous Article in Special Issue
Is Team Resilience More Than the Sum of Its Parts? A Quantitative Study on Emergency Healthcare Teams during the COVID-19 Pandemic
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 19
Issue 17
10.3390/ijerph191710834
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

I Need to Know: Using the CeHRes Roadmap to Develop a Treatment Feedback Tool for Youngsters with Mental Health Problems

by
Ilja L. Bongers
1,2,*,
David C. Buitenweg
1,
Romy E. F. M. van Kuijk
2 and
Chijs van Nieuwenhuizen
1,2
1
Institute for Mental Health Care Eindhoven (GGzE), Centre for Child and Adolescent Psychiatry, P.O. Box 909, 5626 ND Eindhoven, The Netherlands
2
Scientific Center for Care & Wellbeing (Tranzo), Tilburg University, P.O. Box 90153, 5000 LE Tilburg, The Netherlands
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(17), 10834; https://doi.org/10.3390/ijerph191710834
Submission received: 29 June 2022 / Revised: 5 August 2022 / Accepted: 10 August 2022 / Published: 31 August 2022
(This article belongs to the Special Issue New Normal in Mental Health Care)
Browse Figures
Review Reports Versions Notes

Abstract

:
Patient-Reported Outcome Measures (PROMs) are often used to monitor treatment outcomes in youth mental health care. Unfortunately, youngsters are rarely informed about the results of their PROMs or, when they are, it is in an insufficient manner. Therefore, a web application was developed—together with youngsters—aimed at giving them feedback about their PROMs. The aim of this study is to describe the development process of the application. An expert panel consisting of youngsters, web designers and researchers, as well as a representative from a client organisation, developed the e-health application INK (short for ‘I Need to Know’) in an iterative process based on the Centre for eHealth Research roadmap (CeHRes roadmap). Youngsters prefer, among other aspects, a simple, easy-to-use e-health application with a colourful appearance and want to be able to compare their results across different time points and informants. The INK tool provides youngsters with insight into their PROM results. Based on the youngsters’ preferences, INK users can choose which feedback information is visible. INK facilitates youngsters’ active participation in their treatment as well as shared decision-making with their professional caregivers.

1. Introduction

Providing feedback to youngsters in youth mental health services is of utmost importance [1,2,3,4]. Feedback has a consistently positive effect on youngsters during their treatment [5]. It has, for instance, a positive effect on youngsters’ well being [4] and a small effect on dropout reduction [1]. Including regular outcome feedback in youth mental healthcare helps prevent and attenuate the potentially severe long-term effects of mental health problems [6]. Taking into consideration the numerous advantages of providing outcome feedback to youngsters in youth mental health services, it is surprising that this feedback is not always provided [7,8,9]. Particularly in light of the new opportunities that digital technologies provide. E-mental health applications have the flexibility to tailor content in a personalized way [10,11] and engage users in real-time [12]. Both personalisation and real-time engagement have been found to improve responsiveness and patient motivation [13]. Moreover, e-mental health applications seamlessly fit in with everyday experiences and allow youngsters to take the lead [14,15,16]. E-mental health is, therefore, especially suited to providing feedback to youth in mental healthcare.
The importance of co-design and the active involvement of youngsters and other stakeholders across the entire design process of an e-mental health application or intervention has been highlighted in several studies [17,18,19]. Co-design is more than the consultation of stakeholders; it involves exploring and articulating needs and developing solutions together with them. In this way, co-design results in a richer and deeper understanding of the end-users. The Centre for eHealth Research roadmap (CeHRes roadmap; [20]) provides a framework for involving end-users in the development process of e-mental health applications through co-design. Co-design has been identified as an effective design philosophy for guiding the development of e-mental health applications for use in youth mental healthcare [17,19]. The benefits of co-design are well established: it enhances the range of available design ideas [19], increases the understanding of user needs [17] and improves the support and success of the application by focusing on user needs [18,19].
As of yet, little work has been performed to identify youngsters’ preferences regarding the design and functionality of a digital Patient-Reported Outcome Measurement (PROM) feedback tool. A recent qualitative study by Mayworm and colleagues [21] revealed that easy-to-use, personalisation and feedback are important preferences for middle and high school students. It is unclear, however, whether these preferences remain when explored in the context of the development of an e-mental health PROM feedback tool for youngsters with mental health problems. Therefore, the aim of this study involves the co-design-driven development of a digital PROM feedback tool for use in youth mental health care. Youngsters’ preferences regarding the design and functionality of the tool receive special attention.

2. Methods

2.1. Expert Panel

An expert panel was involved throughout the process of the design and development of a digital feedback tool INK (short for ‘I Need to Know’). As the literature emphasises the importance of dialogue between designers, end-users and other important stakeholders [20], the panel consisted of youngsters, web designers with a focus on user experience and a researcher, as well as a representative from a client organisation. Seven youngsters aged 12–18 were invited to participate in the panel, and all of them were required to have (previous) experience with youth mental healthcare and to have been treated in a mental health care institute for psychiatric problems. The representative of the client organisation recruited the youngsters. The client organisation is a network of client advisory boards in care and wellbeing (www.loc.nl) (accessed on 27 July 2022). A minimum of three youngsters was required to proceed with individual panel meetings. Every panel meeting was attended by three to seven youngsters (mean attendance: 5.4 youngsters), two user experience web designers, a researcher and a representative from a client organisation. Panel meetings took place in an informal setting in the region where the youngsters were living on Wednesdays from 17.00–18.30. Various actions were undertaken to minimize power structures that typically exist between adults and children, such as engaging in informal activities to get to know each other, using informal language, wearing casual outfits and eating snacks and dinner together. After each expert panel meeting, the researchers and designers discussed the meeting and prepared the subsequent design steps and the next panel meeting. An iterative process was established as panel meetings were based on the information gathered in the previous panel meetings. The meetings were not recorded, but the researcher took extensive notes and wrote a report of the meetings. At the start of each panel meeting, the general outline of the meeting was shared with all the youngsters. Next, a recap of the previous meeting was given, and the report of the previous panel meeting was validated.

2.2. Development Process

The iterative co-design-driven development of the digital feedback tool INK was structured according to the first three phases of the CeHRes roadmap: (1) Contextual Inquiry, (2) Value Specification and (3) Design [17,20]. In practice, these three phases were not executed sequentially but were interwoven throughout the developmental process. The CeHRes roadmap is based on various evidence-based models and frameworks, such as human-centred design and participatory development [20]. The three phases of the CeHRes roadmap are schematically displayed in Figure 1. Below, the phases and how they have been adapted to fit the current study are described in more detail. To triangulate data and develop a better understanding of youngsters’ needs, the CeHRes roadmap involved a formative evaluation of the results gathered during each phase.

2.2.1. Phase 1: Contextual Inquiry

The main goal in this first phase was to achieve an understanding of future users and the current situation. In the current study, the Contextual Inquiry was initiated by identifying stakeholders and establishing an expert panel to represent future users. In addition, to provide youngsters with insight into the current situation of the PROM feedback that was offered a patient journey was incorporated (Table 1). This is an interactive method to allow youngsters to become more familiar with the research project and the goal of the co-design-driven development of INK [22].

2.2.2. Phase 2: Value Specification

The Value Specification phase served to clarify what values were important to future users and how these values may be translated into user requirements. In the current project, the youngsters were asked what functionalities they would like to see in INK. In addition, they were invited to share factors that would prevent them from using INK or that would encourage them to employ the tool. Furthermore, youngsters were invited to share their preferences regarding the design of INK. Based on this input, the relevant values were formulated.

2.2.3. Phase 3: Design

The Design phase started with a series of sketching exercises by the youngsters. First, they were invited to sketch as many possible visualisations of questionnaire results as possible using pencil and paper. Next, they were asked to draw possible depictions of (1) the distinction between high and low scores, (2) the results of assessments at different time points and (3) the outcome of questionnaires answered by different informants. Based on these images, the designers constructed five possible directions for the design (paper prototypes). The panel discussed these alternative directions and the preferred one was explored further using digital mock-ups. Different colour schemes were compared and discussed by the panel. Next, the designers developed a low-fidelity prototype that was examined extensively with the panel. The youngsters were invited to test the prototypes using either an Apple iPad 2 or Apple iPad 2017 provided by the researcher or their personal smartphones. Based on their feedback, the prototype was adjusted and refined where necessary, resulting in a final version of INK.

2.2.4. Formative Evaluation

In this study, a mixed methods design was employed for the formative evaluation to triangulate the data and gain a better understanding of youngsters’ needs. The needs identified in the panel meetings were categorized into values, and these were translated into design options in INK. Throughout the development of INK, youngsters were invited to evaluate the current sketch, design or prototype using various methods, and after each evaluation, the results were discussed with the panel. During the initial steps of the development, the youngsters were invited to use four different stickers to assess ideas for the design and functionalities of INK (see Figure 2). Later, they used sticky notes to express their opinions about the sketches and prototypes. Moreover, decisions were sometimes made democratically by choosing the option that received the highest number of votes from the panel.
Near the end of the development of INK, the youngsters were invited to evaluate the degree to which their preferences were reflected in the tool. Usability tests were used to observe how youngsters navigate through the application and to see whether they were able to complete typical tasks. The interpretation of feedback circles was tested using a structured interview. They were shown four different feedback circle options. The one that was most often interpreted in a correct way was chosen for the final version of INK. The layout and colour scheme were tested using a questionnaire with a 5-point Likert scale.

2.3. Ethical Approval

This study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of the Tilburg School of Behavioural and Social Sciences at Tilburg University (EC-2017.78). Informed consent was obtained from each participant. The youngsters received 25 EUR and a reimbursement of travel expenses via bank transfer for each panel meeting they attended. Occasionally, they were given small gifts such as a cinema voucher or card game during the course of the development process. Personal details were collected with the participants’ consent to make bank transfers possible.

3. Results

The co-design-driven development process consisted of ten sessions with the expert panel. The results were structured according to the three phases of the CeHRes roadmap: Contextual Inquiry, Value Specification and Design. The results of each phase were visualized with sketches and screenshots of the low- and high-fidelity prototypes. During the Contextual Inquiry (Phase 1), the trajectory of a typical treatment of a youngster was described using a patient journey. Important values (Phase 2) for future users were identified, and their translation into user requirements was discussed in the Value Specification phase. In the Design phase (Phase 3), the values were translated into a prototype of the digital tool INK.

3.1. Phase 1: Contextual Inquiry

During the Contextual Inquiry (Phase 1), the trajectory of a typical treatment of youngsters receiving mental healthcare was described using a patient journey. At the start of treatment, youngsters, their parents and sometimes their teachers were asked to fill out questionnaires in a typical treatment trajectory. In the panel meeting, the youngsters discussed the order of events in a typical treatment trajectory. For instance, whether they would prefer to complete the questionnaire before attending the mental healthcare institute for the first time or after. The youngsters in the panel meeting said that they only rarely received feedback about the questionnaires they filled out. They would, however, like to be given feedback, preferably directly after they finished the questionnaires.

3.2. Phase 2: Value Specification

An overview of all the youngsters’ preferences and needs regarding the functionality of INK is provided in Table 2. The needs were translated into design requirements in INK. Not all needs initially mentioned by youngsters were incorporated into the final version of the INK application due to technical and organisational constraints. Needs that were not part of the final version are denoted with an asterisk in Table 2. As can be seen in Table 2, the needs were categorized based on five values: Being well-informed, Personalized user experience, User friendly, Being in control and Privacy and safety.

3.2.1. Phase 2 Value 1: Being Well-Informed

During the expert panel, the youngsters emphasized that they want to see how their treatment progresses in one screenshot. They also mentioned the importance of being well-informed about specific behaviours and not only receiving general feedback about their treatment progress. They indicated that it was important to view results from several time points and multiple informants in a single overview. The youngsters used a ‘lock’-sticker to indicate that this was essential for the INK tool (see Figure 3).
All needs related to the value of Being well-informed were translated into the final design of INK, and the way in which feedback should be visualized, in particular, was thoroughly discussed with the youngsters during the panel meetings. The general feedback was visualized using the colours green, orange and red to represent normal, subclinical and clinical scores, respectively. Besides the general feedback, specific feedback was added to the INK tool. By clicking on the green, orange or red feedback circle, a feedback score appears in numbers (range 0–100). According to the youngsters, these so-called ‘traffic light’ colours are familiar and likely to evoke similar associations among people (see Figure 4). However, they emphasized that these colours should be used modestly. It can be somewhat startling to see that your own scores are green and those for other informants show red or to see that your previous scores were green but new ones are red. At the same time, the youngsters did find it important not to sugar-coat negative scores: ‘it is how it is’.

3.2.2. Phase 2 Value 2: Personalized User Experience

The youngsters emphasized the importance of the personalisation of their user experience. One suggestion was to create a profile page where youngsters can upload a personal picture and list some information, such as their name, age and personal interests. Later on, in the development process, this idea was replaced by using icons and avatars that are engaging, fun and age-appropriate (12–18 years), according to the expert panel. This resulted in functionality in which youngsters can ‘build’ their own avatar. Within the avatar builder, users can select their preferred gender and clothing as well as skin and hair colour (Figure 5). Youngsters commented that the avatar builder should not be too extensive, as the primary goal of the INK application was not to provide a game-like experience but to provide treatment feedback: ‘It is fun that you’re able to choose your own character, but keep it simple’. Another feature that enhances the personalisation of the experience was the matching of the avatar’s facial expressions with both the topic of the questionnaire and the displayed scores (see Figure 5). Lastly, there is an option for users to select their colour of preference for the general layout of the INK application.

3.2.3. Phase 2 Value 3: User Friendly

Youngsters mentioned that a cheerful, non-depressing appearance of the tool motivated them to use INK, whereas the colour red, as well as sad or avoidant avatars, demotivated them (see Figure 6). The youngsters emphasized that INK should be applicable to all youngsters, so the language used in INK was adapted to the reading level of youngsters, and special attention was paid to avoiding stigmatizing language. For instance, the term ‘emotion and behaviour’ is used instead of ‘complaints’. During the panel meeting, special attention was paid to the usability of INK, resulting in an intuitive structure that makes INK easy to use. Only a few clicks are required to navigate through INK.

3.2.4. Phase 2 Value 4: Being in Control

Youngsters stated that it is important that they are in control during their treatment. They emphasized being able to take control when they receive feedback about their treatment progress. The benefits of being informed were discussed in the panel meeting, and the youngsters mentioned that INK could also facilitate shared decision-making. Several features of INK give the youngsters control over the feedback they want to receive. For example, users are able to select or hide the informant or the scale scores based on what they want to see. INK has a predefined email format, which the youngster can use to ask for help (see Figure 7). All the options and functionalities of INK are explained in a short, clear tutorial that is always accessible to users. Not all preferences expressed by the youngsters that related to the value of Being in control could be implemented in INK. For instance, the youngsters suggested a chat option with their therapists and peers or an option to indicate when their opinions differ from their parents or teachers to facilitate discussions with their clinicians. These options could not be implemented because of technical constraints.

3.2.5. Phase 2 Value 5: Privacy and Safety

The youngsters highlighted the importance of receiving treatment feedback in a safe way and that their activities in INK should not be followed by any organisation. Tracking and personalized advertisements abolished the feelings of INK being safe to use. Therefore, the youngsters in the expert panel were strongly opposed to the tracking of their use of INK. For safety and privacy purposes, only youngsters themselves can use the INK application, using their individual email address provided through the electronic patient file of the mental health institute and a password (see Figure 8). In this way, the safety and privacy of the information shared in INK are assured, and user information collected by INK will not be stored.

3.3. Phase 3: Design

Using paper and pencils, youngsters visualized the different questions or scales of the PROM questionnaires. Most sketches focused on faces and facial expressions (see Figure 9).
The initial suggestions, sketches by youngsters and identified values were used to create the first paper or low-fidelity prototypes of INK (see Figure 3 and Figure 4, for example). During the iterative development process, the list of needs regarding the design of INK expanded. The INK application should have a minimalistic, simple design. Although the questionnaires’ topics can be quite tense, youngsters clearly stated that the INK application should have a cheerful, non-depressing appearance: ‘Seeing a happy face can give a sense of relief, whereas a sad face can also make you feel sad’. According to youngsters, the design should be colourful but not too vibrant in order to maintain a simplistic design. However, most youngsters preferred pastel shades, some preferred bright, cheerful colours for the design of INK. Therefore, youngsters are able to select their preferred colour in the INK application. Based on this developmental process, the first prototype was designed. The usability of the prototype was tested with six youngsters of the expert panel using a structured interview; they all interpreted the feedback circles correctly. The layout and colour scheme were tested, and the youngsters were not satisfied with the colours of INK (see Figure 3, Figure 4 and Figure 7). Therefore, the colours changed from blue in the high-fidelity prototype to purple in the final version. See Figure 10 for the final version of INK that is implemented in a mental healthcare institute for child and adolescent psychiatry.

4. Discussion

This study involved the co-design-driven development of INK, a digital PROM-feedback tool for use in youth mental healthcare. Sharing feedback with youngsters in treatment is ubiquitous, but INK is unique. Although there are feedback tools for adults [2,23] or feedback tools that help clinicians to share feedback with clients [24], feedback tools developed especially for young people in mental healthcare settings in co-creation with youngsters were not available up to now. The contribution by the youngsters in the expert panel resulted in a final version of INK that enables youngsters to view the results of their PROMs in an accessible, appealing and intuitive way. The design requirements articulated by the expert panel related to the values Being well-informed and Privacy have all been implemented in INK. Only some of the requirements related to the value of Being in control could not be implemented in the design due to technical and organisational constraints. Whereas the values involving being User friendly and Personalized user experiences were important for the preferences of the youngsters according to the design and layout of INK, the values Being well-informed in addition to Privacy cover the main goal of INK.
The five values and corresponding user requirements identified in this study are comparable to the ones identified in the literature [4,25]. Being well-informed when receiving treatment feedback is an important value for youngsters. According to the youngsters in the panel, feedback is most useful when it involves insight into changes over time, is provided by multiple informants, includes additional information on questionnaires, scales or items and is comprehensive and well-organized. The importance of this value is in line with previous studies [12,26,27,28,29]. Feedback becomes more valuable for youngsters when it provides them with sufficient information to accurately interpret their PROM results. Although feedback does not directly increase the effectiveness of treatment [7], feedback can support shared decision-making, which increases compliance with the recommended treatment [30]. Hence, the use of INK to complement treatment may indirectly improve the effect of treatment.
Personalisation is frequently cited as one of the important advantages of e-mental health applications and is also related to increased engagement [31,32]. A tool that offers personalisation to the interests and needs of the user has a greater capability of persuasion and increases engagement with and adherence to the application [33]. It is, therefore, not surprising that the youngsters indicated the importance of age-appropriate content and a customizable avatar and background colour scheme. Similar preferences have been identified in studies with children and adolescents by Beck and colleagues [12] and Hetrick and colleagues [27]. In a study by Buitenweg and colleagues [26], however, young adult and adult participants explicitly rejected customizable avatars or colours. This difference between youngsters and adults implies that personalisation and user friendliness is age dependent, so it is important to identify relevant end-users and embrace them in the development process [17].
The youngsters in the panel emphasized the importance of the user friendliness of e-mental health applications in general and digital PROM-feedback in particular. Similar values have been identified by youngsters in virtually all comparable studies [12,34]. The way youngsters interact with a digital tool influences their engagement as well as the extent to which they revisit the tool [35,36]. This finding confirms the importance of ensuring that users find it easy to interact with and navigate through an e-mental health application.
The youngsters extensively indicated their need to be in control when using a digital PROM-feedback tool as well as the secure storage of their data (the Privacy and safety value). Having control over data access was highly valued by youngsters. The ability to notify other individuals and to have control over data access echoes findings by several other works [12,26,37]. It is important to note that youngsters’ preference for having control over data access may conflict with clinicians’ wish to have unrestricted access to youngsters’ PROM results. In addition, studies indicate that feedback is most effective when provided to both youngsters and clinicians [2,5]. Maathuis and colleagues [38] identified this value conflict during the development of a web-based quality of life assessment instrument for youngsters and adults with mental health problems. These results emphasize that there exists a potential conflict between youngsters’ need for autonomy and clinicians’ wish to have unrestricted access, which may influence the effectiveness of feedback. The user requirements identified in this study may inform the future development of digital feedback tools for youngsters with mental health problems. Adhering to youngsters’ preferences whenever possible may improve the often-problematic implementation of treatment feedback for youngsters with mental health problems [39].

5. Strengths and Limitations

The involvement of both youngsters and web designers in the panel is an important strength of this study. As Mulvale and colleagues [18] and Thabrew and colleagues [19] indicated, it is vital that web designers come into contact with the end users throughout the development of an e-mental health application. The consistency of the composition of the panel is another important strength. The youngsters were involved in multiple panel meetings (up to ten meetings). This enabled them to provide feedback on the current version of INK and to monitor to what degree their feedback was processed in subsequent versions. At the same time, three important limitations should be taken into account when interpreting the results of this study. The first limitation pertains to the size of the panel as well as the sampling strategy employed to recruit it. The youngsters joined the panel based on a convenience sampling strategy. The results should, therefore, be generalized with caution. In addition, the number of youngsters who contributed to the panel was limited, and little information regarding their background was available. Some panel meetings involved only three or four youngsters. The small size of the panel, however, also had two important benefits. It facilitated collaboration and resulted in an atmosphere of trust in which the youngsters could share their experiences and opinions. The absence of care professionals in the panel forms a second limitation. The professional perspective has been identified as very important in the development of e-mental health applications [40,41]. The preferences described in this study were not identified in systematic experiments but during an organic development process, which forms the third limitation. The ecological validity of the results, however, may benefit from this.

6. Future Research

This study provides a number of important opportunities for future research. First, it is crucial to assess to what degree the use of INK by youngsters in their treatment improves their treatment satisfaction and symptom levels or other clinical outcomes. Second, research into the optimal implementation strategies for INK is essential. As the literature reveals and experience teaches, many digital mental health tools fail to have an impact on the lives of youngsters and clinical practice. Frameworks such as the Consolidated framework for implementation research [42] or Proctor’s framework of implementation outcomes [43] may be helpful when studying the implementation of INK [44]. Improving the flexibility of INK so that it may be used in multiple care settings and by various groups, such as parents, is a third important avenue for future research. Currently, INK includes a limited number of questionnaires. By adding further ones, INK would become useful for additional patient groups.

7. Conclusions

An easy-to-use tool for treatment feedback has been developed, together with youngsters. The five values of Being well-informed, Personalized user experience, User friendly, Being in control and Privacy and safety were identified during the development and have been incorporated into the final version of the tool. INK enables youngsters to actively participate in their treatment and to make informed decisions together with their professional caregivers.

Author Contributions

Conceptualisation, I.L.B. and C.v.N.; Formal analysis, R.E.F.M.v.K.; Funding acquisition, I.L.B. and C.v.N.; Methodology, I.L.B., R.E.F.M.v.K. and C.v.N.; Project administration, C.v.N.; Supervision, C.v.N.; Writing—original draft, I.L.B., D.C.B. and R.E.F.M.v.K.; Writing—review and editing, I.L.B., D.C.B., R.E.F.M.v.K. and C.v.N. All authors have read and agreed to the published version of the manuscript.

Funding

This research has been financed through a research grant from The Netherlands Organisation for Health Research and Development (ZonMw): 729410008.

Institutional Review Board Statement

This study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of the Tilburg School of Behavioural and Social Sciences at Tilburg University (EC-2017.78, 19 September 2017).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Conflicts of Interest

The authors declare no conflict of interest. The sponsors had no role in the design, execution, interpretation or writing of the study.

References

  1. De Jong, K.; Conijn, J.M.; Gallagher, R.A.; Reshetnikova, A.S.; Heij, M.; Lutz, M.C. Using progress feedback to improve outcomes and reduce drop-out, treatment duration, and deterioration: A multilevel meta-analysis. Clin. Psychol. Rev. 2021, 85, 102002. [Google Scholar] [CrossRef] [PubMed]
  2. De Jong, K.; Timman, R.; Hakkaart-Van Roijen, L.; Vermeulen, P.; Kooiman, K.; Passchier, J.; Busschbach, J.V. The effect of outcome monitoring feedback to clinicians and patients in short and long-term psychotherapy: A randomized controlled trial. Psychother. Res. 2014, 24, 629–639. [Google Scholar] [CrossRef] [PubMed]
  3. Krause, K.R.; Chung, S.; Adewuya, A.O.; Albano, A.M.; Babins-Wagner, R.; Birkinshaw, L.; Brann, P.; Creswell, C.; Delaney, K.; Falissard, B. International consensus on a standard set of outcome measures for child and youth anxiety, depression, obsessive-compulsive disorder, and post-traumatic stress disorder. Lancet Psychiatry 2021, 8, 76–86. [Google Scholar] [CrossRef]
  4. Tam, H.; Ronan, K. The application of a feedback-informed approach in psychological service with youth: Systematic review and meta-analysis. Clin. Psychol. Rev. 2017, 55, 41–55. [Google Scholar] [CrossRef]
  5. Gondek, D.; Edbrooke-Childs, J.; Fink, E.; Deighton, J.; Wolpert, M. Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: A systematic review. Adm. Policy Ment. Health Ment. Health Serv. Res. 2016, 43, 325–343. [Google Scholar] [CrossRef] [PubMed]
  6. Delgadillo, J.; de Jong, K.; Lucock, M.; Lutz, W.; Rubel, J.; Gilbody, S.; Ali, S.; Aguirre, E.; Appleton, M.; Nevin, J. Feedback-informed treatment versus usual psychological treatment for depression and anxiety: A multisite, open-label, cluster randomised controlled trial. Lancet Psychiatry 2018, 5, 564–572. [Google Scholar] [CrossRef]
  7. Garnett, C.V.; Crane, D.; Brown, J.; Kaner EF, S.; Beyer, F.R.; Muirhead, C.R.; Hickman, M.; Beard, E.; Redmore, J.; de Vocht, F.; et al. Behavior change techniques used in digital behavior change interventions to reduce excessive alcohol consumption: A meta-regression. Ann. Behav. Med. 2018, 52, 530–543. [Google Scholar] [CrossRef]
  8. Jensen-Doss, A.; Haimes EM, B.; Smith, A.M.; Lyon, A.R.; Lewis, C.C.; Stanick, C.F.; Hawley, K.M. Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Adm. Policy Ment. Health Ment. Health Serv. Res. 2018, 45, 48–61. [Google Scholar] [CrossRef]
  9. Lyon, A.R.; Lewis, C.C.; Boyd, M.R.; Hendrix, E.; Liu, F. Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Adm. Policy Ment. Health Ment. Health Serv. Res. 2016, 43, 441–466. [Google Scholar] [CrossRef]
  10. Bucci, S.; Morris, R.; Berry, K.; Berry, N.; Haddock, G.; Barrowclough, C.; Lewis, S.; Edge, D. Early psychosis service user views on digital technology: Qualitative analysis. JMIR Ment. Health 2018, 5, e10091. [Google Scholar] [CrossRef]
  11. Hidalgo-Mazzei, D.; Mateu, A.; Reinares, M.; Murru, A.; del Mar Bonnín, C.; Varo, C.; Valentí, M.; Undurraga, J.; Strejilevich, S.; Sánchez-Moreno, J. Psychoeducation in bipolar disorder with a SIMPLe smartphone application: Feasibility, acceptability and satisfaction. J. Affect. Disord. 2016, 200, 58–66. [Google Scholar] [CrossRef] [PubMed]
  12. Beck, A.K.; Kelly, P.J.; Deane, F.P.; Baker, A.L.; Hides, L.; Manning, V.; Shakeshaft, A.; Neale, J.; Kelly, J.F.; Gray, R.M. Developing a mHealth Routine Outcome Monitoring and Feedback App (“SMART Track”) to Support Self-Management of Addictive Behaviours. Front. Psychiatry 2021, 12, 820. [Google Scholar] [CrossRef] [PubMed]
  13. Torous, J.; Nicholas, J.; Larsen, M.E.; Firth, J.; Christensen, H. Clinical review of user engagement with mental health smartphone apps: Evidence, theory and improvements. Evid.-Based Ment. Health 2018, 21, 116–119. [Google Scholar] [CrossRef] [PubMed]
  14. Basnet, S.; Tamminen, M.; Lahti, T. The feasibility of eHealth in mental health care. J. Addict. Res. Ther. 2014, 5, 205. [Google Scholar] [CrossRef]
  15. Lal, S. E-mental health: Promising advancements in policy, research, and practice. Healthc. Manag. Fórum 2019, 32, 56–62. [Google Scholar] [CrossRef]
  16. Van Gemert-Pijnen, J.E.; Kip, H.; Kelders, S.M.; Sanderman, R. Introducing ehealth. In eHealth Research, Theory and Development; Routledge: London, UK, 2018; pp. 3–26. [Google Scholar]
  17. Kip, H.; Kelders, S.M.; Bouman, Y.H.; van Gemert-Pijnen, L.J. The importance of systematically reporting and reflecting on eHealth development: Participatory development process of a virtual reality application for forensic mental health care. J. Med. Internet Res. 2019, 21, e12972. [Google Scholar] [CrossRef]
  18. Mulvale, A.; Miatello, A.; Hackett, C.; Mulvale, G. Applying experience-based co-design with vulnerable populations: Lessons from a systematic review of methods to involve patients, families and service providers in child and youth mental health service improvement. Patient Exp. J. 2016, 3, 117–129. [Google Scholar] [CrossRef]
  19. Thabrew, H.; Fleming, T.; Hetrick, S.; Merry, S. Co-design of eHealth interventions with children and young people. Front. Psychiatry 2018, 9, 481. [Google Scholar] [CrossRef]
  20. Van Gemert-Pijnen, J.E.; Nijland, N.; van Limburg, M.; Ossebaard, H.C.; Kelders, S.M.; Eysenbach, G.; Seydel, E.R. A holistic framework to improve the uptake and impact of eHealth technologies. J. Med. Internet Res. 2011, 13, e1672. [Google Scholar] [CrossRef]
  21. Mayworm, A.M.; Kelly, B.M.; Duong, M.T.; Lyon, A.R. Middle and high school student perspectives on digitally-delivered mental health assessments and measurement feedback systems. Adm. Policy Ment. Health Ment. Health Serv. Res. 2020, 47, 531–544. [Google Scholar] [CrossRef]
  22. Simonse, L.; Albayrak, A.; Starre, S. Patient journey method for integrated service design. Des. Health 2019, 3, 82–97. [Google Scholar] [CrossRef]
  23. Kelly, P.J.; Beck, A.K.; Deane, F.P.; Larance, B.; Baker, A.L.; Hides, L.; Manning, V.; Shakeshaft, A.; Neale, J.; Kelly, J.F.; et al. Feasibility of a Mobile Health App for Routine Outcome Monitoring and Feedback in SMART Recovery Mutual Support Groups: Stage 1 Mixed Methods Pilot Study. J. Med. Internet Res. 2021, 23, e25217. [Google Scholar] [CrossRef]
  24. Cooper, M.; Duncan, B.; Golden, S.; Toth, K. Systematic client feedback in therapy for children with psychological difficulties: Pilot cluster randomised controlled trial. Couns. Psychol. Q. 2021, 34, 21–36. [Google Scholar] [CrossRef]
  25. Oinas-Kukkonen, H.; Harjumaa, M. Persuasive Systems Design: Key Issues, Process Model, and System Features. Commun. Assoc. Inf. Syst. 2009, 24, 28. [Google Scholar] [CrossRef]
  26. Buitenweg, D.C.; van de Mheen, D.; Grund, J.-P.; van Oers, H.; van Nieuwenhuizen, C.H. Visual and personalized quality of life assessment app for people with severe mental health problems: Qualitative evaluation. JMIR Ment. Health 2020, 7, e19593. [Google Scholar] [CrossRef] [PubMed]
  27. Hetrick, S.E.; Robinson, J.; Burge, E.; Blandon, R.; Mobilio, B.; Rice, S.M.; Simmons, M.B.; Alvarez-Jimenez, M.; Goodrich, S.; Davey, C.G. Youth codesign of a mobile phone app to facilitate self-monitoring and management of mood symptoms in young people with major depression, suicidal ideation, and self-harm. JMIR Ment. Health 2018, 5, e9041. [Google Scholar] [CrossRef] [PubMed]
  28. Morton, E.; Hole, R.; Murray, G.; Buzwell, S.; Michalak, E. Experiences of a web-based quality of life self-monitoring tool for individuals with bipolar disorder: A qualitative exploration. JMIR Ment. Health 2019, 6, e16121. [Google Scholar] [CrossRef]
  29. Ospina-Pinillos, L.; Davenport, T.A.; Ricci, C.S.; Milton, A.C.; Scott, E.M.; Hickie, I.B. Developing a mental health eClinic to improve access to and quality of mental health care for young people: Using participatory design as research methodologies. J. Med. Internet Res. 2018, 20, e9716. [Google Scholar] [CrossRef]
  30. Westermann, G.M.; Verheij, F.; Winkens, B.; Verhulst, F.C.; Van Oort, F.V. Structured shared decision-making using dialogue and visualization: A randomized controlled trial. Patient Educ. Couns. 2013, 90, 74–81. [Google Scholar] [CrossRef]
  31. Borghouts, J.; Eikey, E.; Mark, G.; De Leon, C.; Schueller, S.M.; Schneider, M.; Stadnick, N.; Zheng, K.; Mukamel, D.; Sorkin, D.H. Barriers to and facilitators of user engagement with digital mental health interventions: Systematic review. J. Med. Internet Res. 2021, 23, e24387. [Google Scholar] [CrossRef]
  32. Radomski, A.D.; Wozney, L.; McGrath, P.; Huguet, A.; Hartling, L.; Dyson, M.P.; Bennett, K.J.; Newton, A.S. Potential reduction of symptoms with the use of persuasive systems design features in internet-based cognitive behavioral therapy programs for children and adolescents with anxiety: A realist synthesis. JMIR Ment. Health 2019, 6, e13807. [Google Scholar] [CrossRef] [PubMed]
  33. Wahle, F.; Bollhalder, L.; Kowatsch, T.; Fleisch, E. Toward the design of evidence-based mental health information systems for people with depression:A systematic literature review and meta-analysis. J. Med. Internet Res. 2017, 19, 325–356. [Google Scholar] [CrossRef] [PubMed]
  34. Rasing, S.; Stikkelbroek, Y.A.; Bodden, D.H. Is digital treatment the Holy Grail? Literature review on computerized and blended treatment for depressive disorders in youth. Int. J. Environ. Res. Public Health 2020, 17, 153. [Google Scholar] [CrossRef] [PubMed]
  35. Crutzen, R.; de Nooijer, J.; Brouwer, W.; Oenema, A.; Brug, J.; de Vries, N.K. Strategies to facilitate exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults: A systematic review. Health Educ. Behav. 2011, 38, 49–62. [Google Scholar] [CrossRef] [PubMed]
  36. Milward, J.; Drummond, C.; Fincham-Campbell, S.; Deluca, P. What makes online substance-use interventions engaging? A systematic review and narrative synthesis. Digit. Health 2018, 4, 2055207617743354. [Google Scholar] [CrossRef]
  37. Berry, K.; Salter, A.; Morris, R.; James, S.; Bucci, S. Assessing therapeutic alliance in the context of mHealth interventions for mental health problems: Development of the mobile Agnew relationship measure (mARM) questionnaire. J. Med. Internet Res. 2018, 20, e8252. [Google Scholar] [CrossRef]
  38. Maathuis, I.; Niezen, M.; Buitenweg, D.C.; Bongers, I.L.; van Nieuwenhuizen, C.h. Exploring human values in the design of a web-based QoL-Instrument for people with mental health problems: A value sensitive design approach. Sci. Eng. Ethics 2020, 26, 871–898. [Google Scholar] [CrossRef]
  39. Van Sonsbeek, M.A.; Hutschemaekers, G.J.; Veerman, J.W.; Vermulst, A.; Kleinjan, M.; Tiemens, B.G. Challenges in investigating the effective components of feedback from routine outcome monitoring (ROM) in youth mental health care. Child Youth Care Forum 2021, 50, 307–332. [Google Scholar] [CrossRef]
  40. Davies, F.; Shepherd, H.L.; Beatty, L.; Clark, B.; Butow, P.; Shaw, J. Implementing web-based therapy in routine mental health care: Systematic review of health professionals’ perspectives. J. Med. Internet Res. 2020, 22, e17362. [Google Scholar] [CrossRef]
  41. Hennemann, S.; Beutel, M.E.; Zwerenz, R. Ready for eHealth? Health professionals’ acceptance and adoption of eHealth interventions in inpatient routine care. J. Health Commun. 2017, 22, 274–284. [Google Scholar] [CrossRef]
  42. Damschroder, L.J.; Aron, D.C.; Keith, R.E.; Kirsh, S.R.; Alexander, J.A.; Lowery, J.C. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement. Sci. 2009, 4, 50. [Google Scholar] [CrossRef] [PubMed]
  43. Proctor, E.; Silmere, H.; Raghavan, R.; Hovmand, P.; Aarons, G.; Bunger, A.; Griffey, R.; Hensley, M. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm. Policy Ment. Health Ment. Health Serv. Res. 2011, 38, 65–76. [Google Scholar] [CrossRef] [PubMed]
  44. Ellis, L.A.; Augustsson, H.; Grødahl, A.I.; Pomare, C.; Churruca, K.; Long, J.C.; Ludlow, K.; Zurynski, Y.A.; Braithwaite, J. Implementation of e-mental health for depression and anxiety: A critical scoping review. J. Community Psychol. 2020, 48, 904–920. [Google Scholar] [CrossRef] [PubMed]
Ijerph 19 10834 g001 550
Figure 1. Phases of the CeHRes Roadmap.
Figure 1. Phases of the CeHRes Roadmap.
Ijerph 19 10834 g001
Ijerph 19 10834 g002 550
Figure 2. Stickers used by the expert panel.
Figure 2. Stickers used by the expert panel.
Ijerph 19 10834 g002
Ijerph 19 10834 g003 550
Figure 3. Youngsters indicated that a well-organized application was essential.
Figure 3. Youngsters indicated that a well-organized application was essential.
Ijerph 19 10834 g003
Ijerph 19 10834 g004 550
Figure 4. Use of traffic light colours in feedback.
Figure 4. Use of traffic light colours in feedback.
Ijerph 19 10834 g004
Ijerph 19 10834 g005 550
Figure 5. Use of avatars in INK.
Figure 5. Use of avatars in INK.
Ijerph 19 10834 g005
Ijerph 19 10834 g006 550
Figure 6. Bright and cheerful avatars on the specific behaviour page.
Figure 6. Bright and cheerful avatars on the specific behaviour page.
Ijerph 19 10834 g006
Ijerph 19 10834 g007 550
Figure 7. Notification ‘I want to discuss these results with others’.
Figure 7. Notification ‘I want to discuss these results with others’.
Ijerph 19 10834 g007
Ijerph 19 10834 g008 550
Figure 8. Privacy and Safety: Accounts are only accessible by youngsters.
Figure 8. Privacy and Safety: Accounts are only accessible by youngsters.
Ijerph 19 10834 g008
Ijerph 19 10834 g009 550
Figure 9. Visualisation questionnaires.
Figure 9. Visualisation questionnaires.
Ijerph 19 10834 g009
Ijerph 19 10834 g010a 550Ijerph 19 10834 g010b 550
Figure 10. Final version of the INK tool.
Figure 10. Final version of the INK tool.
Ijerph 19 10834 g010aIjerph 19 10834 g010b
Table
Table 1. Tools and techniques used during the development of INK.
Table 1. Tools and techniques used during the development of INK.
CeHRes PhasesGoalsActivities
Contextual InquiryIdentify stakeholdersCompose expert panel
Investigate the strong and weak pointsPatient journey
Value SpecificationIdentify values of stakeholdersBrainstorm
Prioritize valuesUse stickers and smileys
Technology requirementsSticky notes and votes
DesignIntroduction of questionnaires Sketches by youngsters
Low-fidelity prototypesUsability test
High-fidelity prototypesStructured interview
Table
Table 2. Overview of preferences of the youngsters for the design of INK-Value Specification.
Table 2. Overview of preferences of the youngsters for the design of INK-Value Specification.
ValueNeedsTranslation of Needs in INK
Being well-informedTrack feedback over timeThe INK application enables users to keep track of their treatment feedback over time.
Being well-informedMulti-informant feedbackThe INK application allows users to select and compare treatment feedback from multiple informants.
Being well-informedComprehensive and well-organized feedbackTake in everything at a single glance.
The INK application shows all feedback related to one (subscale of a) questionnaire at a glance.
Being well-informedGeneral and specific feedbackThe INK application provides both general (normal, subclinical and clinical) and specific scores (0–100).
Being well-informedGeneral feedbackThe colours green, orange and red are used modestly to represent normal, subclinical and clinical scores, respectively.
Being well-informedSpecific feedbackThe INK application provides insight into the treatment progress within a specific score on a subscale of a questionnaire (0–100).
Being well-informedInformation about (subscales of) questionnairesThe INK application informs users about the meaning of (subscales of) questionnaires in easy-to-understand language. Mentioning positive and negative examples.
Personalized user experienceProfile page *Profile page, including the possibility to upload a picture and personal information such as name, age and personal interests.
Personalized user experienceTailoringYoungsters expressed different ways to adapt the application to youngsters’ characteristics. For example: showing different icons based on age, as well as allowing users to ‘build’ their own character and upload their own picture.
Personalized user experienceAvatarThe avatar in INK shows personality and facial expressions.
Personalized user experienceContent tailoringThe avatar in INK shows different emotions based on the topic of the questionnaire and the score.
Personalized user experienceAge-appropriateThe design and user experience of the INK application should be age-appropriate (12–18 years).
Personalized user experienceMinimalistic designThe INK application has a minimalistic, simple design, which is colourful but not too vibrant in order to maintain a simplistic design. Users are able to select their colour scheme of preference in the INK application: pastel shades or bright, cheerful colours.
User friendlyCheerful designThe INK application has a cheerful, non-depressing appearance. Predominant use of the colour red and avoidance of sad facial expressions.
User friendlyAge-appropriateNames and descriptions of questionnaires are adapted to the reading level of youngsters.
User friendlyPositive affirmationThe INK application rewards end-users for filling out the questionnaires by means of positive affirmations (e.g., ’Thanks for filling out the questionnaire’ or ‘Good job!’)
User friendlyEasy to useThe INK application is easy to use.
User friendlyEasy to navigate/speed of useUsers are able to reach the desired questionnaire results in only a few clicks.
User friendlyWell-functioningThe INK application can be used on a mobile phone, tablet or computer.
Being in controlExpress agreement *A desirable feature is to report whether you agree with the reported questionnaire results. For example, when youngsters observe that their parents, teachers and/or clinicians hold different opinions about the youngsters’ behavioural and emotional problems, it is nice to discuss these different opinions.
Being in controlNotification to other informantsAnother desirable feature is to mark specific questionnaire results you would like to discuss with others, such as one’s relatives or therapist.
Being in controlList of marked questions *‘Marked’ questions should be saved on a list, so they are easy to retrieve. This list can serve as a conversation aid in one’s therapy session.
Being in controlReminder *Reminder when a new questionnaire is available to fill out and/or when there are new results available.
Being in controlChat *The INK application should have a chat function to ask feedback-related questions to your therapist and/or (anonymous) contact with peers to share feelings.
Being in controlCalendar *The INK application should have a calendar displaying all (treatment-related) appointments and tasks.
Being in controlTutorialThe INK application shows a pop-up with short instructions on first use (tutorial).
Being in controlHomepageAttractive homepage and an indication of the application’s main features.
Being in controlSelf-managementThe youngsters decide who can access their questionnaire results.
Being in controlData accessOnly youngsters have access to INK.
Privacy and safetySafe storage of dataData should be handled confidentially and stored in a safe manner.
Privacy and safetyAdvertisementsThe INK application is free of (personalized) advertisements.
*: Needs are not added to the final version of INK.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Bongers, I.L.; Buitenweg, D.C.; van Kuijk, R.E.F.M.; van Nieuwenhuizen, C. I Need to Know: Using the CeHRes Roadmap to Develop a Treatment Feedback Tool for Youngsters with Mental Health Problems. Int. J. Environ. Res. Public Health 2022, 19, 10834. https://doi.org/10.3390/ijerph191710834

AMA Style

Bongers IL, Buitenweg DC, van Kuijk REFM, van Nieuwenhuizen C. I Need to Know: Using the CeHRes Roadmap to Develop a Treatment Feedback Tool for Youngsters with Mental Health Problems. International Journal of Environmental Research and Public Health. 2022; 19(17):10834. https://doi.org/10.3390/ijerph191710834

Chicago/Turabian Style

Bongers, Ilja L., David C. Buitenweg, Romy E. F. M. van Kuijk, and Chijs van Nieuwenhuizen. 2022. "I Need to Know: Using the CeHRes Roadmap to Develop a Treatment Feedback Tool for Youngsters with Mental Health Problems" International Journal of Environmental Research and Public Health 19, no. 17: 10834. https://doi.org/10.3390/ijerph191710834

APA Style

Bongers, I. L., Buitenweg, D. C., van Kuijk, R. E. F. M., & van Nieuwenhuizen, C. (2022). I Need to Know: Using the CeHRes Roadmap to Develop a Treatment Feedback Tool for Youngsters with Mental Health Problems. International Journal of Environmental Research and Public Health, 19(17), 10834. https://doi.org/10.3390/ijerph191710834

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop