Outcome Quality of Inpatient and Day-Clinic Treatment in Child and Adolescent Psychiatry—A Naturalistic Study

Background: Child and adolescent psychiatry has only recently been established as a separate specialty and is practiced in different settings. The epidemiology of psychological problems in childhood is high and varied, thus qualitative work is essential. Assessment of outcome as part of quality management is central to assure the service of psychiatric care to be effective. Method: Over a three-year period consecutively admitted patients from inpatient and day-clinic treatment were prospectively evaluated. A total of 200 from 442 patients (m = 80, f = 120; age 15.1 ± 2.8 y) agreed to participate. Patients, caregivers, and therapists answered a range of questionnaires to provide a multi-personnel rating. Questionnaires used for outcome assessment were Child Behavior Checklist (CBCL) and Youth-Self-Report (YSR) (at admission, discharge, and 6 weeks after discharge) and the problem score of the Inventory of Quality of Life for children (ILK), treatment satisfaction, and process quality by the Questionnaire for Treatment Satisfaction (FBB, at discharge) and as real-life outcome control assessment of quality of life (ILK) was added (admission, discharge, and 6 wks after discharge). Results: There was a significant reduction in psychopathologicalsymptoms (CBCL, YSR) and in the problem score. Furthermore, there was a significant increase in quality of life. QoL score and YSR/CBCL scores returned to normal levels. Treatment satisfaction was high and so was satisfaction with process quality. Factors significantly influencing outcome were severity of disease and the relationship to the therapist. No differences were found for gender and setting. Conclusion: The quality management analysis revealed significant improvements of symptom load, a significant increase in QoL and a high treatment satisfaction. Furthermore, process quality was scored highly by parents and therapists.


Introduction
In Austria, child and adolescent psychiatry (CAP) has only recently become a separate medical specialty by Austrian federal law. Up to 2007, when law was passed [1], it used to be an additive special medical education, only accessible for pediatricians, neurologists, and psychiatrists. The prevalence of psychological impairment among children and adolescents affects over 13.0% of the age group worldwide, 17.0% in Germany, and up to 35.0% in Austria [2]. Both late creation of the specialty and the epidemiology necessitate an increased need for intervention and prevention [3]. Diagnostic and therapeutic services are provided in private and public practice, day-clinics, outpatient clinics, and hospitals of different levels. During the last ten years, CAP services in Austria have grown significantly, but not sufficiently to meet the need for treatment [4].
Thus, the topic of quality management in the treatment of those affected is of particular importance. Quality management generally refers to-as defined by Austrian federal child and adolescent psychiatry hospital under naturalistic conditions. This work was used as a basis for the present evaluation project's structure and intention. Although Remschmidt and Mattejat's study is one of the pioneering works in Germany, there are only two Austrian studies on inpatient treatment outcomes in Austrian CAP departments [18,19].
The primary focus of assessment of outcome quality lies on the success of treatment. The comparison of a pre-post measurement of the extent of symptoms provides the central measure for evaluating treatment success. Evaluative studies in child and adolescent psychiatry consistently show a positive change in symptoms regardless of the disorder [19][20][21][22][23] as well as in a disorder-specific context [16]. The quality of the relationship between the patients and the therapist contributes significantly to the success of the treatment [24].
In addition to this primary parameter, recent research has been increasingly focused on the analysis of additional factors affecting treatment conditions in evaluations [16,20]. Closely associated with the success of treatment is treatment satisfaction within all participants in the treatment process. In the literature, the two parameters are regarded as the same construct (i.e., treatment success is determined by the treatment satisfaction; [22]) and as parallel constructs that correlate positively with each other [16,21]. In general, at least a moderate treatment satisfaction is achieved at the end of a successful treatment [17,25].
Treatment success and treatment satisfaction do not predict whether the children are going to do well in real life. It is important to look at the clinical significance of the results and, thus, the aspect of quality of life of patients has recently gained importance in medical evaluation research [26]. This is crucial, since, simply considering the change in clinical symptoms does not suffice to make valid statements on the improvement of function for the patients [27]. The additional recording of quality of life as a separate construct can, therefore, provide valuable additional information [28]. Reduced quality of life turns out to be a systematic feature of mentally impaired children and adolescents [29,30]. Consequently, this is increasingly regarded as a recommendation for therapeutic practice [31,32].
It has been shown that successful treatment is accompanied by an increase in the quality of life [16,28,33,34], as well as an increased accordance between the perspectives of all participants in the treatment process. [18,19,30]. Different perspectives can also provide exclusive information about the quality of life of patients [35].
Thus, for evaluating the results of the organisational project and the therapeutic concept a naturalistic study was designed, investigating treatment success, treatment satisfaction, and quality of life. It was assumed that the therapeutic concept provides significant reduction of symptoms with clinical relevance, accompanied by high treatment satisfaction and significant improvement of the quality of life.
In addition, the influences of gender and age on treatment success and treatment satisfaction were examined as various framework conditions of treatment: these included differences in the form of stay and the influence on the therapeutic relationship.

Materials and Methods
The study period lasted from April 2011 to January 2014 and was approved by the Salzburg Ethics Commission under E-1195 (28 April 2010). Within this period, at the time of admission, all patients were asked if they were willing to participate in the study. In the event of consent, participants and custodial providers were asked to sign a written declaration of consent (EVE). In the event of rejection, there were no disadvantages for the patients at any time during treatment. The sample also included the primary caregivers of the children and adolescents, as well as the treating physicians, psychologists or psychotherapists, educators, and social workers. Patients under the age of six years and patients displaying acute suicidality, psychosis, or cognitive impairment were excluded from the study. In the case of insufficient knowledge of German, interpreters were consulted. In the primary data collection, a total of four measurement time points were chosen: admission to the clinic (T1), discharge (T2), catamnesis six weeks after discharge (T3) and another catamnestic survey 18 months after discharge from the clinic (T4). The survey took place in a specially provided room, accompanied by study assistants. In the present work the measurement at timepoint T4 is not considered.

Measures
The data were collected using quantitative questionnaires. Treatment success, treatment satisfaction, and quality of life were measured multi-perspectively with different instruments at three time points: admission, discharge, and 6 weeks after discharge. Questionnaires were filled out digitally; patients were accompanied by a psychologist.

Instruments
Treatment success-as defined by a significant reduction of symptoms i.e., psychopathology, between T1, T2, and T3-was measured by Youth Self-Report (YSR) and Child Behavior Checklist (CBCL), using the Total Problem Scale, Internalizing and Externalizing scale [36].
Treatment satisfaction was measured by the Questionnaire for Treatment Satisfaction (FBB; Mattejat & Remschmidt, [37]), providing rating of treatment satisfaction by patients, parents/caregivers, and therapists. Statements are rated using a 5-point Likert scale, ranging from 0 (don't agree at all) to 4 (agree completely). This questionnaire also can be used as a quality assessment instrument, dividing the results into outcome (items 1,3,18,20 and 6) and process quality (all other items). Subscores for caregivers and patients were calculated for outcome quality: personal success and family success and for process quality: relationship to therapist and framework conditions. The latter were only rated by caregivers, as suggested by the manual. Internal consistency (Cronbach's alpha) is reported to be 0.88.
Quality of life (QoL) was measured by the Inventory for quality of life (ILK; Mattejat & Remschmidt, [38]) in children and adolescents and parents/caregivers. For analysis, the total score for quality of life and the problem score, assessing load of the disease and the treatment, were employed. Internal consistency (Cronbach s alpha) for the total score is reported to be 0.55-0.76. Normative data are provided for healthy and mentally ill children and adolescents.

Statistical Analysis
Both correlation and difference hypotheses were formulated and evaluated with the program IBM SPSS Statistics 24 (IBM, Armonk, NY, USA) and R (version 4.0.1, R Core Team, Vienna, Austria) for Windows. If the requirements for the use of parametric methods were not met, their non-parametric equivalents were used. For the difference hypotheses in independent design, the Mann-Whitney U test was used for the comparison of two groups and the Kruskal-Wallis test for the comparison of more groups. Pairwise comparisons were performed in the Kruskal-Wallis test using the non-parametric post hoc test according to Dunn (Bonferroni correction). For the difference hypotheses in the dependent design, the t-test for dependent samples or the Wilcoxon signed-rank test was chosen for two samples, as well as the Friedman test for more than two groups. The correlation hypotheses were analyzed using spearman's rank correlation coefficient. For statistical description, absolute and relative frequency data, mean values, and standard deviations were used. If possible, 95% confidence intervals and effect sizes were specified. For the description of the effect sizes, the measure of Cohen's d was chosen for the mean value differences, and the correlation coefficient r was employed to evaluate the differences between medians and in the correlation calculations. The hypothesis tests were subject to two-sided calculations with a significance level p of p = 0.05 (*), p = 0.01 (**) and p = 0.001 (***). For the analysis of the hypotheses, in the case of missing data, the list-by-list case exclusion was chosen.

Sample Characteristics
The analyses included data from 442 patients treated during the aforementioned study period. Of this total, 328 were hospitalized once and 114 multiple times throughout the study period. Only the first hospitalization was included in the calculations. The consent for participation was given by n = 200, 148 of which were admitted once and 52 of which were admitted several times. Figure 1 illustrates the sampling process graphically. The sample size varies depending on the questionnaire and time of testing. At test time T1, data from 170 patients (85.0%) were available, while 163 data sets (81.5%) were available for T2 and 158 data sets (79.0%) for T3.
(***). For the analysis of the hypotheses, in the case of missing data, the list-by-list case exclusion was chosen.
Patients with positive and negative EVE differed significantly regarding form of stay and form of admission (planned or unplanned). Patients with positive EVE were more likely to be admitted on a planned basis (U = 15,842, p < 0.001) and patients with negative EVE were more likely to be in crisis stay (U = 17,043, p <.001). Further descriptive information on the sample and a comparison of the characteristics of the subjects included as opposed to the not included are given in the Appendix A.

Results
Overview of all results is summarised in Table 1.

Treatment Satisfaction
On average patients showed positive treatment satisfaction (M = 2.94, SD = 0.63), a result that is significantly different from zero (t = 60.19, p < 0.001) and lies above the normative data (clinical sample mean 2.57 ± 1.31). There were no significant differences between treatment satisfaction of male and female patients (U = 3301.50, z = −0.12, p = 0.903). Treatment satisfaction also turned out to be equivalent in the different age groups (χ 2 = 4.34, p = 0.227). With regard to the form of hospitalization, there was no significant effect (H(15) = 5.64, p = 0.060, η 2 = 0.02, d = 0.39). A significant association was found between the severity of clinical symptoms at T1 and treatment satisfaction at T2. This effect applies equally to the age groups of children and adolescents, i.e., the higher the clinical symptoms were, the lower the treatment satisfaction of both children and adolescents was (children: rS = −0.56, 95% BCa CI [−0.78, −0.23], p < 0.01; Adolescents: rS = −0.27, 95% BCa CI [−0.42, −0.11], p < 0.01).
In relation to the quality assessment, the results of the FBB analysis at T2 are shown in Table 3, showing a high satisfaction of caregivers and patients concerning the different quality measures. Framework quality correlates significantly with outcome quality rated by patients (rS = 0.56, 95% BCa CI [0.43, 0.67], p < 0.001) and caregivers (rS = 0.25, 95% BCa CI [0.06, 0.43], p < 0.05.), i.e., the better the framework the higher the satisfaction with treatment.

Quality of Life
At T1 quality-of-life score was lower as the norm sample and significant improvements in quality of life were observed over the treatment period (see Table 2). At T3 the mean QoL score was above the norm population (19.25) for parents (19.47   In the sample of children admitted to the UK-KJP (n = 24), a significant improvement in quality of life was found as well. The quality-of-life score at T2 is reported significantly higher than at T1 (z = −2.96, p = 0.009, d = −1.52), although this increase in quality of life is no longer detectable at T3 (z = −1.80, p = 0.214, d = −0.79); a result which may occur due to the small sample size.

Discussion
This study measures outcome quality within a naturalistic design at a department for child and adolescent psychiatry. Results show a significant reduction of psychopathology from admission to discharge and until six weeks after discharge. Effect sizes are high during the time of treatment and smaller, yet still significant, after discharge with high effect sizes also for general psychopathology as well as internalizing and externalizing symptoms. This means, the therapeutic setting provided leads to a significant symptom reduction. Symptom reduction is the first and main goal of treatment and, thus, the main outcome criteria for quality assessment [39]. In comparison to the two other Austrian studies [18,19] our results are aligned with previous findings. One of the departments evaluated by the older Austrian studies is also grounded on systemic therapy theory, the second is grounded on psychoanalytical theory. All three studies are naturalistic studies without a control group, without randomization and crossover, thus, mirroring the reality of daily treatment (treatment as usual) all over the world. It would be interesting to analyze the similarities of those departments in treatment provision, structure, and processes, to gain causal information about what "really helps". One principle of evidence-based medicine is to reach informed consent or shared decision-making, which this treatment process provides on a high level [13]. The other reason to perform a naturalistic study was the reality of treatment provision to kids of different ages, sex, with various different diseases and at different stages of disease, criteria which no known EBM protocol could provide. The therapeutic concept used in this study was developed to create a basic treatment situation, which is the grounding basis for an individual's recovery.
In all three Austrian studies treatment is successful in the sense of symptom reduction and clinical relevance, thus, arguably reaching at least a partial recovery-a results which has been documented by comparable international studies as well [20][21][22][23]. Recovery is defined as a "profound personal . . . process to change attitudes, values, feelings, abilities, and roles to achieve a satisfactory, hopeful and productive way of life with the possible limitations of illness" [40]. Certainly, our study can only demonstrate recovery in the sense of symptom reduction and increased quality of life, but any recovery is important for lifelong prognosis of mental disorders. As a longer duration of mental health problems in adolescence is the main prognostic factor, it is essential to reduce symptom load and increase self-efficacy and quality of life [41]. Therefore, it is crucial to achieve a clinically relevant reduction. In our study, this is exemplified by reaching the normative symptom level of the CBCL scales and by the significant increase of the QoL score. Furthermore, the problem score of the QoL questionnaire also decreased significantly, providing even more information about the clinical relevance of the reported outcome quality.
From a quality management perspective treatment success-defined as satisfaction with treatment and with process-is the most important parameter. In our study, treatment satisfaction is high for caregivers, less but also very positive for patients and high for therapists. Looking at the subscores outcome quality is also in the high range and so are framework conditions and relationship to therapist. Both factors of process quality proved to be significantly associated with high satisfaction with treatment. In a recent qualitative survey Schneidtinger and Haslinger-Baumann [42] describe a model of recovery with 3 main facilitating factors. The basic level consists of supporting therapies, help with daily structures and the ward as a safe place. The next level is provided by peers understanding, community, and friendship, the third level by family factors as connectedness and parental support. As parents and patients rate the framework conditions satisfying the concept of department delivers the basic level of the recovery model. As parents are rating the improvement in family function equally positively as their children, effects on the third level can be assumed.
For future quality management in child and adolescent psychiatry there are several conclusions to draw. First, outcome measurement as performed in this study is useful and should be used in a continuous quality improvement concept [5,43] and performed as a measurement feedback system [44]. Reduction of symptoms and treatment satisfaction could be used as benchmarks, quality-of-life measures, and reduction of problem scores added for control of clinical relevance. However, as "recovery" would be the central goal of a profound quality management the quantitative assessment should be enriched by detailed analysis of, e.g., the severity and acuity of the child's symptoms; the child's functional impairment; the child's functional strengths; family functioning. Furthermore, assessment of the goals of treatment; the modality, strategy, and tactics of treatment; readiness for change; the quality of the therapeutic alliance and adherence to treatment should be added to the quantitative assessment of "recovery" [43].
Kelly [44] recommends multidimensional monitoring in three modules: a baseline follow-up module which assesses the child's and family's mental health status starting the entry to the system and lasting throughout treatment; a concurrent module which obtains information throughout the course of treatment; and a background module which assesses factors in the child's and family's background that moderates the course of treatment.
Furthermore, regular assessments of the employees' view of the structures and processes as well as their implementation-together with the measures described abovewould give a 360-degree view of the therapeutic process a team provides. The results should be regularly assessed and reported back to the providing team for control, reflection, and supervision.

Limitations
In the sense of the strict rules of evidence-based medicine with the necessity of double blinded, randomized crossover studies the naturalistic design is the main limitation of this study. Nevertheless, critics of this strict interpretation of evidence-based research suggest also using alternative methods for gaining evidence [15]. The naturalistic approach is thought to evaluate concepts in the real situation, closely monitoring the real processes of the applied treatment service.
Moreover, participation of patients and caregivers was not complete, thus this may have caused a bias, although we checked the anamnestic details and found no difference between the participants and the non-participants concerning age, gender and several psychosocial items (see also Table A1).

Conclusions
Our study demonstrates strong therapeutic effects of a stringently organised and reflected multiprofessional treatment approach. Systemic theory gave rise to the values, vision, and mission of the project; it provided the basis for developing the respective structures and processes. Symptom reduction and treatment satisfaction is shown to be high, effect sizes are strong, and the clinical relevance-as measured by quality of life at discharge-is comparable to healthy children and adolescents. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Data Availability Statement:
The data generated and analysed during the present study are not publicly available due to ethical restrictions but are available from the corresponding author upon reasonable request.

Acknowledgments:
We thank all the patients, their caregivers, and all our employees, who are responsible for the application of the treatment process and supported the study.

Conflicts of Interest:
The authors declare no conflict of interest. Furthermore, the funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.