The Interim Evaluation of Training Processes: The Case of the Second Level Professional Master’s Program in Family and Community Mediation

Abstract

This contribution illustrates the training evaluation system developed within the Master’s Program in Family and Community Mediation at Università Cattolica del Sacro Cuore in Milan. This is an interim evaluation which focuses on the training process and which considers the collaboration with the subjects of the training to be fundamental. The peculiarity of this work concerns the possibility of inserting research within the training process, following a logic of mutual enrichment both in terms of content and learning. The contribution illustrates in detail the outcome and the process evaluation system, defining the perspective, the objectives, and the methodology of implementation. In particular, the outcome evaluation focuses on the distal and proximal outcomes of the training, while the process analysis focuses on the dynamics within the group of participants. Preliminary findings from 33 participants highlighted that the majority of participants (19) have a “regular” profile, that is, they appreciate both the theoretical contents, as well as the practical activities proposed during the training program. As for the process, the findings showed the importance of reflexivity as a major factor of change. Although these findings are referred to the specific experience of a particular group, and so further evaluations involving different training groups and other training processes are needed, this training evaluation system sheds light on both the topic and the context in which training is delivered. The integration between different points of view and several levels of analysis allows the researchers to deepen the individual path of each participant as well as to obtain feedbacks on the progress of the training group as a whole and allows participants to better understand their contexts of work thanks to the use of reflexivity. This can guarantee a sustainable growth both at individual and interpersonal level.

1. Introduction. Defining Our Standpoint

What does it mean to assess training processes? More specifically, what does evaluating such processes in a professionalizing environment that involves a stable group of participants over a long period of time mean? These questions are the starting point of the research that will be illustrated hereafter. Specifically, our aim was to define our focus starting from a fundamental need: that of setting up an interim assessment of the training process by actively involving those who were attending that training. What we had in mind was a way of doing research that, by favoring involvement, would create a direct relationship with participants, in a process of reciprocity, exchange, and continuous information processing (in contrast with the “mere” data collection that is typical of other research traditions).

In this perspective, evaluation is seen as being aimed at understanding rather than judging, at seeking meaning through reflection rather than at achieving a simple match between goals and results, and at resorting to dynamism and processualism rather than adopting a static standpoint [1,2,3,4]. Within this framework, it is, therefore, important to value everyone’s subjectivity and unique contribution to gain a perspective capable of taking into account all the actors involved (multi-actor approach). Furthermore, this perspective aims at promoting the well-being of each actor by directly involving him/her as an active promoter of knowledge and change [5,6]. In so doing, we aim to promote sustainable growth while granting all the parties involved an active role in the building of knowledge and in the definition of goals both with regards to training and clinical intervention. Finally, undertaking a training evaluation also means pursuing sustainable growth, as it enables an optimization of the assets while at the same time reducing potential costs or wastes due to ineffective or unsuccessful training. Indeed, training evaluation was found to be associated with transfer of training in organizations, being related to improvements in training programs and creating greater accountability among stakeholders for training outcomes [7,8,9].

When analyzing our object of study, we are aware we are travelling a rather unexplored and still unclear path. Indeed, although there is a growing body of literature on training evaluation reporting the main current evaluation practices as well as training designs [10,11,12], to our knowledge, no studies have provided a systematic, interim assessment of training programs, especially when such programs involve postgraduate students (such as in the case of family mediators). This is also because evaluating the effectiveness of training is not easy, dealing with “objects” that are difficult to quantify and to measure and requiring a precise definition of educational goals and the controllability of educational results [13]. Therefore, our reflections are based upon the research tradition on the evaluation of clinical interventions [14,15,16,17]. In this perspective, it seems important to underline that the goal of training is that of producing significant change within the trainees’ expertise and knowledge; in our case, such change implied the development of a set of professional clinical and relational skills. For such reason, we decided to investigate our topic by embracing the longstanding and consolidated research tradition on the evaluation of clinical interventions. In this field, the research tradition on psychotherapies is very well-established [18,19,20,21,22,23,24]. Hence, while taking into account all the differences with the above-mentioned evaluation, we decided to base the structure of our research on Di Nuovo’s work [25]; in particular, following Di Nuovo’s suggestion, we asked ourselves how, when, what, and why should we evaluate. We think that the goal of evaluating the training process is important for improving the quality of training itself. It provides quality assurance, and it ensures that training programs uphold high quality standards. Moreover, it provides advanced training programs benchmarked against international standards [26,27].

Moreover, we deem it necessary to put theoretical models underlying any intervention to test by conducting a rigorous empirical evaluation capable of providing information on the robustness of the constructs involved as well as on their operationalization. In other words, we build on the assumption that clinical interventions and research are two sides of the same coin [28]. Such a strong connection between conducting research and providing an intervention allowed us to adopt a peculiar and specific stance in viewing our training within the second level Master’s program: Research and intervention need to be strictly connected rather than opposed to one another. Moreover, thanks to this standpoint, the research becomes useful as it provides an understanding on how interventions work while also allowing the involvement of a third perspective—distinct from the one adopted by those who operate on the field- and granting an ongoing meta-reflection on one’s work.

Starting from these premises, in this contribution we will present the training evaluation model developed by our research team with the main aim of evaluating the outcome and process of the Master’s program in family and community mediation. However, it should be noted that the intent is to develop a transversal evaluation system, that is, a system applicable, at least in its overall structure and aims, to other training contexts as well. Therefore, the purpose of this contribution is descriptive, as it aims at describing a useful method of training evaluation, and its nature is theoretical (rather than strictly empirical) [29,30].

In the next paragraphs, we will present this evaluation system and some preliminary findings of its application to one edition of the Master. Other research data will be presented in a following contribution.

2. The second Level Professional Master’s Program

In the Italian education system’s second level Master’s programs are postgraduate programs that can only be accessed after having gained a five-year bachelor degree and generally last two years. The second level professional Master’s program in family and community mediation issued by Università Cattolica has a longstanding tradition stemming from the weaving of historical events, cultural changes, and professional definition of family mediation as a specific practice [31,32,33]. Indeed, having properly qualified mediators is an increasing goal of quality, and it derives from a high level of training and internship. Italian Technical Standard UNI 11644:2016, published in August 2016, “Activities unregulated occupations—Family Mediator—Requirements for knowledge, skills and competence” sets quality standards for the training of professional mediators. There is, in fact, broad consensus on the impact of family mediation quality on its outcomes [34,35,36,37]. Back in 1994/1996, the first training courses in family mediation were conceived as specialization courses and envisaged family mediation as a skill to be acquired and integrated in one’s existing profession. For such reason, the first editions of the course were provided in collaboration with some local Institutions and Agencies. From 2003–2004 family mediation progressively started to be introduced in Italy as a tool to manage and resolve family conflicts—especially after the European norm nr. 98 (1) was issued in 1998— and therefore the University decided to institute a second level professional Master’s program recognized by both the Postgraduate School of Psychology “A. Gemelli” and the Athenaeum Center for Family Studies and Research. After over 20 years of experience, XI editions, and thanks to the great interest and support we have always met both from the local agencies as well as from the people interested in attending our type of training, we decided to study and analyze our Master’s program. Specifically, we decided to conduct an explorative study aimed at providing both a process and outcome evaluation. The evaluation of this Master’s program is particularly important considering that mediators work with and within conflicts with the main goal to manage such conflicts; the successful handling of conflicts and disputes is one of the key goals to achieve a sustainable growth that promotes and fosters the wellbeing of individuals and relationships as well as of the society as a whole. A successful conflict management is, in fact, a key prerequisite to social sustainability, which, in turn, favors psychological wellbeing and progress.

Before moving forward, it is important to point out that the choice of uniting family and community mediation in one course meets a precise set of theoretical and methodological standards defined at the onset of the Master’s program itself. Our way of working is firmly supported by the relational-symbolic model [38,39,40], whose theoretical principles can be applied in different contexts and are therefore valid both with families as well as with communities. This also entails several possible forms and types of interventions, such as divorce, intergenerational, criminal, health, neighborhood, and peer mediation.

Another distinctive trait of our way of working is that, during their training, participants are given the chance to put to the test the key passages of mediation thanks to a practical approach to learning that relies heavily upon exercises and group work. For such reason, the Master’s program is defined as “professional”. The logic underlying such choice postulates an analogy between the mediation and the training processes. In other words, participants experience some of the key passages and turning points of the mediation process within the microcosm of the classroom. They get to exercise, work, and generate new knowledge based on practical situations, similar to the ones they will face during an actual mediation meeting.

As an example, we report that the work participants are requested to carry out around the image of family or community during the first module of the training program. Specifically, trainees are asked to comment on their inner image of family or community starting from some paintings by famous artists. After having selected an image, participants reflect upon the reason of such choice, first in a plenary session and, subsequently, in smaller subgroups.

Participants work indirectly on several—at times highly emotionally charged—elements connected to their way of experiencing and representing their own relational world in order to gain a greater awareness that could benefit their work with separating couples or groups in conflict with one another. At the same time, this exercise allows participants to reflect on the fact that the subjects they are going to work with hold their own image of family and community and that they will need to connect to and understand it.

3. The Research Context

In this paragraph we will outline the context underlying our system of analysis and assessment. At the macro-level, reflexivity is an essential element to start the work process. In our perspective, this translates into reflecting upon the course contents, considering the emotions experienced during the classroom activities, and going over the group dynamics that participants establish with each other during the training (The whole training process is divided in 15 monthly modules (12 are theoretical and the remaining 3 are devoted to the revision of practice), each lasting 2 or 3 days.). These three elements need to be connected with the different professionals making up the classroom group. First, we are referring to the mediators in training—the participants to the Master’s program—but we need not forget about the classroom trainers, that is, those who accompany the participants through all the 15 modules. Finally, we should also consider the Master’s program staff members: While most of these people will not necessarily attend the course, they need to be provided with a comprehensive and detailed account of the process development, both in terms of contents as well as of group dynamics.

In the next paragraphs, we are going to provide a clearer definition of all these aspects and to clarify the reasons why this research project can be viewed as an enrichment to the whole training process. The specificity of our proposal is the chance, for the mediators in training, to allow themselves some time to stop and reflect upon their experience. More specifically, this means participants have the chance to focus their attention both on the contents provided, as well as on the ways they emotionally reacted to those contents and related to their colleagues in a group setting. Another specificity of our training is that it provides a space to meta-reflect, together with the trainees, on the research outcomes, thus allowing a recursive thinking process on key topics regarding mediation.

In light of the participatory perspective we chose to embrace, the research is envisaged as an integral part of the training process. In other words, research becomes an element that is capable of adding new meaning both to the classroom activities and to the practical experiences each of the participants makes outside of the classroom. Moreover, research helps the process of generating new ideas and reflections starting from both the individual experience and the exchange between the members of that specific group, each with their own skills and expertise.

Combining research outputs and the training activity also allowed staff members to benefit from the ongoing evaluation process: Goals and activities are subject to a constant reorganization and redefinition. Similarly to what happens in the mediation room, process management is one of the elements that needs continuous reflection and attention in order to test the robustness of our proposal throughout the participants’ feedback. In this view, participants are considered as key informants: Rather than being conceived as mere data research outputs-suppliers, they are called to share their thoughts on the training process with the staff members at regular times during specific moments of the Master’s program. Such moments, devoted to the circular exchange of ideas, involve: the trainees’ group (conceived as a subject on its own), the individual professionals, the classroom tutors, as well as the research staff. Exchanges are handled according to a logic of integration that promotes dialogue between different points of view. During these moments, participants are also invited to raise the topics they consider relevant based on the data collection. We firmly believe that the usefulness of this assessment lies in its immediate impact on the training. Borrowing a term from mediation, we can say that research in this context symbolically represents the involvement of a “third party” that—according to the relational-symbolic model [38,39,40]—is capable of infusing new energy to the process, enriching it with meaning.

4. Research Framework

The overall research framework that will be presented in this article is structured around the “classic” distinction between process and outcome research, typical of the studies evaluating clinical interventions, and is aimed at investigating both these levels. In particular, the outcome research focuses on two different levels: The first level compares different key moments of the training process, specifically, the beginning, the middle and the end; the second level evaluates the proximal (not only the distal) effects of the training through an assessment carried out at the end of each module. The choice of including two levels of analysis with regards to the outcome research is based upon the different focuses that each of the two levels have: the whole group of participants and the individuals, respectively. Moreover, these two levels of analysis have partially different objectives and are aimed to grasp the complexity of the training path by combining together different—and somewhat complementary—focuses, levels, and goals [41,42].

In general terms, outcome research is carried out by administering a set of self-report questionnaires to both the participants and the classroom trainers; process research, instead, is based on the analysis of the videorecording of some key moments of the training. Such moments are chosen in light of the theoretical presuppositions underlying the Master’s program structure. In the following paragraphs, we will provide a detailed description of the different levels of analysis.

4.1. The outcome Research

As mentioned above, the research on outcomes includes two different levels of analysis: one that focuses on the distal effects of training and that implies a comparison between different time points (i.e., beginning, middle, and end) in the process and another that focuses on proximal effects and therefore requires a module-by-module assessment throughout the whole training. Both these levels include the administration of an ad hoc self-report questionnaire to all participants.

4.1.1. Distal Outcomes

The assessment of distal outcomes has three main goals: (1) to explore the participants’ representations, motivations, and expectations towards the Master’s program (especially with regards to the expectations and motivation), as well as towards mediation itself (especially with regards to the representations); (2) to analyze the trainees’ knowledge and competencies/skills with regards to mediation and to their role as (prospective) mediators; and (3) to evaluate the participants’ perception of some specific aspects of the training process, such as the most meaningful or critical moments, and so on. To this end, we developed an ad hoc self-report questionnaire, consisting of two main parts, one common to all the three assessment phases and another specific to each phase; subsequently, participants were administered the questionnaire before the beginning of the first module, at the end of the intermediate module, and at the end of the last module of the program. Specifically, the common part mainly pertains the evaluation of the representations, knowledge, and skills regarding mediation, and it is aimed at both analyzing such constructs in the different phases of the process as well as determining whether and how they change over time thanks to the learning process.

We are aware that the main goal of a Master’s program should be that of favoring the development of both theoretical knowledge and practical skills; in this perspective it is extremely important to understand the group’s background as well as how it changes over time. In order to explore the participants’ representations regarding mediation and to evaluate their competencies and skills both at the beginning and during the training, they were asked a set of open- and closed-ended questions. Closed-ended questions were mainly theoretical (e.g., “Can a judge order parties to go to family mediation?”, “Is community mediation an instrument for social control?”) and had three possible answers: yes, no, or not sure. Open-ended questions, on their side, were more evocative (e.g., “Could you write the first five words that come to your mind when you think if family/community mediation?”) (Questions regarding mediation always concerned both family and community mediation. It seems important to remind readers that this program aimed to train participants on both types of mediation.).

Non-verbal cues were presented together with verbal ones, starting from the assumption that non-verbal stimuli might elicit different thoughts and feelings and therefore provide a more detailed and rich vision of the emerging representations [43]. In this respect, the Staff members selected a few images capable of eliciting different representations regarding the training, and participants were asked to choose the image that most “called to them” and best expressed their idea of this specific training program. This last question has been asked at all time points in order to analyze both each answer per se and to compare answers given at different times during the training path.

With regards to the questions specific to each phase, the initial questionnaire mainly focused on investigating the participants’ expectations and motivation (e.g., “Why did you decide to enroll in this program?”), the intermediate questionnaire aimed to review the past training in order to plan and redefine future goals (e.g., “Could you list three further skills you would like to acquire during this course?”), whereas, in the final questionnaire, participants were asked to provide their impression on the whole program (e.g., “What in the class helped your learning the most?”). As it happened for the previous (common) part of the questionnaire, closed-ended questions were accompanied with open-ended questions.

With regards to this first level of analysis, data have been aggregated, that is, the whole group of participants has been considered as unit of analysis. Focusing on the group as a whole rather than on individual participants allowed us to capture the group’s viewpoint throughout all the stages of the training process. Specifically, this analysis aimed to describe the main and most relevant characteristics of the group and to monitor its changes over time (see

Table 1. Distal outcomes.

Three Assessment	The Beginning	The Middle Phase	The End
Common questions	Indicators
	Knowledge competences/skills representations images
	Indicators
Specific questions	motivation	competences in progress	learning
focus on the whole group of participants

).

The results of this first-level analysis were presented to the classroom during the various modules. Specifically, participants were delivered three presentations of the research outcomes: one during the first module, one during the intermediate module, and one during the final module. More specifically, researchers presented the main results by focusing on the most characteristic traits of the group, those that were common to all participants. This participatory approach made it possible to start an exchange between the participants themselves, the trainers, and the research staff and also allowed a better and more comprehensive understanding of the results.

4.1.2. Proximal Outcomes

The second level of analysis focused on outcomes has two main goals: (1) assess each participant’s notion of the different modules both with respect to the contents as well as to the activities scheduled during the training; (2) explore each participant’s position both with respect to the process and to the relationship with the other trainees and with the group as a whole. In other words, this level of analysis aims to accurately and continuously (at the end of each module of the Master’s program) examine the participants’ and the trainers’ perceptions regarding the course contents (topics, ways, and times of learning) as well as their personal stance towards the “learning objects”. To reach these goals, another ad hoc self-report questionnaire was developed and administered to participants at the end of each module. Specifically, such questionnaire included both multiple choice, closed-ended questions as well as open-ended questions that allowed participants to freely express their thoughts and opinions (see

Table 2. Proximal outcomes.

15 Assessment-Module by Module
Module evaluation		Process evaluation
Indicators	-contents -activities (small group; plenary) -trainers	Indicators	-group atmosphere -personal stance within the group
Relevance of training
personal		Professional
focus on individuals

).

With respect to the first type of questions (i.e., multiple choice, closed-ended), the questionnaire is composed of three blocks: The first block refers to the module’s evaluation (e.g., “Do you think that this module’s contents were useful and interesting?”; “Do you believe trainers have been effective in managing sub-groups activities?”), the second block pertains the evaluation of the process (e.g., “To what extent did you feel free to express your thoughts and/or feelings during this module?”; “Did you find it hard to work in small groups during this module?”), whereas the third block focuses on the overall worth appraisal of the module with respect to its personal and professional relevance (“To what extent do you consider this module as relevant to your professional development?”; “To what extent do you consider this module as relevant to your personal development?”). The questionnaire used a 4-point Likert scale (Not at all, Very little, Somewhat, To a great extent) for the first two blocks, while the last block used a 10-point scale (where 0 represented no relevance at all and 10 represented the maximum relevance).

The choice of splitting the last question in two and of investigating the relevance of the modules’ contents on both the personal and professional levels are led by the same logic underlying the first two blocks of questions. Specifically, we wanted to investigate both the participants’ subjective perception and their stance and position towards the training (see Table 2). In this latter case, the focus is placed on individual participants with the aim of pinpointing potential critical aspects in each of the modules and outlining an “individual profile” to monitor the participant’s progresses over time. In particular, a set of seven prototypical profiles, that is, some profiles that are indicative of a specific way of functioning within the group and over time, have been outlined. The first, which we named “regular”, includes all participants that, although facing some minor challenges, do not make any particular effort either at a personal level or with regards the course contents and teaching modalities; the second profile, named “group-related struggles”, includes those participants that report constant struggles either with respect to their evaluation of their way of experiencing the program or the relationship with other members of the working group (fellow participants and/or trainers); the third profile comprises those participants that refer a “worsening” over time; on the contrary, the fourth profile includes participants that show an “improvement” in their evaluation of single modules or of the training over time; the fifth group shows a “fluctuating” trend and includes those participants that alternate modules free of significant challenges to others filled with difficulties and downfalls (beyond the greatest success of some modules compared to others according to the participants’ overall evaluation); a further profile, called “constantly critical”, comprises all participants that experience significant challenges and difficulties with respect to the module and/or the process throughout all the modules; finally, one last profile includes those participants that, on the contrary, provide positive evaluations throughout all the modules without reporting any trouble or complication (“too good”).

Participants’ profiles, outlined by analyzing both the evaluations of single modules as well by comparing modules between them, are made available to both the classroom tutors and the participants themselves. In this view, profiles become cues for reflecting on the training process, the resources that can be leveraged and the difficulties that need to be faced before becoming chronic. In other words, such profiles can be used throughout the training path and dealt with during the supervision meetings scheduled between individual participants and tutors; at the end of their training, participants are generally provided their own folder (made up of all the reports drafted for each module) as memory and testimony of their training path. While this level of analysis is specifically focused on individual participants with the aim of describing their personal experience of training, outcomes of the analysis of each module also allow an evaluation of the module itself and of the evolution of the various modules overall. In other words, the analysis of the accounts of individual participants with regards to each module allow to pinpoint both the difficulties ascribed to personal struggles as well as the faults connected to the module itself.

If, on the whole, a module receives mostly positive evaluations and only a few participants recount significant struggles, it is likely that such struggle is to be ascribed to personal factors; on the contrary, if many participants report serious difficulties with regards to one module, we need to assume that that module is problematic per se (e.g., there are issues with the contents or the activities or the timing is not adequate).

Finally, it seems important to underline that these two levels (i.e., proximal and distal outcomes) of outcome evaluation can be integrated, and the results can complement each other. Specifically, we refer to the fact that the conjoint analysis of proximal and distal outcomes permits a deeper and more articulate reflection, for example, by allowing the connection between participants’ expectations and their evaluation of single modules both in terms of content and activities as well as with regards to personal and professional relevance.

5. The Process Research

As mentioned above, along with the outcome evaluation, we also carried out a process evaluation aimed at determining not only whether a change occurred but also how it occurred; in order words, process evaluation focuses on the analysis of the mechanisms that enable transformation. While outcome research analyzes—in this case, by means of qualitative methods—the key moments of the training process while also allowing an understanding of the individual trajectories, process research provides a deeper understanding of the dynamics characterizing each module. In this latter case, the final goal is that of studying key moments in the training program and determining the factors that can elicit change both within the individual participants as well as within the group as a whole.

More specifically, process research focuses on the group and the exchanges that occur between participants and aims at identifying the key elements underlying transformative processes. To this end, tape recordings are analyzed along with some specific moments that, according to the theoretical principles underlying the training model, are considered to be particularly important. The choice of such units of analysis is made according to both a top-down as well as a bottom-up logic. Specifically, units of analysis are chosen based on the relevance of the contents and activities and of their connection to the training goals and the skills required in each module, as well as on grounds of available data.

From a methodological viewpoint, the process evaluation consisted of the application of an observation grid developed ad hoc and composed of some indicators which, in turn, are based upon the reference theory leading our idea of training and, specifically, of a training in mediation. Starting from the above-mentioned framework and from the analogy between the training and the mediation process, we identified three main indicators; each indicator is split into other sub-indicators according to a tree-like logic.

The first indicator focuses on reflexivity, that is, the likelihood for participants’ exchanges to promote and favor meaning seeking within the group. With respect to this indicator, along with the presence versus absence of reflexivity (in this latter case, participants’ exchange do not bring and added value to the discussion), it is also important to detect who is promoting reflexivity (i.e., the trainers versus the participants), in what ways (e.g., by posing questions, reformulating, accounting), and finally, on what kind of content (i.e., a specific topic versus the relational dynamics).

The second indicator has to do with difference, that is, the presence of different point of views or stances among the participants; such difference might not be handled (for example, it is ignored, denied, or inadequately managed) and therefore lead to disagreements that may, in turn, generate conflicts; on the contrary, difference can be managed in a functional way by developing a shared narration that is capable of integrating the different points of view and/or of generating a third, joint perspective. In this latter case, initial differences are not denied but rather embraced and integrated allowing, also thanks to the participation of all group members, to develop a shared meaning around the topic of discussion (a specific issue or the relational dynamics)

The last indicator pertains emotionality, that is the explicit reference to their emotional state by the participants and/or the trainers; emotions can be positively or negatively connoted, they can be embraced (by the participants and the trainers) or denied (e.g., downplayed or mocked); lastly, similarly to what happened for reflexivity, it is important to understand how such emotionality is expressed and later embraced or denied (e.g., by posing questions, making comments, etc.) and what it is about (i.e., a specific topic or the relational dynamics).

The analysis of some video-recorded fragments is based upon the description of the group interactions, starting from the above-mentioned indicators and counting the frequencies of the indicators themselves in order to assess the key elements of the training process. Specifically, the count of the frequency of each indicator and the analysis of the modalities in which such indicators are expressed allows the recognition of particularly significant moments within each module; moreover, such analysis also allows us to identify the activities that might be useful and functional in eliciting some specific aspects of the individual or group functioning. The outlining of these aspects is key as it favors the achievement of the goals set in the Master’s program.

The results of this process analysis are presented at staff meetings and shared with the classroom tutors in order to promote a reflection upon the training process and, specifically, the dynamics occurring among the participants and between the participants and the tutors. The final goal is to outline the characteristics of both transformative as well as standstill moments. We firmly believe that, similarly to therapy, training processes are not linear and cannot be viewed as a progressive accumulation of knowledge and skills but rather need to be conceived as winding roads filled with turning points—that lead to individual and group improvement—and hindrances that reveal the struggles of both the individuals and the group alike.

Identifying such moments and trying to make some hypotheses on their underlying mechanisms is therefore a fundamental step in evaluating a training program. Moreover, such results can be shared with the participants’ group: Watching the video-recordings of some of the exchanges in which they were directly involved helps meta-reflection while promoting active participation. In summary, as shown above, the three different levels of analysis (i.e., distal outcome, proximal outcome, and process research) have partially different goals and therefore required specific and diversified methods. It seems important to underline that, beyond their specificities, the outcomes coming from these levels of analysis need to be integrated in order to promote a reflection among the staff members on the meaning and direction of the training process and to rethink the planning of the Master’s program itself.

In the next paragraph, findings showing the application of this model of evaluation to the XI edition of the Master’s program are reported as an example.

6. Preliminary Findings

This edition of the Master’s program involved 33 participants: Most were lawyers (30%) and psychologists (16.7%), while the remaining had a degree in educational science/pedagogy (16.7%). Over half of the participants (51.6%) stated they had chosen this course due to its peculiar theoretical and practical approach.

With regards to the words associated with “family mediation” and “community mediation”, reflecting the subjective representation of each participant about mediation at the beginning of the Master’s program, the most cited word was “conflict”, followed by “family”, “children”, and “parenthood” for family mediation and “diversity”, “difference”, and “dialogue” for community mediation. At the intermediate assessment point, the most cited words were “conflict”, “agreement”, “relationships/bonds”, and “collaboration/cooperation” for family mediation and “conflict”, “community”, “society”, and “groups/people” for community mediation. In the final assessment phase, the most cited word was “conflict”, followed by “agreement”, “relationships/bonds”, “collaboration/cooperation”, and some specific terms referring to process and practices used in family mediation, such as “process”, “third-party status”, and “suitability for mediation” for family mediation; as for community mediation, the most cited words were “conflict”, “community”, “groups”, and “sense of community”, together with some words related to specific aspects of community mediation, such as “negotiation strategies”, “third-party status”, and “grounds for reflection”.

When called to list the skills a successful mediator should possess, participants named “listening skills” in the first place, then “neutrality/impartiality”, “empathy”, and “patience”. Answers to this question did not show differences between the three assessment points (i.e., at the beginning, halfway, and at the end of the Master’s program). In this direction, participants wished they gained the ability to “recognize” and “handle conflicts”.

The most frequently selected images recall the concepts of cooperation, collaboration, and team spirit in order to have a positive experience. Moreover, they all somehow refer to the inevitable risks connected to such an activity (the first image represents five people jumping together with the parachute, holding hands; the second image represents six people doing rafting).

Lastly, according to participants in this course, the two key elements that facilitated the training were: the way the topics were discussed and the trainers’ classroom management skills. Such an evaluation is connected to the pragmatism and practicality of the exercises that accompanied the development of a shared theoretical knowledge.

The above-mentioned elements allow both the entire staff and the trainers to acknowledge and better understand the group’s representation of mediation and of the role of the mediator as well as to explore their expectations and desires with respects to the Master’s program. Moreover, answers to those questions allowed a comparison between different time-points throughout the training, providing a qualitative evaluation of the training process.

When considering the proximal outcomes and therefore analyzing each participant and his/her individual profile, 19 participants have been found to show a “regular” profile, that is, they positively and purposefully responded to the proposals made during the training and seemed to appreciate both the theoretical contents as well as the practical activities carried out individually or in groups.

Six participants fell in the “fluctuating” profile as they found it somewhat harder to appreciate the Master’s program as a whole. Participants in this profile were constantly shifting between showing appreciation and involvement towards some contents and discontentment towards others. Five participants belonged to the “group-related struggles” profile as they made constant efforts to integrate with the group, build significant relationships, and express their own opinion during group activities. One participant fell in the “worsening” category, as her attitude toward the program became progressively more critical over time. On the contrary, another participant significantly improved her view of the course over time, thus falling in the “improvement” category. Lastly, one participant showed a “constantly critical” stance throughout the whole course (both with regards to its contents as well as to the practical activities).

With regards to process research, the grid we developed allowed us to pinpoint a few key issues. First, our outcomes clearly showed the importance and major role played by reflexivity. In this respect, trainees have been asked several questions to elicit their thoughts and collect their reflections. Answers promote the exchange of ideas and the development of shared knowledge in terms of reflexivity. Moreover, communication exchange can lead to agreement or disagreement. When disagreement occurs, rather than being censored, it leads to further discussion as the group negotiates between contrasting positions and reflexivity increases. This negotiation process usually helps the group gain further knowledge and/or develop different opinions than they previously had. In the few cases when disagreements were not discussed further, reflexivity could not be achieved. Lastly, process research allowed us to realize that, when strong emotional contents were explicitly and consciously shared among participants, they were acknowledged and validated by the group, even when they involved negative or particularly intense feelings (i.e., anger). It seems important to point out that, consistently with the learning environment in which they were assessed, the indicator regarding emotions was much less frequent than those related to reflexivity or disagreement management: While the first concerns a purely emotional dimension, the latter also has to do with cognition and thought.

7. Implications

In conclusion, the training process evaluation we presented in the present paper has been developed with the aim of combining research and practice. The underlying assumption at basis of our work is that research and classroom training should be conceived as interdependent and complementary processes, as the connection between the two allows for a constant monitoring of the training process while also taking into account the participants’ point of view. Moreover, the adoption of this standpoint on research and practice provides ground for the introduction of a “third perspective”—that of the researcher—other than that of the subjects involved. This favors dialogue and exchange between different points of view while at the same time eliciting a reflection on the meaning behind the choices made with respect to the training. That of the researcher represents a “third”, “outsider” position that dialogues and blends with that of the actors directly involved in the training process and enables a more articulate and deeper view of the whole process.

In this perspective, the constant exchange of ideas and the ongoing reflection among staff members elicit the adoption of a meta-perspective, which, in turn, results in a continuous review of the practice and in an assessment of the validity of both the modules and the training process as a whole. Specifically, such validity is measured both in terms of the correlation between the choices made and the goals achieved, as well as with regards to the evaluations provided by participants. In summary, it can be stated that this particular evaluation system allows change to happen within the working group, once again showing a significant analogy between the training and therapy processes.

8. Limitations and Future Perspectives

The evaluation system presented in the present contribution was developed at the start of the XI edition of the two-year second level Master’s program in family and community mediation and has been implemented involving the participants enrolled in this edition (academic years: 2018–2019 and 2019–2020) in order to test its efficacy and suitability to the goals of the program. Therefore, the results need to be contextualized within the specific experience of this particular training group. Specifically, this group was numerous (i.e., 33 participants) and heterogeneous (in terms of both sociodemographic characteristics and participants’ professional backgrounds).

In this perspective, further evaluations involving different training groups and other training processes, not necessarily connected to the field of mediation, are needed. In the long term, we aim to make this evaluation system flexible enough for it to be used across all the current and prospective Master’s programs in mediation as well as to be extended—with all the due changes—to other training contexts that, while sharing our view on training, deal with completely different topics. Although this contribution is descriptive and its goal is to present an evaluation system of training within a theoretical framework, we believe that it can be useful both for practitioners and researchers interested in the theme of evaluation of training programs, describing a complex system of evaluation that integrated outcomes and process dimensions [29].

We strongly believe that combining research and training practice together helped us shed light on the importance of working according to a multi-level and multi-actor logic, capable of taking into account the complexity of both the topic and the context in which training is delivered.

Lastly, we are convinced that our results prove a fundamental presupposition asserting the indissolubly circular tie between research and (training) intervention. Specifically, this presupposition allows us to understand the inherently transformative nature of group processes and supports the ongoing evaluation and redesign of training programs. In this view, practices are constantly reviewed, and the robustness of the training structure continuously monitored.