Make a Move: A Multi-Method, Quasi-Experimental Study of a Program Targeting Psychosexual Health and Sexual/Dating Violence for Dutch Male Adolescents

Abstract

Adolescent sexual and dating violence (SDV) is a worldwide problem. Although male adolescents in vocational education or youth care may be at increased risk of perpetrating SDV, little is known about effective gender-specific prevention. Therefore, we conducted a quasi-experimental evaluation of a Dutch group counseling program promoting psychosexual health and preventing SDV among male adolescents aged 12–18 years: Make a Move. The 66 participating male adolescents completed three questionnaires (baseline, post-test, 3-month follow-up; 48.5% retention). We also conducted interviews with a subsample of four adolescents and two program trainers and performed observations in one group. With these multi-method, multi-informant data, we evaluated program effectiveness on the six intended outcomes (attitudes, social norms, self-efficacy, skills, intentions, and SDV perpetration) by (1) statistically comparing self-reports between the intervention and control groups; (2) thematically analyzing interview data; and (3) describing three individual male adolescent cases, triangulating questionnaire, interview, and observation data. We found limitations in program integrity, evidence for program effectiveness on skills, and mixed evidence for effects on attitudes, but no evidence for effects on socials norms, self-efficacy, or SDV perpetration. Yet our interviews indicated perceived effectiveness on self-efficacy and intentions. We also found indications of adverse effects on attitudes and intentions. We offer suggestions for program refinement and future program evaluations.

1. Introduction

The Netherlands is known for its open climate around sexuality and good adolescent psychosexual health (e.g., relatively low rates of unintended pregnancies and sexually transmitted infections) compared to other counties (Harden, 2014; Schalet et al., 2014). Yet, both in the Netherlands and internationally, sexual and dating violence (SDV) among adolescents remain prevalent problems (De Graaf et al., 2024; Tomaszewska & Schuster, 2021). Sexual violence includes “any unwanted sexual activity where consent is not received or freely given” and “can occur within romantic relationships but also between acquaintances or strangers” (Graham et al., 2021, p. 439). Experiences with sexual violence can range from unwanted touching and rape to the non-consensual sharing of explicit online content. Dating violence includes emotional, physical, and sexual violence between dating partners (Center for Disease Control and Prevention, 2020). Young victims of SDV (male and female) may experience both short-term and long-term negative outcomes on multiple domains, including their physical, psychosexual, and mental health, academic performance, and future re-victimization (Basile et al., 2020; Campo-Tena et al., 2024). Preventing SDV experiences and subsequent negative consequences is therefore paramount, both for individual well-being and public health.

A recent national population study amongst 10,000 Dutch adolescents aged 13–25 years indicated that 23% of male adolescents and 54% of female adolescents had experienced some form of sexual violence, ranging from undergoing sexual touching to being forced to perform sexual acts against their will (De Graaf et al., 2024). Moreover, a large international meta-analytic review on adolescent dating violence indicated a prevalence of 20% for physical and 9% for sexual dating violence worldwide (Wincentak et al., 2017). Male adolescents are at an increased risk of engaging in SDV perpetration compared to female adolescents (De Graaf et al., 2024; Tomaszewska & Schuster, 2021). Specifically, male adolescents of a lower socio-economic status (e.g., in terms of vocational or educational tracks) or those in youth care often experience multiple risk factors for perpetrating SDV, including violence at home, relationships with delinquent peers, and psychosocial issues such as complex problems (Tharp et al., 2013; Ybarra & Thompson, 2018). Adolescents from lower socio-economic backgrounds are also typically more difficult to reach for prevention (Bonevski et al., 2014; De Gee et al., 2022).

Despite the pressing need for SDV prevention among male adolescents, studies evaluating such efforts are scarce. A systematic review found only a handful of evaluation studies of programs tailored to male adolescents compared to adult men (DeGue et al., 2014). Moreover, we know little about the prevention efforts for male adolescents from lower socioeconomic backgrounds. For instance, a systematic review of SDV prevention programs for male adolescents that were not enrolled in college found only 15 studies reporting on the effectiveness of 13 programs worldwide (Verbeek et al., 2023b), far fewer than the 98 studies on programs identified as being specifically aimed at the college students identified previously (DeGue et al., 2014). Finally, a global review on efforts targeting boys and men in sexual violence prevention found that less than 5% of the included studies evaluated programs specifically aimed at high-risk populations (Ricardo et al., 2011).

Also, we know little about the integrity and effectiveness of these programs. Most of the evaluation studies on SDV prevention programs for male adolescents limited their assessments to attitudes and behaviors, ignoring outcomes such as skills, self-efficacy, and social norms, even though many programs aim to change these outcomes (Fellmeth et al., 2015; Verbeek et al., 2023b). This means our current knowledge regarding the effectiveness of the programs targeting high-risk male adolescents and including a broad range of outcomes is limited. Moreover, small or null effects are commonly found in SDV prevention research, and they are difficult to explain without extra information on the context in which a program was implemented (DeGue et al., 2014; Verbeek et al., 2023b). Partially for this reason, there is an increasing call from the field of intervention research for using multi- or mixed-methods research when evaluating health-related and educational programs (Drabble & O’Cathain, 2015; Fetters & Molina-Azorin, 2020), especially when working with groups that are difficult to reach, on sensitive topics (Banyard et al., 2023; Condomines & Hennequin, 2014).

Four key benefits of employing multiple methods alongside rigorous quantitative program evaluations were outlined by Drabble and O’Cathain (2015). First, qualitative methods such as interviews and observations can provide alternative explanations for program (in)effectiveness. Second, assessing a program’s reception by stakeholders (i.e., organizations, program trainers, and participants) can gauge its real-world feasibility. Third, gathering diverse data types can address issues of statistical power in RCT analyses when recruitment fails among difficult-to-reach or small target populations. Finally, complex social interventions often involve multiple components and contexts, challenging standardized implementation. As such, employing multiple methods allows researchers to paint a better picture of a program’s complexity (Crooks et al., 2019).

In the current study, we evaluated the effectiveness of a group counseling program for male adolescents aged 12–18 years in vocational education, or those in youth care, called Make a Move (MaM, Jonker et al., 2020). MaM was developed in 2012 by Rutgers (Dutch Centre of Expertise on Sexual and Reproductive Health and Rights; https://rutgers.nl, accessed on 6 April 2025). The MaM program takes a self-proclaimed positive approach toward prevention based on cognitive–behavioral theories. It aims to promote psychosexual health through improving attitudes, social norms, romantic and sexual interaction skills, self-efficacy, and intentions underlying (SDV) behaviors. The program’s final goal is to prevent SDV perpetration. For a detailed description of the program characteristics, see Supplement S1. A previous randomized controlled trial (RCT) on MaM’s integrity and effectiveness found no significant effects (Van Lieshout et al., 2019). This may be partially explained by difficulties in program implementation and retention in both the program and the study (Van Lieshout et al., 2014, 2019). Based on this previous evaluation, Rutgers made changes to the MaM program’s set-up (e.g., fewer exercises per session) content (e.g., variations for some exercises), and certification training for program trainers (e.g., certifying staff within organizations and more attention to facilitating program elements such as intakes).

In the present study, we conducted a re-evaluation of the adapted MaM program regarding its current integrity and effectiveness. This re-evaluation was carried out in a real-world setting in which the program is currently being carried out independently of this research study (i.e., the researchers evaluated but did not influence program implementation). In line with the current best practices in prevention program evaluation research, we used multiple methods to assess the intended program outcomes. We combined data from self-report questionnaires from male adolescent participants, interviews with both male adolescent participants and trainers, and researcher observations of how male adolescents interacted with the program, each other, and the trainers.

As such, our first research aim was to evaluate where we see evidence for effectiveness on MaM’s intended outcomes. Specifically, (1) increasing positive attitudes toward sexual communication and acceptable sexual behavior, decreasing adversarial sexual beliefs, decreasing positive attitudes toward dating violence and rape myth acceptance, and decreasing heterosexual double standards; (2) reducing social norms accepting of SDV; (3) improving sexual interaction competency and peer pressure resilience skills; (4) increasing self-efficacy for sexual self-regulation and stating and respecting sexual desires and boundaries; (5) increasing positive and reducing negative intentions following sexual rejection; and (6) preventing the perpetration of SDV. We tested our hypothesis that changes between the pre- and post-test and between pre-test and follow-up would be more favorable in the intervention group than in the control group on all of the program’s fifteen intended outcomes (Supplement S1; Verbeek et al., 2021). Our second aim was to explain evidence for effectiveness on the outcomes with findings on program integrity. This dual focus allows not only for assessing MaM’s effectiveness in promoting psychosexual health and positive sexual behaviors but also understanding the role of program integrity in achieving these outcomes.

2. Methods

2.1. Make a Move

Make a Move (MaM) was developed by Rutgers in 2012 and updated in 2018 (Jonker et al., 2020). The final goal is to improve psychosexual health and prevent SDV perpetration among male adolescents aged 12–18 years. MaM consists of eight weekly 90 min group counseling sessions, facilitated by two trainers certified by Rutgers, to be implemented in schools, youth care institutions, or at youth service locations. For detailed descriptions of the goals, inclusion criteria, content, set-up, and trainers, see Supplement S1.

2.2. Design

The current multi-method, multi-informant study was part of the Move Up! Project (Verbeek et al., 2021), in which we evaluated two Dutch group counseling programs, Make a Move (MaM) and Make a Move+ (MaM+), an adapted version for adolescents and young adults with mild intellectual disabilities (for our evaluation of this program, see (Verbeek et al., 2025). The Move Up! Project was funded by The Netherlands Organization for Health Research and Development (ZonMw), project number 5550002017. The hypotheses and methods for the Move Up! Project (Verbeek et al., 2021), as well as the specific aims and analysis strategy for the current study (Verbeek et al., 2023a), were pre-registered on the Open Science Framework.

To evaluate the effectiveness of MaM, we conducted a multi-method, quasi-experimental study with an intervention and control group. The final design of this study deviates from the original project design, in which we planned to randomize organizations after enrollment (i.e., block RCT). Sample size was calculated a priori using G*Power (Faul et al., 2009), estimating a necessary 138 participants (69 per intervention arm) to detect small effects. However, when recruitment started in October 2020, a small number of organizations was able to implement the MaM program during the study due to staff shortages, scheduling issues, and COVID-19 precautions. Therefore, we decided to allocate all organizations that could implement MaM to the intervention group and recruited a separate control group that did not (yet) implement MaM, conducting a multi-method quasi-experimental study instead of a multi-method RCT.

Adolescents in intervention group organizations followed MaM during the study (i.e., between January 2022 and July 2023). Adolescents in control group organizations either did not follow MaM (i.e., treatment as usual) or followed MaM after the study ended (i.e., waitlist control group). The main trial consisted of a baseline, immediate post-test and a 3-month follow-up measurement (see Figure 1) to quantitatively measure the outcomes using self-report questionnaires. Questionnaires were complemented by qualitative data, for which we aimed to recruit a subsample of four groups from the intervention condition. From this subsample, one trainer and three male adolescents per group (i.e., four trainers and twelve adolescents) would participate in two interviews, and one intervention group would additionally participate in qualitative observations of all eight sessions. Recruiting a subsample was chosen to balance completeness of data with feasibility of participation in a real-world setting, considering the already large burden placed on school/youth care personnel and adolescents.

2.3. Procedure

2.3.1. Recruitment

We recruited organizations by contacting all trainers who had ever been certified to facilitate MaM via the program developer’s database with an information letter about the study, asking whether they and their organization were willing to participate. Next, we recruited trainers and organizations via school newspapers, LinkedIn, and already recruited trainers (i.e., snowball method). The inclusion criterium for organizations in the intervention group were that they implemented or planned to implement MaM with at least one group of male adolescents within the study timeframe. The inclusion criterion for control group organizations was that they had access to the target group of MaM (i.e., male adolescents in youth care or vocational education).

After inclusion, intervention group organizations selected eligible male adolescents to participate in MaM, who were then invited to participate in the study. Control group organizations identified 6–16 male adolescents (i.e., equal to one or two MaM groups) to participate in the study. Organizations selected adolescents—without involvement of the researchers—based on whether they considered them eligible for MaM as they would in a real-world setting outside of the study. Inclusion and exclusion criteria of the program included the following: some interest in romantic relationships and sexuality, no history of sexual offense or traumatic SDV experiences, and being able to function within a group setting. Fitting a real-world setting, some organizations also applied other criteria (e.g., “All boys in year 3 follow MaM”).

2.3.2. Consent and Ethical Standards

After selection, an individual from the organization informed eligible male adolescents through an information video and letter from the researchers covering the study’s procedure and ethical standards, whereafter those agreeing to participate signed informed consent. For those under 16 years old, parents or legal guardians also signed informed consent. Those not participating in the study could still join MaM.

From the intervention group, we recruited a subsample of four groups (i.e., two youth care and two schools)—of which one trainer and a maximum of four male adolescents per group were to participate in two individual interviews, before and after the program. After trainers agreed to participate, male adolescents from their MaM group were recruited via an additional information video and letter from the researchers, after which they—and if applicable, their parents or legal guardians—signed informed consent. The research protocol of Move Up! was evaluated by the Ethics Review Committee of the Department of Psychology, Education, and Child Studies of Erasmus University Rotterdam and approved (decision number 21-026).

2.3.3. Data Collection

Self-report questionnaires measuring outcomes: Online self-report questionnaires measuring demographic characteristics and program outcomes were collected physically at organizations using QualtricsXM (Qualtrics, Provo, UT, USA) on a laptop, computer, smartphone, or tablet. Male adolescents did not have to travel to participate in the research; researchers or trained research assistants visited their schools or youth care institutions at a moment when the adolescents were present, where they explained the procedure and remained available to answer any questions. We simplified the questionnaire language using “Language for all” (Moonen, 2021) and built-in audio clips to listen to longer parts of the questionnaire and provided headphones to keep as a gift.

Interviews gauging perceptions of effectiveness: The qualitative interviews and observations were conducted using a semi-structured topic list by two researchers from Atria, Institute on Gender Equality and Women’s History (https://atria.nl/, accessed on 18 December 2024). Researcher JK, a 29-year-old cis man of color with a bi+ sexual orientation and an educational university background in sociology, handled all pre-interviews and half of the post-interviews and observed the first two sessions. Researcher BM, a 34-year-old white cis woman with a heterosexual orientation and a PhD in Medical Anthropology, conducted half of the post-interviews and observed the final six sessions. Interviews with male adolescents were conducted face-to-face and lasted around 20 min (range 15–40 min). Interviews with trainers were conducted face-to-face or, in cases where it was not possible to match the trainer’s agenda otherwise, online via Teams. These interviews lasted around 40 min. The interviews were recorded using a digital recording device. At the start of each interview, just after starting the recording, participants were once again asked to verbally confirm their consent to participate in the interview. The audio files were securely stored in a secure online cloud service (SURFdrive). Once the audio files were uploaded to SURFdrive, they were deleted from the recording device. The audio files were transcribed verbatim by an external professional transcribing bureau, which ensured anonymity of the transcripts by removing all identifying information. Only anonymized transcripts were used in the qualitative analyses for in this study. During the qualitative observations, the researchers physically attended the sessions and made field notes using a semi-structured observation form, without interacting with the trainers or adolescents during the sessions.

2.3.4. Compensation

All organizations, trainers, and adolescents were compensated for participating in this study. Participating organizations could have one (additional) employee certified as a MaM trainer (EUR 375 value). All trainers received the latest program materials (EUR 45 value) and a EUR 15 gift card per interview if they participated in the pre- and post-interviews. Male adolescents received a EUR 7.50 gift card for each interview and received a EUR 5 gift card for each completed questionnaire. Those completing all three questionnaires were eligible to win a EUR 20 gift card.

2.4. Participants

2.4.1. Organizations

The ten participating organizations were six special secondary vocational education schools (e.g., including youth care elements or additional personal guidance for those with complex problems), three vocational colleges, and one residential youth care organization.

The intervention group consisted of five special secondary vocational education schools, one youth care organization, and one regular vocational secondary school. The control group consisted of three vocational colleges and one special secondary school. Recruitment and obtained numbers are presented in the flowchart in Figure 1.

2.4.2. Total Male Adolescent Sample

Sixty-six male adolescents participated, aged 12 to 21 years (M = 16.6, SD = 2.4; 43.9% in the intervention group, see Figure 1). This sample size is smaller than the originally intended sample size of 138 adolescents based on a priori power analyses, pre-registered at the start of the project (Verbeek et al., 2021), due to difficulties in recruitment described under Design. Moreover, we experienced difficulties in retention (48.5% at follow-up); see Figure 1 for more details.

Characteristics of the male adolescents at baseline are presented in

Table 1. Characteristics of the participating male adolescents by condition at baseline.

Baseline Characteristics	Total N = 66	Intervention n = 29	Control n = 37
Age (M, SD)	16.59 (2.42)	14.52 (1.15)	18.14 (1.84)
	% Total	% Intervention	% Control
Ethnocultural identity a
Fully Dutch	69.1	62.1	72.2
Surinamese(-Dutch)	7.3	13.8	2.9
Turkish(-Dutch)	1.5	0	2.8
Asian(-Dutch)	5.8	6.9	5.6
Mixed ethnicity, non-Dutch	1.5	0	2.8
Other European(-Dutch)	7.4	10.3	5.6
Moroccan(-Dutch)	4.4	6.8	2.8
African(-Dutch)	2.9	0	5.6
Religious	22.9	20.7	21.6
Daily occupation
School	89.5	95.5	87.1
Work	10.5	4.5	12.9
Level of education
Elementary School	1.6	3.6	0
Special Secondary Education	9.4	0	15.6
Secondary Education	53.1	70.8	37.5
Vocational Secondary School	25.0	3.6	43.8
Unknown/no education	10.9	21.4	3.1
Living situation
With both parents	45.7	31.0	56.8
With one parent	30.0	37.9	24.3
In youth care	8.6	17.2	2.7
With one parent and stepparent	5.7	6.9	8.1
Other (e.g., foster, brother, alone)	5.7	3.4	8.1
Switching between divorced parents	2.9	6.9	0
Romantic relationship experience	84.1	89.7	81.1
Sexual Orientation
Heterosexual	92.8	86.2	97.3
Bisexual	2.8	6.8	0
Homosexual	1.4	3.4	0
Not sure yet/would rather not say	2.8	3.4	2.7
Any interpersonal sexual experience	65.2	58.6	73.0
Of non-experienced: Intention to have sex next year
Yes (probably)	26.7	18.8	35.7
Maybe, maybe not	33.3	25.0	42.9
No (probably not)	40.0	56.2	21.4

^a Ethnocultural identity was subjectively measured as “I feel … [e.g., Dutch, Moroccan, etc.]”.

. Over half identified as fully Dutch; the rest identified as a mix of ethnocultural identities (e.g., Turkish-Dutch) or fully as another ethnocultural identity (e.g., Moroccan). The sample was representative of Dutch adolescents in vocational education in ethnic identity composition (Walhout et al., 2022) and religiosity (Statistics Netherlands, 2020).

Most adolescents were in school, mostly vocational secondary education or vocational college. Just under half of the adolescents lived with both parents. Others lived with one parent or switched between parents, in youth care, or otherwise (e.g., with grandparents). Almost all adolescents had been in love and reported themselves as heterosexual. Most had French kissed and just under half had experience with vaginal intercourse. Most had received some education regarding love and/or sexuality (86.4%) and rated this a 6.6 out of 10 on average.

Regarding psychosocial functioning, assessed with the Strengths and Difficulties Questionnaire (SDQ, Van Widenfelt et al., 2003), 52.3% scored in the normal, 26.2% in the borderline, and 21.5% in the clinical range on Total Problems. Hence, 47.7% of our sample (55.2% of the intervention group) experienced at least some psychosocial problems (Vugteveen et al., 2022).

During the pre-test, the intervention (n = 23) and control group (n = 43) did not differ in ethnic identity, religiosity, experience with love or sex, or psychosocial functioning. However, the intervention group was younger (M = 14.52, SD = 1.15) than the control group (M = 18.14, SD = 1.84), t(61.3) = 9.75, p = 0.010, and therefore more often enrolled in secondary vocational school, whereas control group adolescents were more often enrolled in vocational college, Χ²(7) = 24.92, p < 0.001. Also, fewer adolescents in the intervention group lived with both parents (e.g., with one parent, or in youth care), Χ²(1) = 4.34, p = 0.037.

2.4.3. Qualitative Subsample

The two youth care organizations that we recruited for the qualitative part of the study never started MaM after enrollment in the study, so we could not use their data for this effectiveness evaluation. The final subsample for the qualitative part of the study consisted of two program trainers, with whom we had conducted complete pre- and post-program interviews, and the adolescents from their groups who completed the pre- and post-program interviews (n = 4, two from each group).

Both trainers were female, 39 and 43 years old, and certified to deliver MaM. One worked as a coach and supervisor within a secondary school with youth care elements, and the other worked as a clinical psychologist within a special education secondary school. They had 10 and 14 years of experience working with the target group, respectively.

The observations were carried out at a special education secondary school in the group of the second trainer, facilitated by this trainer and two additional not (yet) certified co-trainers—one 26-year-old women and one 58-year-old man. This group consisted of seven male adolescents, and descriptions of three of these were included in the current study’s multi-method case descriptions; see Analysis Step 3.

2.5. Program Integrity Evaluation

We assessed four elements of program integrity (Dane & Schneider, 1998; Lemire et al., 2023): (1) adherence (i.e., completeness of program delivery), (2) dosage (i.e., session length and number of sessions attended), (3) quality of delivery by trainers, and (4) participant responsivity (e.g., active participation and appreciation). See Point 12 in Supplement S1 for a description of the methods used.

2.6. Outcome Measures

2.6.1. Questionnaires

Based on MaM’s final program goal and change objectives (see Supplement S1), we assessed five types of sexuality-related attitudes, two types of SDV-related social norms, peer pressure resilience skills, three aspects of sexual self-efficacy, two types of intentions following sexual rejection, and SDV perpetration. These 15 program goals and used instruments were described in detail in the project pre-registration (Verbeek et al., 2021) and are summarized in

Table 2. Questionnaire instruments and interview questions measuring program outcomes.

Concept	Instrument	Number of Items	Example Item	Answer Options	Scale Score	Cronbach’s α
Attitudes
Adversarial Sexual Beliefs	Burt (1980), adapted by Van Lieshout et al. (2019)	3	I think women mostly date men to make use of them	1 = Strongly disagree—5 = Strongly agree	Mean	0.61
Attitudes Toward Dating Violence	Attitudes toward male dating violence (AMDV) scale (Price et al., 1999) used in Van Lieshout et al. (2019)	8	Some girls/boys deserve to be slapped by their boyfriends	1 = Strongly disagree—5 = Strongly agree	Mean	0.73
Attitudes Toward Positive Sexual Behavior	Perceived Sexual Competence Scale (Deković et al., 2018)	5	When I have sex … (Adapted from ‘I [do] …’ to) I think it is important to … pay a lot of attention to what the person whom I have sex with likes	1 = Strongly disagree—5 = Strongly agree	Mean	0.86
Attitudes Toward Sexual Communication	Van Lieshout et al. (2019)	3	Asking my girlfriend/boyfriend what they do and do not want during sex, seems to me … Good/Important/Comfortable	[item 1]; 1 = Not good at all—5 = Very good	Mean	0.91
Heterosexual Double Standards	Scale for the Assessment of Sexual Standards Among Youth (SASSY; Emmerink et al., 2017)	6	I think that a girl who takes the initiative in sex is pushy	1 = Strongly disagree—5 = Strongly agree	Mean	0.82
Rape Myth Acceptance	Items from Illinois Rape Myth Acceptance Scale Short Form (IRMA-SF; Bendixen & Kennair, 2017)	4	Rephrased into short vignettes: A girl got drunk at a party. Someone fingered her while she did not want to. A friend of yours says ‘If she was that drunk	1 = Strongly disagree—5 = Strongly agree	Mean	0.76
Attitudes
Rape Myth Acceptance	Items from Illinois Rape Myth Acceptance Scale Short Form (IRMA-SF; Bendixen & Kennair, 2017)	4	it’s kind of her own fault something like that happened’, to what extent do you agree with this friend?	1 = Strongly disagree—5 = Strongly agree	Mean	0.76
Attitudes and Cognitions about Love, Intimacy and Sexuality	Interview Questions	Youth: Did MaM change your mind about love and sexuality? If answered affirmatively: Can you give an example? If you’ll have sex in the future, what do you think is important you do? And your partner? Or worded differently, as: What is ‘good’/’bad’ sex, according to you? Trainers: To what extent do you think adolescents think differently about love, intimacy and sexuality after MaM? ⚬ If answered affirmatively: About which things do they think differently? Possible follow-up question: How did you notice that? To what extent do you think adolescents internalized the messages about indicating and respecting desires and boundaries? ⚬ Was it new information or did they already have an opinion about this? ⚬ Did they quickly get it, or did it take longer? Did the group reach consensus about this? ⚬ Do you think adolescents now think differently about indicating their own desires and boundaries?
Social norms
Injunctive Norms	Self-constructed, based on previous work (e.g., see Pedneault et al., 2022)	2	You kiss a person you like, even though this person did not want to. How acceptable do your (1) friends; (2) parents think this is?	1 = Not at all acceptable—5 = Very acceptable	Mean	0.80
Perception of Normalcy of SDV	Perceived Social Norms Scale (De Lijster et al., 2016), rephrased	5	A friend of yours randomly pinches someone’s butt. How normal do you consider this behavior to be?	1 = Not normal at all—5 = Very normal.	Mean	0.59
Skills
Resilience to Peer Pressure	Peer Pressure Scale (Santor et al., 2000) translated and item added by Deković et al. (2018)	6	I’ve done dangerous or foolish things because others dared me to	1 = Never—6 = Very often	Mean	0.57
Romantic and Sexual Interaction Competency	Interview questions	Adolescents: After following MaM, is it now easier for you to talk about love and sexuality? Which things that you learned during MaM+ will you use when you’re going to have sex? And which things in a relationship? If you would rate your current confidence in flirting/dating on a scale of 1–10, what would it be? Follow-up after a rating was given: Why? Follow-up after MaM and after a rating was given: That is the same/higher/lower than you said before MaM. Is this because of MaM? If you would rate your current confidence in sex on a scale of 1–10, what would it be? Follow-up after a rating was given: Why? Follow-up after MaM and after a rating was given: That is the same/higher/lower than you said before MaM. Is this because of MaM?
Self-efficacy
Self-efficacy to ‘say no’	‘Say No’ subscale from the Sexual Self Efficacy Scale (SSE; Rosenthal et al., 1991)	5	Rephrased into short vignettes: You’ve been with you partner for a while. You’ve kissed before, but you haven’t engaged in anything sexual yet. One night you’re lying on the bed. Suddenly they push your hand into their pants. You don’t feel comfortable with this yet. How good are you at indicating that you don’t want to do this yet?	1 = Not good at all—5 = Very good	Mean	0.80
Self-efficacy for Self-Regulation	Van Lieshout et al. (2019)	4	I am able not to get angry when my partner isn’t in the mood for sex	1 = Not at all—5 = Completely	Mean	0.92
Self-efficacy for Sexual Communication	Van Lieshout et al. (2019)	5	I am able to talk to the person I’m having sex with about what I do and don’t want	1 = Not at all—5 = Completely	Mean	0.91
Intentions			Vignette: Imagine you are in a relationship. You are in bed together, talking. You are starting to feel like you want to have sex, but the other person does not want to. What do you do?
Positive	One item from Van Lieshout et al. (2019), two self-constructed	3	When I want to have sex, but the other person does not, I will: Example item: Leave them alone	1 block = Very unlikely to react like this—7 blocks = Certain to react like this	Mean	0.67
Negative	Van Lieshout et al. (2019)	3	When I want to have sex, but the other person does not, I will: Example item: Get angry	1 block = Very unlikely to react like this—7 blocks = Certain to react like this	Mean	0.76
SDV Perpetration	Sexual Abuse Subscale 5 items, Foshee et al. (2015); one item from De Haas et al. (2012); one item self-constructed	7	How often have you sent or shown a naked picture of someone else to others? At post-test & follow-up: Since the last measurement	0 = Never, 1 = 1 or 2 times, 2 = 3 or 4 times, 3 = More than 4 times.	Sum (0–7) & Dichotomous (0 = No experience, 1 = At least one experience)	NA
Behavioral Change	Interview questions:	Adolescents: Which things that you learned during MaM will you use when you’re going to have sex? And in a relationship?
General Estimates of Program Effectiveness by Participants	Interviews questions:	Adolescents Can you name two things you learned during MaM? Trainers Did you notice anything with the adolescents in conversations or their behavior that indicated that MaM had an effect? If yes, what? External: Did you hear from the adolescents or organization after MaM? If yes: What kind of reactions? Do you feel like adolescents can still remember and actively use the things they learned? ⚬ Possible follow-up question: Did you notice this in any way (for instance, see behavior or heard about incidents)?

NA = Not applicable.

.

2.6.2. Interviews

As per our multi-method design—to supplement the quantitatively measured outcomes—we used the pre- and post-program interviews with male adolescents and trainers to assess program effects qualitatively as well. This included asking male adolescents specific questions regarding their self-perceived sexual and relational interaction competency skills and asking trainers and male adolescents about (changes in) cognitions around sex; see Table 2.

More broadly, we also asked trainers and male adolescents about (intended) behavioral changes after MaM. Lastly, fitting a qualitative design, we asked open-ended questions before and after the program to elucidate their perspectives on what adolescents would learn (before) and had learned (after), without asking specifically about a priori program goals; see Analysis Step 2.

2.6.3. Observations

During the observations of all sessions in one group, a semi-structured observation form was used. First, a chronological observation report was created for each session, with descriptions of interactions among the male adolescents, between the male adolescents and the program trainers, and their engagement with the MaM materials. Throughout this note-taking process, the researcher described events as objectively as possible. If a specific interpretation was added to an observed moment, it was explicitly noted in the report (i.e., field notes). Second, after the program session, the researcher reflected on several specific topics, such as how the adolescents responded to the materials and whether they appeared to understand them. These observations were used to illustrate possible program outcomes; see Analysis Step 3.

2.7. Analyses

We used a three-step analytic strategy (pre-registered on OSF; see Verbeek et al. (2023a), using (1) confirmatory quantitative group comparisons on program outcomes measured with the questionnaires, according to our pre-specified hypotheses; (2) perspectives of trainers and adolescent participants on program effectiveness as reported in the interviews; and (3) an exploratory, person-centered description of program effectiveness integrating questionnaire, interview, and observation data of three adolescents. We based this strategy on the multi-method set-up of our study (i.e., to assess program effectiveness both quantitatively and qualitatively) and the fit with the obtained sample size and available multi-method, multi-informant data (i.e., adding exploratory case descriptions). Due to the low numbers of participants, we did not employ methods to minimize potential bias induced due to non-randomization. We also did not distinguish between primary and secondary outcomes, as Rutgers also made no such distinction in the program manual.

2.7.1. Step 1: Quantitative Group Comparisons of Program Outcomes

In Step 1, we analyzed the questionnaire data on all program outcomes to compare the intervention and control groups (intention to treat) in pre-post (n = 37) and pre-follow-up (n = 30) change scores using Bayesian analyses in JASP Version 0.18.3 (JASP Team, 2024). For analyzing group differences in the change scores, we used Bayesian t-tests and for analyzing differences in frequencies of SDV perpetration (none versus at least one type) at post-test and follow-up, we used Bayesian contingency tables.

Bayesian analyses are recommended in case of small samples and multiple outcomes, as this method has no risks of type I or type II errors (Van de Schoot et al., 2014). Moreover, it provides a “magnitude of support” for H₁ (i.e., there is a group difference in mean change over time or frequency) versus H₀ (i.e., the groups are equal) instead of an all-or-nothing decision based on p-values. Resulting statistics from this method are Bayes Factors (BFs) and confidence intervals for the posteriorly estimated effect size. BFs of 1 indicate equal support for both hypotheses; BFs of 1–3 indicate “weak” evidence for H₁, BFs of 3–10 indicate “moderate” evidence for H₁; and BFs > 10 indicate “strong” evidence for H₁ (Van Doorn et al., 2021). BFs < 1 provide more support for H₀.

Finally, Bayesian analyses allow for inclusion of information from previous studies to accumulate evidence (Miočević et al., 2020). To this end, we used prior information from the previous effect evaluation of MaM for outcomes where this was available (Van Lieshout et al., 2019). For the three outcomes that were not used in this previous evaluation, we used the default in JASP, a non-informative Cauchy prior of 0.707.

2.7.2. Step 2: Perspectives of Program Trainers and Male Adolescents

In Step 2, we analyzed the pre- and post-program interview transcripts. The interviews were analyzed in four steps by the first author and a research assistant, both with educational university backgrounds in developmental psychology and with 6 and 2 years of qualitative research experience, respectively. First, the research assistant openly coded all interviews using ATLAS.ti Version 24 (Scientific Software Development GmbH, 2023) to capture relevant information without imposing preconceived categories or themes, which could already highlight participants’ accounts of program effects (Boeije, 2014). Second, guided by our research questions, we used selective coding to identify codes related to program effectiveness—specifically, trainers’ (n = 2) verbal reports on what they felt male adolescents had learned from MaM and, for male adolescents (n = 4), their responses on what they learned, changes in their confidence regarding flirting/dating and sex, changes in their attitudes and sexual or romantic behavior, and other (un)intended program outcomes mentioned. Third, we organized emerging themes into pre-specified categories related to the program goals. Fourth, we analyzed differences and commonalities in responses to discern whether program effects were found in interviews with all male adolescents or trainers, selectively appeared among certain individuals, or were identified by only one participant. We translated interview quotes from Dutch to English to support and illustrate the findings.

Regarding the positionality of the first author and research assistant analyzing the qualitative data, the research assistant’s neurodivergence (i.e., autism) and interest in gender and sexual diversity may have increased sensitivity to related issues, while the first author’s specialization in normative adolescent development may have heightened alertness to risk-focused and restrictive views on adolescents’ sexuality.

2.7.3. Step 3: Multi-Method Case Descriptions

In Step 3, we exploratively analyzed questionnaire, interview, and observation data from the three adolescents from the one intervention group that participated in the observations that we also interviewed. We assessed how they perceived MaM and how it affected them using a person-specific approach (Howard & Hoffman, 2018) based on Ridderinkhof et al. (2021). We described adolescents’ background (i.e., age, living situation, and experience with love, relationships and sexuality) and psychosocial functioning at the start of MaM. Next, we integrated their questionnaire data using individual plots per goal, their interviews (see Step 2), and observations on how they interacted with the program. This resulted in indications of effectiveness on the program goals and unexpected outcomes for each adolescent.

3. Results

3.1. Baseline Characteristics

Average scores on the questionnaire outcomes at pre-test, post-test, and follow-up for the intervention and control group are presented in

Table 3. Scores on the continuous outcomes at pre-test, post-test, and 3-month follow-up for the groups.

	Pre-Test M (SD)		Post-Test M (SD)		Follow-Up M (SD)
	Intervention	Control	Intervention	Control	Intervention	Control
Attitudes	n = 27	n = 35 a	n = 16 b	n = 23	n = 7	n = 24 c
Adversarial sexual beliefs	2.33 (0.82)	2.56 (0.68)	2.78 (1.19)	2.59 (1.02)	3.33 (1.20)	2.51 (0.86)
Attitudes toward communication	4.15 (1.00)	3.99 (1.06)	4.15 (0.88)	3.90 (0.94)	4.14 (0.63)	4.03 (1.11)
Attitudes toward dating violence	2.07 (0.59)	1.83 (0.49)	2.38 (0.74)	2.06 (0.67)	2.86 (0.84)	2.18 (0.88)
Attitudes toward positive sexual behavior	3.94 (1.10)	3.87 (0.92)	4.41 (0.65)	3.86 (0.94)	3.80 (1.20)	3.99 (0.88)
Heterosexual double standards	1.93 (0.69)	2.05 (0.77)	2.09 (0.96)	2.09 (0.91)	2.38 (0.95)	2.13 (0.74)
Rape myth acceptance	2.36 (0.99)	2.07 (0.77)	2.16 (0.58)	2.13 (0.72)	2.75 (0.79)	1.89 (0.80)
Social Norms	n = 27	n = 31	n = 16	n = 23	n = 7	n = 24
Injunctive norms	1.85 (0.86)	2.15 (1.29)	1.63 (0.70)	1.67 (0.86)	1.71 (0.39)	1.63 (0.73)
Perceiving SDV as normal	1.81 (0.72)	1.73 (0.47)	1.74 (0.53)	1.83 (0.72)	1.94 (0.57)	1.53 (0.49)
Skills	n = 27	n = 35	n = 16	n = 23	n = 7	n = 24
Resilience to peer pressure	5.11 (0.56)	5.26 (0.52)	5.31 (0.49)	5.01 (0.72)	5.45 (0.49)	5.43 (0.51)
Self-efficacy	n = 27	n = 31	n = 16	n = 23	n = 7	n = 24 d
Say no	3.84 (0.80)	3.87 (0.94)	4.00 (0.69)	3.85 (0.69)	4.17 (0.48)	3.51 (1.21)
Self-regulation	4.28 (1.05)	4.02 (1.00)	4.38 (0.72)	4.14 (0.99)	4.11 (0.72)	4.03 (1.01)
Sexual communication	4.17 (1.00)	3.76 (1.13)	4.13 (0.65)	3.78 (1.01)	4.25 (0.51)	3.77 (1.07)
Intentions	n = 27	n = 35	n = 16	n = 23	n = 7	n = 24
Positive intentions	5.78 (1.38)	4.77 (1.36) **	5.48 (1.33)	5.22 (1.39)	4.10 (1.72)	4.51 (1.65)
Negative intentions	1.51 (0.91)	1.82 (1.14)	1.46 (0.83)	1.88 (0.99)	1.71 (0.80)	1.65 (1.10)

^a Due to technical issues with the questionnaires, we obtained n = 31 for rape myth acceptance; ^b n = 15 for adversarial sexual beliefs and heterosexual double standards; ^c n = 23 for adversarial sexual beliefs, heterosexual double standards, and rape myth acceptance; and ^d n = 23 for self-efficacy to ‘Say no’. ** p < 0.01 for the difference in mean scores between groups at this timepoint.

and

Table 4. SDV perpetration behaviors at the different timepoints for the intervention and control groups.

	Pre-Test (%)		Post-Test (%)		Follow-Up (%)
	Intervention (n = 27)	Control (n = 34)	Intervention (n = 16)	Control (n = 23)	Intervention (n = 7)	Control (n = 24)
Coercion into sexual acts	0.0	0.0	0.0	0.0	0.0	0.0
Forwarding or showing others’ nude/sexy photos without consent	22.2	11.8	25	0.0	14.3	0.0
Kissing without consent	18.5	8.6	6.3	4.3	0.0	8.3
Persuading into sexual acts	7.4	8.8	12.5	8.7	28.6	8.3
Showing genitals without consent	0.0	5.9	6.3	8.7	0.0	0.0
Showing pictures/videos of naked people without consent	14.8	34.3	18.8	4.3	0.0	4.2
Touching private parts without consent	3.7	2.9	12.5	4.3	0.0	0.0
Range in sum of experiences	0–4	0–3	0–6	0–3	0–2	0–1
Any SDV in %	37.0	50.0	25.0	13.0	28.6	20.8

Notes. This table presents frequencies of SDV experiences per group at pre-test, post-test, and follow-up. The range represents the minimum and maximum number of SDV experiences reported by adolescents. Any SDV in % presents the percentage of adolescents who reported at least one SDV experience versus none. These are the percentages used for the Bayesian contingency table analysis.

. At pre-test, the groups did not differ on 14 of the 15 program outcomes, but intervention adolescents reported more positive intentions following sexual rejection than the control group, t(60) = 2.87, p = 0.003. Comparisons between retained adolescents and those lost to follow-up on baseline demographic characteristics and program outcomes showed that retained adolescents were older, t(64) = 3.28, p < 0.001, scored higher on adversarial sexual beliefs, t(60) = 2.21, p = 0.031, adhered more to heterosexual double standards, t(60) = 2.64, p = 0.010, scored higher on injunctive norms accepting of SDV, t(56) = 2.03, p = 0.047, and scored higher on negative intentions following sexual rejection, t(60) = 2.30, p = 0.025, than those lost to follow-up. We could not compare retention in the study arms separately, as numbers were too small.

3.2. Program Integrity

See Point 12 in Supplement S1 for a description of the results, summarized below.

3.2.1. Adherence

On average, trainers reported implementing 63.4% of the program manual content completely. Trainers mostly skipped introductory and concluding elements of the sessions and of separate exercises and the roleplays that were often programmed after group discussions to practice what was discussed (e.g., how to recognize non-verbal signals). The main reason given for skipping elements was lack of time. Moreover, groups were regularly larger than six to eight adolescents (e.g., up to twelve) and sometimes included adolescents with contra-indices (e.g., cannot function in a group setting).

3.2.2. Dosage

Two organizations implemented all eight sessions, one organization implemented MaM in four sessions of 45 min, and three did not start or fully finish the program. Session length ranged from 45 to 70 min (M = 54 min), instead of 90 min. Reasons for limitations in dosage were lack of time and trainers’ estimated concentration span of the participating adolescents. Adolescents attended between two and eight sessions, with an average of six sessions (75% of the program).

3.2.3. Quality of Delivery

Program trainers were aware of the program goals, and they prepared and adapted program content and intensity to the group. Assessed with the B-test questionnaire (Van Erve et al., 2007; Harder et al., 2013) all adolescents except one scored trainers’ skills as satisfactory or higher on a scale of 1–6, M = 5.40. They scored trainers most favorably on reliability and respectfulness. However, not all trainers were certified in MaM. Also, trainers sometimes had trouble reacting or chose not to react to adolescents’ negative behavior or inappropriate (e.g., sexist or homophobic) statements during the sessions. As such, quality of delivery may have varied between trainers.

3.2.4. Participant Responsivity

After MaM, adolescents rated the program positively (Range = 7.0–7.5 out of 10). They said MaM was active, interesting, and clear, and many did not make suggestions for improvement. The recommended 15 min individual intake conversations to assess and possibly increase adolescents’ motivation and check for contra-indices were not always conducted, leaving some adolescents not knowing what to expect of MaM. Nevertheless, adolescents were neutral to positive towards following the program. During MaM, adolescents showed active, but also distracted, joking, and unmotivated, behavior. Adolescents seemed to appreciate having a male trainer yet did not complain if they had female trainers.

3.3. Effectiveness Step 1: Quantitative Group Comparisons on Program Outcomes

First, we found statistical evidence for program effectiveness on one attitude outcome and on skills. Our Bayesian independent samples t-test indicated that rape myth acceptance decreased more in the intervention group at post-test than in the control group, with a BF₁₀ of 2.00 indicating weak evidence for this difference; see

Table 5. Outcome of Bayesian comparisons of the intervention and control group on the change scores baseline to post-test and baseline to follow-up.

Outcome	Baseline—Post-Test				Baseline—Follow-Up
	Prior M (SD)	BF10	Posterior δ	95% CI	Prior M (SD)	BF10	Posterior δ	95% CI
Adversarial sexual beliefs	−0.324 (0.672)	0.519	0.212	−0.376, 0.820	−0.377 (0.588)	1.442	0.525	−0.181, 1.232
Attitudes toward communication	0.411 (0.661)	0.377	−0.010	−0.602, 0.581	0.361 (0.612)	0.487	0.033	−0.659, 0.725
Attitudes toward dating violence	0.042 (0.507)	0.558	0.035	−0.517, 0.588	−0.436 (0.409)	0.664	−0.291	−0.873, 0.292
Attitudes toward positive behavior	NI Cauchy (0.707)	0.598	0.325	−0.256, 0.961	NI Cauchy (0.707)	0.443	−0.176	−0.927, 0.514
Heterosexual double standards	−0.115 (0.589)	0.502	−0.039	−0.625, 0.547	−0.401 (0.499)	0.480	−0.046	−0.688, 0.597
Rape myth acceptance	−0.249 (0.607)	2.004	−0.528	−1.136, 0.079	−0.064 (0.445)	0.786	0.150	−0.470, 0.769
Injunctive social norms	0.176 (1.114)	0.323	0.135	−0.523, 0.794	0.324 (1.036)	0.691	0.452	−0.353, 1.257
Perception of normalcy of DV	0.176 (1.114)	0.308	−0.087	−0.746, 0.571	0.324 (1.036)	0.491	0.299	−0.501, 1.099
Resilience to peer pressure	0.333 (0.771)	5.367	0.738	0.110, 1.367	0.328 (0.837)	0.923	0.480	−0.280, 1.239
Self-efficacy to ‘say no’	NI Cauchy (0.707)	0.346	−0.074	−0.686, 0.520	NI Cauchy (0.707)	0.849	0.459	−0.273, 1.319
Self-efficacy for self-regulation	0.019 (0.760)	0.487	−0.173	−0.801, 0.455	−0.446 (0.721)	0.455	0.117	−0.624, 0.859
Self-efficacy for sexual communication	0.487 (0.572)	0.368	−0.030	−0.619, 0.559	0.352 (0.609)	0.776	0.338	−0.336, 1.042
Positive intentions	NI Cauchy (0.707)	0.421	−0.211	−0.810, 0.351	NI Cauchy (0.707)	1.070	−0.489	−1.209, 0.151
Negative intentions	0.011 (0.870)	0.364	0.016	−0.603, 0.636	0.285 (0.946)	0.403	−0.070	−0.841, 0.700

Note. This table presents Bayesian t-test analyses of the change scores over time. The prior indicates the effect used as prior information, the BF₁₀ is the Bayes factor for the alternative hypothesis over the null hypothesis, and the posterior δ and 95% CI represent the estimated effect based on data from the current study considering the prior information. NI = non-informative.

. Moreover, from pre-test to post-test, resilience to peer pressure increased more in the group of male adolescents who participated in MaM than in the control group, with a BF₁₀ of 5.37 indicating moderate evidence for this difference.

Second, we found no statistical evidence for differences between the groups (BF₁₀ < 1) in changes in attitudes toward positive and acceptable sexual behavior, attitudes toward communication with a sexual partner, attitudes toward dating violence or adherence to heterosexual double standards, or any of the self-efficacy outcomes, social norms or negative intentions following sexual rejection; see Table 5.

From the Bayesian contingency tables, we also found no statistical evidence for differences between the groups in the percentage of adolescents that reported no versus any SDV perpetration at post-test (BF₁₀ = 0.723) or follow-up (BF₁₀ = 0.471); see Table 4.

Finally, we found evidence for program outcomes in the opposite direction of what would be expected on two outcomes: the intervention group showed a larger increase in adversarial sexual beliefs than the control group from pre-test to follow-up, with a BF₁₀ of 1.44 indicating weak evidence for this difference. Moreover, the intervention group decreased more in positive intentions following sexual rejection at follow-up than the control group, with a BF₁₀ = 1.07 indicating weak evidence for this difference.

3.4. Effectiveness Step 2: Perspectives of Program Trainers and Male Adolescent Participants

The following section describes the results of interviews before and after MaM with two trainers and four adolescent participants on expected and experienced program outcomes.

3.4.1. Program Trainers

Before MaM, both program trainers thought adolescents would learn about sexual boundaries, communicating about sex, differences and similarities between boys and girls, and that “everybody is different”. Other topics were how sex and relationships could be something beautiful and positive, differences between friendships and romantic relationships, anatomy, and peer pressure.

After MaM, both trainers perceived that, through MaM, adolescents had started to think about sexuality and romantic relationships. Yet both trainers did not think that all adolescents thought differently about love, sexuality, and intimacy after MaM. One trainer thought, ‘Some will, some will not. But I think, secretly, look, they don’t express it, but I secretly think they do start thinking differently sometimes […] or at least form an opinion on what is important to them’. Both trainers did perceive that, possibly, adolescents would have learned about indicating and communicating sexual boundaries:

‘I think a lot can go wrong in communication. I think they really started to think about that, that you can’t just assume that someone wants to [have sex] or assume that someone doesn’t […] and that you’re like: Well, let’s go! You know? That they are much more aware of that. […] that it has planted a seed. That is the least, I hope, yes’.

However, trainers were hesitant in their perceptions of how MaM resulted in behavioral change, at least in the short term:

‘Whether they will immediately act on it, that is often not the case. But it is clear to them. […] The information often reaches them, but directly acting on it is difficult. And I understand that. […] For some [I see behavioral changes] very clearly. And some just find that difficult’.

The other trainer perceived that the adolescents understood MaM’s content regarding desires and boundaries, but

‘I always say: what you like, doesn’t have to be the same for another person. And having that conversation is very important, and don’t just do something. So, they know, but they also say that they find it difficult to interpret another person’s signals’.

Moreover, regarding peer pressure, one trainer mentioned that adolescents often deny giving in to peer pressure, but she saw otherwise. She also felt that the possibility of discussing peer pressure depended on group composition, as adolescents needed to feel safe enough amongst each other to admit that they sometimes would give in to peer pressure.

3.4.2. Male Adolescents

Program Outcomes: Before MaM, we asked adolescents what they wanted or expected to learn. Three out of four said they would see what the program would bring: ‘I don’t know exactly what I will get. I guess I’ll just let it come over me’. And they were curious to learn more: ‘I just want to get to know as much as possible’. Two adolescents did not expect to learn anything, and indicated they already had a lot of knowledge: ‘I think I know everything. […]. I just have the knowledge, I think. But I’ll see’. Other expectations were to learn more about recognizing and indicating sexual boundaries: ‘Because imagine I would be with a girl and she does not want to, I must indicate that. And right now, I don’t know how to do that. So, I can learn that’.

After MaM, three male adolescents perceived that they had learned some new things, and one participant perceived he did not learn anything, insisting that he already knew everything. Two adolescents perceived they had learned “enough”. When prompting them to be more specific about what they learned, they perceived that they learned that respecting sexual boundaries is important and that they had learned to talk more openly about sex, one with friends and the other with a sexual partner: ‘That it’s just normal to talk about it, I have learned that’. Other topics mentioned were sex toys, sexually transmitted infections, how to not make a woman bleed during sex, and contraception. Except one adolescent who perceived he learned from other adolescents’ experiences during MaM, none mentioned (a perceived change in) how they believed others thought about sex (i.e., social norms), or dealing with peer pressure. One adolescent said he did not learn enough about romantic relationships. Regarding changing their behavior after MaM, most adolescents said they would not use what they had learned: ‘Not really, I already knew everything, or most of it’. Another said, ‘I’ll just do it my own way’.

Changes in Beliefs About Sex: Upon asking adolescents before the program what they found important during sex regarding their own and/or their partner’s behavior, often rephrased into what adolescents considered “good” sex and “bad” sex, two did not know what “good sex” entailed. Others mentioned something about pleasure for both: ‘Just, that you both enjoy it. And that for both of you it is fine what you are doing. And of course, not too short’. Regarding bad sex, one did not know, and others said it was when (at least) one person did not want to have sex. One mentioned that bad sex was when you hurt the girl.

After MaM, all four had an answer to what good sex was. Three mentioned both partners—both want to have sex, both indicate boundaries, and both enjoy it:

‘Well first you would have to communicate with each other, what you like and don’t like. That is important for sure. And otherwise just prepare a bit. […] Mentally, of course, but also physically and then I mean, for example, have condoms or a pill for the woman. […] Just respect each other’s wishes and boundaries, as well. […] What do I like? What does she like? What can I do, what can I not do?’

Regarding bad sex, adolescents mentioned it was when only one person wanted to or enjoyed it, when you had sex with someone you do not have feelings for, when you did not indicate boundaries, or when you had sex too soon. They perceived that their beliefs about love and sex had not really changed after the program: ‘No, no, no. […] I don’t know, it just stayed the same’. This contrasts with the fact that three out of four adolescents perceived themselves as having learned that it is important to know, indicate, and/or respect sexual boundaries and to communicate with a sexual partner, which does indicate some change in attitudes, awareness and/or intentions regarding SDV.

Sexual and Relational Interaction Competency: Regarding confidence in flirting/dating before the program, adolescents rated themselves as 3, 6, and 9 out of 10, and one reported ‘I don’t know’. After the program, all adolescents rated themselves the same or higher, with 7, 6, 10, and 7, respectively. The adolescent whose opinion changed from ‘I don’t know’ to a 7, indicated that MaM had contributed a little to his increased confidence.

Regarding adolescents’ confidence in sex, two gave themselves a 7, one an 8.4 out of 10, and one, the same as with flirting/dating, reported: ‘I don’t know’. Those who gave their confidence regarding sex themselves a 7, improved to an 8 after MaM. Both reported that this was due to MaM: ‘The more I know about something, the more confident I get’, and ‘At first, I was always scared to say the wrong thing. And now after the sessions, it does give me some more confidence’. The one who rated himself an 8.4 before MaM retained his grade, as ‘he did not learn anything new’. The one who did not know what to rate his confidence in sex before MaM gave it a 7, ‘I don’t know everything yet. But it’s not nothing for sure’.

3.5. Effectiveness Step 3: Multi-Method Case Descriptions

The following section describes how three male adolescents who participated in MaM and every part of the study (i.e., questionnaires, interviews, and observations) responded before and after, and how they experienced the program. Names are pseudonyms. Plots of the three adolescents’ program outcomes at the three timepoints are presented in Supplement S2.

3.5.1. Sam’s Case Description

Sam identified as fully Dutch and heterosexual. At the time of the study, he was 15 years old, lived with his mother, and was in secondary vocational education. Sam scored in the clinical range on psychosocial adjustment problems; specifically on hyperactivity and conduct problems, he scored higher than 90% of his peers, and he scored higher than 80% of his peers but lower than the upper 10% on problems with peers (Vugteveen et al., 2022).

He had experience with being in love and romantic relationships but was not in a relationship during the study. Sam had experienced French kissing, naked touching, manual and oral sex, and vaginal intercourse. He received his first kiss at 6 years old and had his first sexual experiences (naked touching and vaginal intercourse) at 15 years old. He had received sexual education before and rated this positively (7 out of 10). Before the program, Sam scored high on positive attitudes and intentions, relatively high on self-efficacy, and low on negative attitudes, negative intentions, and social norms approving of SDV.

Sam did not have any learning goals for MaM because he had just experienced a breakup and ‘would rather focus on school than love’. Nevertheless, he said he thought learning about sex was important, because ‘kids started having sex very early’. He said he was open to learning about sex and would ‘go with the flow’. His understanding of good sex was when both people had fun, and his understanding of bad sex was when one person did not want to. He rated his confidence in sex prior to MaM a 7 out of 10, because he did not have much experience yet. He rated his confidence in dating/flirting a three, because he was afraid that he would say something wrong and usually got too nervous.

Sam missed two out of eight sessions, namely Sessions 2, ‘Friends’, and 6, ‘Dating’; see Supplement S1. During the program, he had an overall calm demeanor and participated actively most of the time. From the observations, it seemed he was knowledgeable about sex and thus served as an example for the other adolescents. They seemed to look to him for correct answers and were curious about his experiences. In the post-interview, he stated he had learned new things but could not specify what exactly.

From pre- to post-test, he remained relatively stable on most questionnaire outcomes, although his initially positive sexual attitudes decreased a bit, and his adversarial sexual beliefs increased from generally disagreeing to agreeing with the statements. This was also indicated in our observations, where he sometimes uttered negative attitudes toward girls (e.g., in the last two meetings he wondered why it is a boy’s responsibility to wear a condom when women can also wear a female condom, and said he believed it is worse when a girl cheats than when a boy cheats).

From the pre- to post interview, his confidence in flirting/dating increased from 3 to 7 and in sex from 7 to 8 out of 10, and he perceived that MaM had contributed to this a bit because he became more comfortable talking about sex. In contrast, his questionnaire scores showed that both his attitude and self-efficacy for communication with a sexual partner had decreased. Similar to the pre-interview, he said his confidence in flirting/dating was lower than in sex, because he was afraid to say the wrong things. Also, he said that MaM had not helped in his confidence in flirting/dating, as he missed the sessions on these topics.

In summary, Sam was knowledgeable and confident about sex before MaM but felt nervous about flirting/dating. He entered MaM with a neutral attitude toward the program and participated actively. Afterwards, Sam perceived that he learned more about sex from MaM but thought that his increased confidence in flirting/dating was not because of MaM, because he missed the sessions on the topic. His scores on MaM’s intended outcomes, which initially aligned with the MaM goals, mostly remained stable, but some had slightly deteriorated after MaM. Sam’s case description highlights the significance of repeating core program content, as he missed one session on the topic he was most interested in learning about, and consequently—in his own perception—did not obtain that program goal.

3.5.2. Marc’s Case Description

Marc identified as fully Dutch and heterosexual. At the time of the study, he was 15 years old, lived with both parents, and was in secondary vocational education. He scored in the clinical range on psychosocial adjustment problems overall and on all subscales, except Problems with Peers, on which he scored in the borderline range (Vugteveen et al., 2022). He had experience with romantic relationships and being in love but was not currently in a relationship. He had experience with French kissing and masturbation but had no sexual experience with another person. He had received sex education before and rated this experience positively (7 out of 10). In the interview, Marc indicated he did not want to learn anything specific in MaM and would just see how things would go. He also said he already knew everything. At the same time, when asking what he believed “good” and “bad” sex was, he said he did not know. Similarly, he did not know how to rate his confidence in flirting/dating or sex. Overall, Marc scored relatively neutral on most outcomes during the pre-test, sometimes slightly more toward a negative attitude (e.g., on communication attitudes); see Supplementary Materials.

Marc attended all eight MaM sessions. During MaM, he often showed hyperactive, distracted, and disturbing behavior. At the start, his trainers mentioned to the observing researcher that they felt Marc had a negative attitude toward the program. During Sessions 3 and 4, he seemed to participate more actively, but toward the end he was often distracted and unmotivated.

Over time, Marc seemed to show positive changes on some outcomes—increased attitudes toward communication, self-efficacy for self-regulation and communication, and peer pressure resilience and decreased negative intentions following sexual rejection. Specifically, the changes in self-efficacy and communication attitudes seemed to be substantiated by his interview responses. During the pre-test, he could not rate his confidence in flirting/dating and sex, whereas after MaM he gave both aspects a 7 out of 10, and he perceived that MaM had contributed a little to this change. The decrease in negative intentions following sexual rejection might be reflected in that before MaM he was unable to answer what he thought “bad” and “good” sex was, whereas after he perceived to have learned that setting boundaries was important. He said this both in the final session, as shown in the observations, and in the post-MaM interview. Moreover, we observed that he often seemed to correctly recognize when boundaries were being crossed. However, his decreased self-efficacy to “say no” himself in the questionnaires seemingly contradicts this.

Regarding negative attitudes such as rape myth acceptance and adherence to heterosexual double standards, he showed a slight dip at post-test, but these attitudes returned to the neutral pre-test levels at follow-up. This is in line with his response in the interview, in which he said that he did not perceive a change in how he thought about sex after the program. Also, we observed that he regularly made sexist jokes or comments throughout the sessions. For example, during a true/not true exercise in Session 3, he yelled “whore” in response to the statement “A girl with a lot of sexual experience is cool, just like a boy”. His increased resilience to peer pressure on the questionnaire was not evident from other sources. From the observations, it even appeared that, in general, he seemed to be influenced by the other adolescents, joking around and posturing.

In summary, Marc scored relatively neutral across all pre-test scales and self-proclaimed he already knew everything about sex prior to MaM. He attended the full program but regularly showed low participation. After MaM, Marc seemed to have improved on multiple outcomes, even though he could not entirely word this himself in the interviews. Marc’s case indicates that despite resistance and low participation and motivation, improvements on the program outcomes and thus benefits from participating may still appear (e.g., based on some of his post-test and follow-up scores). Marc’s case also shows that there may be a difference between quantitatively measured and qualitatively measured effects, pointing to the importance of combining various methods to gain a holistic picture.

3.5.3. Alex’s Case Description

Alex identified as fully Dutch and heterosexual. At the time of the study, he was 14 years old, lived with his mother, and was in secondary vocational education (Dutch: vmbo-kader). He scored in the clinical range regarding psychosocial adjustment problems overall, and specifically on conduct problems and hyperactivity (Vugteveen et al., 2022). Alex had experience with romantic relationships and being in love but was not currently in a relationship. He had experienced French kissing and masturbation but had no sexual experience with another person. Alex had received sex education before and rated it negatively (5 out of 10).

Prior to MaM, Alex was curious to learn about love and sex in general but did not know what exactly. He thought participation in MaM would be fine because the sessions were not too long. He thought “good sex” was when everyone involved liked it; he thought “bad sex:” was when one person did not. He rated his confidence for flirting/dating a 6 out of 10, saying he usually does not take it seriously and makes jokes about it. He gave his confidence in sex “a 7 or 8”. He said he did not really know why, it was just a feeling, and that this was also something he might still want to learn during MaM. Overall, at pre-test, Alex scored high on positive attitudes and intentions, relatively high on self-efficacy, and low on negative attitudes, negative intentions, and social norms approving of SDV.

Alex attended the first five MaM sessions. During the sessions, he sometimes participated seriously but mostly seemed to have a difficult time concentrating and staying focused. For instance, he volunteered to referee in Session 1 but complained that the group drew up too many rules. One inclusion criterium for MaM is that participating adolescents need to be able to function within a group setting. This was not the case for Alex. During the first four sessions, he required much attention from the trainers to help him participate and stay focused. Alex regularly disturbed the sessions with unrelated/provoking comments or by joking around with or physically bothering the other adolescents (e.g., shoving, hanging by their arms). The trainers explained to the observing researcher that Alex had ADHD.

In Session 4, the trainers asked Alex to leave the room, which seemed to positively affect the other adolescents’ participation. During Session 5, they also asked him to leave the room early during the session, and he did not return. Alex was absent during Session 6, and from Session 7 onwards the trainers decided to exclude Alex from MaM due to his disruptive presence. The other male adolescents did not object. As such, Alex completed less than half of MaM. His exclusion was not communicated clearly to him and thus came as a negative surprise. He subsequently also refused to participate further in the study, so we have no post-program questionnaires or interview data from him.

In summary, Alex was curious to learn about love and sex but had trouble concentrating and disrupted the MaM sessions. This led to him being excluded from more than half of the program and dropping out of the study. Alex’s case indicates that despite an initial willingness to learn about the topics of MaM, individual psychosocial functioning may obstruct the ability to benefit from a group counseling program. This indicates the importance of the initial intake phase and inclusion criteria to gauge adolescents’ possible participation.

4. Discussion

The aim of this study was to evaluate the effectiveness of the Dutch group counseling program Make a Move—aimed at sexual health promotion and SDV prevention for male adolescents in vocational education and youth care. Consistent with current best practices for dealing with the challenges of effectiveness evaluations of complex programs (i.e., with multiple outcomes, implemented in various contexts, with heterogeneous target groups) in real-world settings, we triangulated multi-method and multi-informant data from male adolescent participants (questionnaires, interviews), trainers (questionnaires, interviews), and researchers (observations), collected in a quasi-experimental design.

4.1. Evidence for Program Effectiveness

The first aim of this study was to evaluate the effectiveness of the MaM program on six outcome domains. In this study, we found evidence for program effectiveness on adolescents’ skills but mixed or no evidence for program effectiveness on other domains.

First, we found evidence for MaM’s effectiveness on skills. After participating in MaM, self-report questionnaires indicated that male adolescents became more resilient to peer pressure compared to the control group, and, in the interviews, they reported increased romantic and sexual interaction competency skills. Second, our findings regarding attitudes were mixed. We found evidence for effectiveness on decreased rape myth acceptance after MaM from the self-report questionnaires but not on other attitudes. Also, in the interviews, male adolescents and trainers perceived that adolescents’ attitudes surrounding sexuality had not changed due to MaM. Effects on self-efficacy and intentions did not become apparent from the self-report questionnaires, but there were some indications that effects might have appeared in the interviews, which may warrant future investigation. Regarding self-efficacy, we found no evidence for effectiveness on self-efficacy from the self-report questionnaires. Yet in the interviews, male adolescents perceived themselves as having stronger skills for communicating with a sexual partner and indicating their own and respecting someone else’s (sexual) boundaries. Regarding intentions, we found no evidence for effectiveness from the self-report questionnaires. Yet in the interviews, we noticed a shift in male adolescents’ expression of the importance of respecting sexual boundaries. Third, we found no evidence for effectiveness on social norms or SDV perpetration. Fourth, we found indications of some adverse effects on attitudes and intentions. In the self-report questionnaires, male adolescents increased in their adversarial sexual beliefs after MaM. Also, male adolescents’ self-reported positive intentions following sexual rejection had decreased after MaM.

Looking thematically across the six intended outcome domains, we could say that we found some evidence for effectiveness on the themes “communication with a (sexual) partner” and “indicating and respecting (sexual) boundaries” but no evidence for effectiveness on aspects of improved gender-equal attitudes or social norms, and no effects on SDV perpetration. Hereafter, we discuss possible explanations for these findings.

4.1.1. Effects on (Inter)Personal Versus Sociocultural Factors

A possible explanation for (perceived) program effects on outcomes related to communication with a partner and indicating and respecting (sexual) boundaries may be that these outcomes most closely match the programs’ content and delivery method. A large proportion of the program is allocated to pleasurable sex and indicating and respecting sexual boundaries and resilience to peer pressure through group discussions, facilitated through statements or video vignettes that are then discussed. As such, the program largely consists of discussing communication and boundaries in sex, relationships, and friendships (i.e., peer pressure), which may have increased adolescents’ self-perceived skills concerning these themes and the realization of their importance. A second explanation may be that MaM possibly has the most effect on factors that are most proximal to the individual and interactional (i.e., between-person) level. Our study, as well as several studies on comparable programs, indicate that, possibly, individual-level factors such as skills may be more easily changed than broader societal social norms and the (gender-related) attitudes that are influenced by them (E. Miller et al., 2012, 2020).

Relatedly, one explanation for why we found no evidence for effectiveness on social norms and some attitudes is that researchers suggest that social norms and attitudes regarding gender equality are embedded in and reproduced at larger ecological levels (i.e., on community, structural, and societal levels, Brush & Miller, 2019). Therefore, these factors should also be targeted at these broader levels and on a larger (time)scale to produce meaningful change (Dworkin & Barker, 2019; Ruane-McAteer et al., 2020). Individually focused SDV prevention programs may even backfire when participating adolescents feel that they “need to bear individual responsibility” for these massive social problems (Dworkin et al., 2015). This may also explain the slightly negative attitudes some participating adolescents showed before and during the program (e.g., as trainers indicated with adolescent Marc). Similarly, that male adolescents denied thinking differently after the program and the fact that male adolescents’ positive intentions and negative attitudes toward girls and women seemed to have deteriorated after MaM also support this possibility. These issues may (in part) be resolved by changing Make a Move’s gender-sensitive approach (i.e., recognizing gender differences but not challenging them) to using a gender-transformative approach (Brush & Miller, 2019; Ricardo & Verani, 2010). Research suggests that gender-transformative approaches that actively challenge masculine norms and promote vulnerability can decrease some of male adolescents’ experienced gender-related barriers to actively participate in and benefit from these programs (Dworkin & Barker, 2019; Ruane-McAteer et al., 2020).

Other possible explanations are more closely related to our sample and program integrity. In our observations, obtaining a new positive group norm sometimes appeared difficult to accomplish for trainers, as adolescents continuously made negative remarks or sexist jokes that trainers did not always adequately address. Also, adolescents already scored very low on social norms conductive to SDV at pre-test, and these scores remained stable over time, making it difficult to observe changes (i.e., floor effects).

4.1.2. Effects on Behavior

A possible explanation for not finding evidence for effectiveness on SDV perpetration was indicated by the trainers in our study, mentioning that adolescents had become more aware of boundaries but that they did not expect to see short-term program effects on behavior. As such, our 3-month follow-up might have been too short to find effects on behavior (Fellmeth et al., 2015). Indeed, most studies that did find effects of programs like MaM on behavior had follow-ups of 12 months or longer (for a review, see Verbeek et al., 2023b). Another possible explanation for a lack of effects on SDV perpetration and some of the gender-equal attitudes and social norms that researchers are recently exploring (Brush & Miller, 2019; Orchowski, 2019) is that SDV prevention programs (or their outcome evaluations) tend to focus solely on males as (possible) perpetrators. As such, they ignore that boys and men also experience violence victimization (J. A. Miller et al., 2021; Reidy et al., 2017; Verbeek et al., 2023b). As for both physical and online SDV, victimization is a well-established risk factor for perpetration (Curtis et al., 2023; Van den Eynde et al., 2023); SDV perpetration programs that do address male adolescents’ victimization are promising (see Orchowski, 2019, for some examples).

4.2. Program Integrity

The second aim of this study was to interpret the evidence for program effectiveness in the context of program integrity. First, analysis revealed that program integrity was compromised, with lower-than-intended levels of dosage and adherence. Consequently, with six intended main outcomes and fifteen specific change objectives, it is improbable that each topic received adequate attention. This was also illustrated by male adolescent Sam mentioning that he did not learn about flirting/dating because he missed the sessions on this topic. This may explain why we found effects mostly on topics seeming to be core program elements (i.e., repeated often and therefore received more by adolescents).

Second, considering quality of delivery, we observed that (mostly untrained) trainers did not always transfer the message that MaM intended to bring across during certain exercises. This is important because human relationships and sexuality, and SDV specifically, are inherently sensitive themes. As such, adolescent SDV prevention programs might also do more harm than good if not properly supervised and facilitated (Flood, 2020; Lyndon et al., 2011). Some research suggests that trainers may underestimate how their own behavior and attitudes may affect the adolescents they work with (Jaime et al., 2015). However, more research on the effects of trainers on SDV prevention programs is needed. Subsequently, creating awareness of the norms that trainers may be perpetuating is an important task for program developers when certifying their trainers (Clark, 2017).

Third, participant responsivity influences program reception (Giannotta et al., 2019; Hutchison et al., 2022). Over half of the adolescents in the intervention group reported and displayed psychosocial problems, including inattentive, hyperactive, and disruptive behavior during the program sessions. This indicates that MaM is being implemented among a population of adoelscents with various developmental and behavioral problems, which complicates implementation for trainers and may limit possible uptake for adolescents themselves, for instance if their lived reality is too far from MaM’s ultimately intended outcomes (Ball et al., 2015). In another study of a similar program for high-risk adolescents, trainers also expected that “to have planted a seed” was the most attainable outcome in the short term (Ball et al., 2015). Additionally, our person-centered approach highlighted considerable variability in participants’ experiences and outcomes, suggesting that program effectiveness on the intended outcomes may not be universal. In future research, similar approaches may further inform how to tailor programs effectively to individual needs.

4.3. Limitations

The limitations of the current study are that, first, we had to deviate from the intended rigorous evaluation design (a cluster RCT) to a quasi-experimental study. Consequently, the intervention and control group were not fully equal and comparable on background variables such as age, education level, living situation, and program outcomes at baseline (i.e., intentions following sexual rejection). This limits the groups’ comparability for program effects, although it is unclear in which direction any bias may have occurred. Second, our sample size was smaller than intended due to a small initial population to sample from and issues in recruitment and retention. In addition, the sample size limited power for finding statistical evidence for observed differences between the intervention and control groups, of effects mentioned in the interviews, and of effects observed in the program sessions. In the future, sustainable collaboration between researchers, program users, and program developers throughout the program development cycle may mitigate some of these issues (Crooks et al., 2019). Similar to other research on SDV prevention, our study possibly contained common biases as well, including selection bias (e.g., adolescents already scoring worse on the intended outcomes and being less willing to participate in the study) and self-report questionnaires, possibly resulting in the underreporting of sensitive topics (e.g., see E. Miller et al., 2020). Limitations notwithstanding, the multi-method, multi-informant design of this study, together with the dual focus on program effectiveness and integrity, allowed for a nuanced assessment that provides valuable insights into the practical impact and integrity of the MaM program in a real-world setting.

4.4. Conclusions

The implementation and evaluation of programs such as MaM in a real-world setting with this specific target group comes with many challenges (Crooks et al., 2019; Hein & Weeland, 2019). Overall, the MaM program was positively evaluated by trainers and male adolescents. We found evidence that in its current form and implementation, in the short-term, MaM is effective in improving skills, possibly effective in changing attitudes, but not in affecting social norms or SDV perpetration. Moreover, following the analyses of the interviews, MaM seems effective in improving self-efficacy and intentions. This indicated areas where the program may be enhanced and further evaluated, with different methods and larger samples. Yet, considering we could only quantitatively confirm our hypothesis for two of the fifteen intended program outcomes and only on one of the two timepoints, we found limited evidence for effectiveness. These insights underscore the importance of aligning program content closely with participant experiences and highlight the need for SDV prevention programs to address broader socio-cultural factors in addition to individual skill-building to achieve more substantial changes in SDV-related attitudes, social norms, and behaviors. Together, these findings add to a growing body of literature about effective ways to promote psychosexual health and prevent SDV for male adolescents, whose specific challenges and needs are relatively understudied and underserved. Continuous collaboration between researchers, program developers, and users is crucial for tailoring programs like Make a Move to the specific needs of male adolescents, optimizing both their integrity and effectiveness. When tailored and delivered with high integrity, such programs can support healthy, safe, and positive sexual development for male adolescents and their (future) partners.