Evaluation of a Tobacco Treatment Specialist Training Program Targeted to Behavioral Health Professionals

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

ijerph-4209665 

This paper reports the results of an online, asynchronous, tobacco treatment program delivered among providers and staff working at behavioral health clinics in Kentucky. Given the high prevalence of tobacco smoking among people experiencing mental health challenges, this topic is of public health importance. 

Intro 

I believe it would be helpful to add 1-2 sentences specifying why a “targeted” training for behavioral health professionals is necessary. This is, the authors jump from mentioning the underrepresentation of these providers in currently accredited tobacco treatment training programs to the evaluation of the outcomes of their program. In between these statements, it seems necessary to add why a “targeted” program. Is it the rationale that the tailored training would increase behavioral health providers engagement in the actual training or that they need a specific content? Both? Something else? Added statements may be included in lines 89-92. 

Materials and Methods 

• Line 115- what means that the supplemental resources are tailored to participant needs? An example may be useful 

• Line 135- what was the eligibility criteria? What would constitute ineligibility? 

• The rationale for this targeted training is built on the argument that is needed for behavioral health providers but the demographics indicate that participants could represent a wide range of providers and administrative staff... Given the claimed focus on behavioral health providers, the current categorization is a missed opportunity to report data specific to these providers. So, I think authors should, either, re-categorize the sample to highlight the behavioral health providers and re-run analyses based on that; or edit the intro to focus on behavioral health care settings (and not providers) 

• Related to the process, was it possible for providers to choose the order in which they engage with the materials, or the modules were designed to follow a specific order? 

• Sentence in line 152-154 seems to belong to the next paragraph about knowledge and attitudes 

• Program engagement—partial completion would also be useful info. Among those who didn’t complete the program, how much did they complete? To be labeled as completer, the person had to complete the 1-hour synchronous interaction with simulated patient? Did everyone reaching that step passed? 

• Minor: TTS was spelled out in line 110, but was already mentioned in line 102.  

• How, if at all, the research staff maximized that participants answers to quizzes were reliable and a true reflection of their knowledge (vs group work, use of AI, etc.) 

Results 

• Since only 11% worked in SUD treatment facilities, it does not seem appropriate that this frequency is included in a statement that is listing how “most” of the sample was distributed. 

• There was a great variability in the time to complete the program. So, was there any additional efforts required to motivate people to complete the program or follow-through (calls, emails)? 

• How many participants in the non-medical were administrative staff? I wonder if the low engagement/completion could be due in part to a reduced follow-up from these individuals. 

Discussion 

• The attitudes measure was a combination of perceived knowledge, confidence and available time. This operationalization may or may not be similar to other programs’ measurement of “attitudes”. This is worth acknowledging. 

• Given we are living in an Artificial Intelligence era and there is always a chance participants of online classes/trainings access AI tools during assessments, it would be useful the authors acknowledge their approach to this threat during their program and any recommendations for future research and implementation.  

Author Response

Intro

Comment 1: I believe it would be helpful to add 1-2 sentences specifying why a “targeted” training for behavioral health professionals is necessary. This is, the authors jump from mentioning the underrepresentation of these providers in currently accredited tobacco treatment training programs to the evaluation of the outcomes of their program. In between these statements, it seems necessary to add why a “targeted” program. Is it the rationale that the tailored training would increase behavioral health providers engagement in the actual training or that they need a specific content? Both? Something else? Added statements may be included in lines 89-92. 

Response 1: Thank you for this suggestion. We have added clarifying language to strengthen the rationale for targeted training.

Materials and Methods 

Comment 2: Line 115- what means that the supplemental resources are tailored to participant needs? An example may be useful 

Response 2: We have clarified this statement to specify how materials were tailored.

Comment 3: Line 135- what was the eligibility criteria? What would constitute ineligibility? 

Response 3: We have added a detailed description of eligibility criteria.

Comment 4: The rationale for this targeted training is built on the argument that is needed for behavioral health providers but the demographics indicate that participants could represent a wide range of providers and administrative staff... Given the claimed focus on behavioral health providers, the current categorization is a missed opportunity to report data specific to these providers. So, I think authors should, either, re-categorize the sample to highlight the behavioral health providers and re-run analyses based on that; or edit the intro to focus on behavioral health care settings (and not providers)

Response 4: Thank you for this observation. We agree that clarification of the sample composition strengthens the manuscript. To address this, we have revised the manuscript to more clearly describe the sample as an interdisciplinary behavioral health workforce, rather than focusing narrowly on a single provider type. Specifically, participants were categorized into medical and non-medical roles, but all participants were professionals working within behavioral health or related care settings. Medical roles included nurses (RN, LPN), nurse practitioners, advanced practice providers, respiratory therapists, and pharmacists, while non-medical roles included mental health counselors, substance use counselors, social workers, health educators, and public health professionals. This revised framing better reflects the team-based nature of behavioral health care delivery, where both medical and non-medical providers play critical roles in tobacco treatment.

Comment 5: Related to the process, was it possible for providers to choose the order in which they engage with the materials, or the modules were designed to follow a specific order? 

Response 5: We have clarified module sequencing.

Comment 6: Sentence in line 152-154 seems to belong to the next paragraph about knowledge and attitudes 

Response 6: We have corrected the formatting issue.

Comment 7: Program engagement—partial completion would also be useful info. Among those who didn’t complete the program, how much did they complete? To be labeled as completer, the person had to complete the 1-hour synchronous interaction with simulated patient? Did everyone reaching that step passed? 

Response 7: We have addressed this to the extent possible given available data.

Comment 8: Minor: TTS was spelled out in line 110, but was already mentioned in line 102. 

Response 8:  Thank you for this observation. This has been revised throughout the manuscript (i.e., highlights, abstract, and main body of manuscript) to ensure an initial spelled out Tobacco Treatment Specialist, followed by abbreviated TTS

Comment 9: How, if at all, the research staff maximized that participants answers to quizzes were reliable and a true reflection of their knowledge (vs group work, use of AI, etc.) 

Response 9: Thank you for raising this point. Objective verification of independent quiz completion (e.g., proctoring, AI detection) was not available within the platform at the time of the study. At the time of implementation, the training platform did not include proctoring or monitoring features to verify independent completion.

Results 

Comment 10: Since only 11% worked in SUD treatment facilities, it does not seem appropriate that this frequency is included in a statement that is listing how “most” of the sample was distributed. 

Response 10: We agree and have revised the wording to avoid mischaracterizing the distribution of the sample.

Comment 11: There was a great variability in the time to complete the program. So, was there any additional efforts required to motivate people to complete the program or follow-through (calls, emails)? 

Response 11: We have clarified participant follow-up procedures. Engagement reminders were limited and primarily consisted of periodic email communications with the individual to determine if there were any challenges to completion and to encourage module completion. Individuals requesting additional time for completion were allotted time as needed.

Comment 12: How many participants in the non-medical were administrative staff? I wonder if the low engagement/completion could be due in part to a reduced follow-up from these individuals. 

Response 12: Thank you for this question. Administrative roles were not separately categorized within the non-medical group in the original dataset, limiting our ability to examine this subgroup. This has been acknowledged as a limitation.

Discussion 

Comment 13: The attitudes measure was a combination of perceived knowledge, confidence and available time. This operationalization may or may not be similar to other programs’ measurement of “attitudes”. This is worth acknowledging. 

Response 13: We agree and have clarified this limitation.

Comment 14: Given we are living in an Artificial Intelligence era and there is always a chance participants of online classes/trainings access AI tools during assessments, it would be useful the authors acknowledge their approach to this threat during their program and any recommendations for future research and implementation.  

Response 14: We have added discussion acknowledging the evolving role of artificial intelligence and its implications for training evaluation.

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review this manuscript on evaluating an asynchronous tobacco treatment specialists training program for behavioral health providers. The manuscript has several strengths, including addressing tobacco use in a state that has a high prevalence through TTS training models. Below, I will highlight several areas of clarification or improvement.

Highlights

• In public health relevance you might want to introduce TTS as a possible training option for behavioral health providers
• Then in public health significance you can add that an asynchronous TTS training program

Abstract

• I think you can highlight the gap a bit more: You might consider defining what a tobacco treatment specialist training program – even a few words might suffice and what it means to deliver it asynchronously. Maybe a sentence on the fact that in person synchronous trainings can be a barrier for some healthcare workforces, and that offering an asynchronous option for the TTS training may reduce such barriers. You had the opportunity to evaluate a targeted TTS training program delivery asynchronously and you evaluated….
• Could you share how many registered and how many ultimately enrolled? Of the xx registrants, xxx enrolled. How was the program announced and made available to attendees.

General clarification: All TTS trainings have a 10 hour asynchronous learning requirement prior to joining the synchronous TTS training. It is unclear if that is what you mean or if you mean that the entire TTS training was asynchronously delivered. Maybe worth clarifying.

Introduction

In paragraph 3 of the introduction, I would add a bit more background on the tobacco treatment training program and their modalities of delivery. Generally they tend to be in person and require an asynchronous portion, but you may want to provide information on how the asynchronous online version is different and how it might appeal to certain audiences. You may also comment on the fact that synchronous version have a high rate of completion because people are required to attend and whether there are concerns about that with asynchronous. Some background on this topic could be helpful.

Perhaps also add background on how much improvement you are likely to get in providers providing treatment if they engaged in TTS training. This will support the need to invest in these trainings for behavioral health providers.

Methods

Were you able to reach saturation in the community mental health clinics and behavioral health organizations that you reached and how were those organizations selected

The 26 clinics and health programs represents what proportion of the total number of health programs or clinics in the region or all of KY?

What were the eligibility criteria for participating in the program? Did they need to have access to their own devide/computer to participate or did their clinic provide one? What other support did their leadership provide to support their completion.

Who paid for the program?

Results

I am interested by the finding that most participants were female, white, and masters educated. These participants were referred by their leadership and I wonder if you can address implications of who is being selected for trainings/leadership/professional development by behavioral health organizations and whether that group adequately represent the clients who are smoking.

Discussion

One of your explanations for lower-than-expected completion rates was that organizational commitment may be lacking but your recruitment strategy involved gaining buy-in from the organizational leadership of the organizations that the providers were recruited from. Isn’t that a metric of organizational support? How is this different that what other organizations offered their providers?

Your second explanation was workforce compositions but I am wondering if all of your comparison TTS programs were online asynchronous TTS programs or some were in person. Could you comment on this?

I wonder if you can comment on whether the online training module, although it improved accessibility, could have made it harder to complete on your own without the external reinforcement of a classroom culture. Moreover, could digital accessibility issues be addressed as a reason for not completing if you have that data.

Perhaps ending with a statement on the need for behavioral health workforce capacity building for tobacco use is needed in this population. 

Author Response

Highlights

Comment 1: In public health relevance, you might want to introduce TTS as a possible training option for behavioral health providers

Response 1: Thank you for this suggestion. We have revised the highlights to explicitly introduce TTS training as an evidence-based option for behavioral health providers.

Comment 2: Then, in public health significance, you can add that an asynchronous TTS training program

Response 2: Thank you. We have revised the Highlights to emphasize the public health relevance of asynchronous delivery as follows.

Abstract

Comment 3: I think you can highlight the gap a bit more: You might consider defining what a tobacco treatment specialist training program – even a few words might suffice, and what it means to deliver it asynchronously. Maybe a sentence on the fact that in-person synchronous trainings can be a barrier for some healthcare workforces, and that offering an asynchronous option for the TTS training may reduce such barriers. You had the opportunity to evaluate a targeted TTS training program delivery asynchronously, and you evaluated….

Response 3: Thank you for these suggestions. We have revised to define TTS training, clarify asynchronous delivery, and highlight barriers in traditional models.

Comment 4: Could you share how many registered and how many ultimately enrolled? Of the xx registrants, xxx enrolled. How was the program announced and made available to attendees?

Comment 4: Thank you for this suggestion. We have clarified in the abstract and manuscript as follows: Of 100 individuals who registered for the program, 30 did not enroll, while 70 enrolled and initiated training.” Additionally, there was no program announcement. Instead, recruitment for this program occurred as described in the manuscript as follows: “The administrative team in each CMHC or behavioral healthcare organization identified and nominated eligible staff members to enroll in the program between December 2019 and April 2024.”

Comment 5: General clarification: All TTS trainings have a 10-hour asynchronous learning requirement before joining the synchronous TTS training. It is unclear if that is what you mean or if you mean that the entire TTS training was asynchronously delivered. Maybe worth clarifying.

Response 5: We have clarified the distinction between the asynchronous training portion of the course and the one-hour synchronous portion of the course (i.e., the simulated patient interaction completed with the course instructor).

Introduction

Comment 6: In paragraph 3 of the introduction, I would add a bit more background on the tobacco treatment training program and its modalities of delivery. Generally, they tend to be in person and require an asynchronous portion, but you may want to provide information on how the asynchronous online version is different and how it might appeal to certain audiences. You may also comment on the fact that synchronous versions have a high rate of completion because people are required to attend, and whether there are concerns about that with asynchronous. Some background on this topic could be helpful.

Response 6: Thank you for this suggestion. We have now addressed this in the introduction, and expanded further in the discussion section.

Comment 7: Perhaps also add background on how much improvement you are likely to get in providers providing treatment if they engaged in TTS training. This will support the need to invest in these trainings for behavioral health providers.

Response 7: Thank you for this comment. We have added a brief sentence in the introduction section showing that TTS trainings improve knowledge, attitudes, confidence, and engagement of healthcare providers in tobacco treatment delivery.

Methods

Comment 8: Were you able to reach saturation in the community mental health clinics and behavioral health organizations that you reached, and how were those organizations selected

Response 8: Saturation was not an objective of this study design. We have clarified recruitment and selection processes.

Comment 9: The 26 clinics and health programs represent what proportion of the total number of health programs or clinics in the region or all of KY?

Response 9: Unfortunately, we cannot provide the proportion of the total number of health programs or clinics in KY that the 26 clinics and health programs represent. However, to administrate mental health services, the state of Kentucky is divided into 14- State funded Community Mental Health Centers covering all regions of Kentucky:  https://dbhdid.ky.gov/cmhc

Based on our purposive sampling design, the participants were derived from every state-funded community mental health center and a few additional behavioral health programs. Thus, the TTS trainees represent all the community mental health regions of Kentucky. We have addressed this issue in the limitation section. 

Comment 10: What were the eligibility criteria for participating in the program? Did they need to have access to their own device/computer to participate, or did their clinic provide one? What other support did their leadership provide to support their completion?

Response 10: These are all excellent points; thank you for the suggestion to address these factors. We have expanded the Methods section accordingly.

Comment 11: Who paid for the program?

Response 11: To clarify, we have included a statement as follows “The training program was provided at no cost to participants, with funding support provided through Kentucky Cabinet for Health and Family Services, Department for Medicaid Services and the Department of Public Health, Tobacco Prevention and Cessation Program.”

Results

Comment 12: I am interested in the finding that most participants were female, white, and master's educated. These participants were referred by their leadership, and I wonder if you can address the implications of who is being selected for trainings/leadership/professional development by behavioral health organizations, and whether that group adequately represents the clients who are smoking.

Response 12: We have expanded the discussion section to include the implications of this finding.

Discussion

Comment 13: One of your explanations for lower-than-expected completion rates was that organizational commitment may be lacking, but your recruitment strategy involved gaining buy-in from the organizational leadership of the organizations from which the providers were recruited. Isn’t that a metric of organizational support? How is this different from what other organizations offered their providers?

Response 13: We have addressed this in the discussion section as follows: “For example, although organizational leadership supported recruitment and participation, this form of support may differ from ongoing structural support (e.g., protected time, workload adjustments) that facilitates program completion. The absence of sustained organizational reinforcement may contribute to lower completion rates in asynchronous training environments.”

Comment 14: Your second explanation was workforce compositions, but I am wondering if all of your comparison TTS programs were online asynchronous TTS programs, or some were in-person. Could you comment on this?

Response 14: Thank you for this suggestion. We agree that training modality is an important contextual factor when comparing completion rates across programs. We have revised the Discussion to clarify that the comparison programs cited (e.g., MD Anderson and TEACH) were delivered online but in synchronous formats, whereas the program evaluated in this study was fully asynchronous. We maintain that workforce composition remains a possible explanatory factor, as programs enrolling predominantly medical providers may benefit from stronger role alignment, institutional expectations, and clinical incentives to complete training. We have clarified this distinction in the revised text.

Comment 15: I wonder if you can comment on whether the online training module, although it improved accessibility, could have made it harder to complete on your own without the external reinforcement of a classroom culture. Moreover, could digital accessibility issues be addressed as a reason for not completing if you have that data.

Response 15: Thank you for these points and suggestions. We have addressed this by expanding in the discussion section: “While asynchronous delivery improves accessibility, it may also reduce accountability and peer engagement typically reinforced in synchronous learning environments. Additionally, variability in digital access and technological resources may have influenced participants’ ability to complete the training, although these factors were not directly measured.”

Comment 16: Perhaps ending with a statement on the need for behavioral health workforce capacity building for tobacco use is needed in this population. 

Response 16: Thank you. We have included this as the concluding statement: “These findings underscore the need for continued investment in behavioral health workforce capacity building to address tobacco use among populations with disproportionately high burden.”

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Authors have satisfactorily addressed most of this reviewers' concerns or questions. 

There seems to be a typo/grammar error in the Discussion section, lines 355-356

Author Response

Thank you for point out, this sentence is repetitive and unclear. So, we have deleted it. 

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for your response. I do not have further comments.