The data for this study were obtained from a type III hybrid effectiveness-implementation trial [21], KIU!, which compared 2 delivery approaches: direct-to-consumer and CBO-based implementation. The study protocol has been discussed elsewhere [22]. This manuscript focuses on the CBO-based implementation approach to recruitment and retention.

KIU! is a CDC-designated best-evidence HIV risk reduction DHI. Previous iterations of KIU! have been discussed elsewhere [23-26]. KIU! was designed with and for young cisgender men who have sex with men (YMSM) aged 18‐29 years, who recently tested HIV-negative. KIU! supports risk reduction and promotes protection practices to maintain a negative HIV status. KIU! includes 3 modules focused on different prevention and risk negotiation content informed by the information-motivation-behavioral (IMB) skills model of HIV risk behavior change [27-29]. These modules are followed by 2 “booster” sessions 3 and 6 weeks after completion of the main intervention sessions. Booster sessions include additional encouragement of regular HIV testing, condom use, and pre-exposure prophylaxis (PrEP) uptake, and information on negotiating condom use with long-term partners.

As the effectiveness of KIU! has been previously demonstrated on behavioral outcomes (eg, condom use and casual anal sex) and biomedical outcomes (eg, sexually transmitted infection [STI] incidence [23,25]), the main aim of KIU! 3.0 was to (1) compare the 2 implementation delivery strategies, (2) examine the effect of strategies and determinants on variability in implementation outcomes using a mixed-methods research approach, and (3) explore the sustainment of KIU! at the completion of the study. This study (KIU! 3.0) was implemented in 22 CBOs located in counties with high HIV rates and large populations of YMSM.

Twenty-two CBOs were selected through a request for proposal process similar to that used by HIV-prevention funders. Thirteen CBOs launched implementation in the fall of 2019, 7 in the summer of 2020, and 2 in the fall of 2020. KIU! 3.0 staff provided virtual training to CBOs on the intervention, provided capacity building support on how best to integrate it into routine HIV testing, and made monthly technical assistance calls. As part of the training, CBO staff completed the KIU! intervention to best describe it to clients during recruitment. CBO staff also received information about project logistics (ie, an online administrative dashboard to track recruitment and retention and manage project reporting via REDCap) and had time to ask any questions related to implementation within their organizations. While all CBOs recruited participants for the same intervention, how participants were recruited and retained varied across CBOs, as no set recruitment or retention plan was provided to them. CBOs were able to decide how to spend their implementation funding, how many staff members to include in the implementation, and whether to provide incentives to YMSM participants.

This paper provides a comprehensive discussion and outline of the diverse range of activities that CBOs undertook while implementing KIU!. Furthermore, the manuscript details insights and lessons learned to enhance future implementation. Data for this paper were collected in several ways from CBOs: applications in response to a request for proposals, monthly standing calls, interviews, and the KIU! 3.0 application dashboard (Table 1). Data collection and analyses are reported in line with the Standards for Reporting Qualitative Research (SRQR; Checklist 1) [30]. While this manuscript uses quantitative data, the entire analysis is qualitative (ie, categorical) and descriptive.

Client volume was divided (ie, annual HIV testing volume) roughly into thirds. CBOs having an annual HIV testing volume of YMSM between 20 and 139 were categorized as low volume. Those with a YMSM HIV testing volume of 140‐339 were categorized as medium volume, and those with a YMSM HIV testing volume of 340 or more were categorized as high volume. Regarding years of HIV service provision, we divided CBOs into 3 categories based on decades of experience. Those with 3‐9 years of experience were categorized as low (n=3), those with 10‐19 years of experience as medium (n=5), and those with 20 or more years of experience as high (n=14). CBO characteristics were compared by client volume and years of HIV service provision.

The analysis plan was not prespecified as the study was not intended to include a statistical analysis detailing whether one recruitment and retention approach was superior or more effective than another. Instead, the aim was to document these various approaches and detail the lessons learned, as CBO recruitment and retention for DHIs are relatively new concepts. This manuscript thus provides a contextual description of CBO approaches to reach YMSM for an HIV-prevention DHI, as a means to disseminate this information for future CBOs implementing KIU! or other DHIs. A report involving complete statistical analysis of the effectiveness and implementation of KIU! is currently under review [33], and the research team has recently published a latent class analysis examining the differences in implementation outcomes across classes of substance abuse [34].

The research study team included 1 research study assistant, 2 investigators with master’s-level experience, and 4 investigators with doctoral-level experience. One investigator had over 10 years of experience working in a local health department (NB), and 2 were experts in qualitative research methods (AZ and JPZ). The research study team predominantly included white individuals and women.

The study protocol was approved by the institutional review board (IRB) at Northwestern University (IRB reference number: STU00207476) [22]. YMSM and CBO staff participants were informed of the aims of the study as well as data protection. All participants provided consent to participate in our trial. No compensation was provided to CBO staff for participating in interviews. Identifiable information about participating organizations has been removed.

The 22 participating CBOs were located within the South, West, Midwest, and Northeast regions of the United States, with 1 outside the continental United States. All CBOs provided HIV testing services and either directly offered or referred individuals to PrEP services. Among the YMSM who accessed HIV and STI services from these CBOs between 2016 and 2019, 43.0% (3336/7759) were white, 22.0% (1707/7759) Black, 10.0% (776/7759) Asian, 9.0% (698/7759) multiracial, and 8.0% (620/7759) American Indian/Alaskan Native. Moreover, 68.0% (5276/7759) of clients were non-Hispanic/Latino, and 32.0% (2483/7759) were Hispanic/Latino. CBOs tested an average of 352.68 YMSM in 2018 (median 241; range 20‐1025).

Of the 22 CBOs, 13 (59%) had prior experience implementing evidence-based interventions (EBIs) such as VOICES/VOCES [35], CLEAR [36], and MPowerment [37]. Moreover, 14 (64%) CBOs had served YMSM for more than 20 years, 5 (23%) had served YMSM for 11‐20 years, and 3 (14%) had served YMSM for 10 or fewer years. While not required to do so, all CBOs offered some form of incentive to participants (eg, cash incentives or free HIV/STI testing). Incentive timing varied by CBO, with 16 (73%) offering incentives at baseline, 18 (82%) offering incentives after all 3 main episodes, and 20 (91%) offering incentives after booster episodes (Multimedia Appendix 1). CBOs, on average, provided a greater proportion of the total intervention incentive after boosters (48.3% of total incentives offered to participants; US $35.91 out of US $72.27), compared with 15.3% (US $11.36 out of US $72.27) at baseline and 33.6% (US $25.00 out of US $72.27) after completion of main episodes. Two CBOs provided incentives for participant referrals. Over half of CBOs (13/22, 59%) had their project managers directly involved in the implementation of KIU!. Project managers were mid- and high-level leaders, including deputy directors, program directors, and project coordinators.

Medium-volume CBOs had slightly more staff involved in KIU! delivery than high- and low-volume CBOs (2.29 vs 2.13 and 1.43, respectively) (Table 2). Medium-volume CBOs also reported higher rates of project manager involvement in KIU! than high- and low-volume CBOs (5/7, 71% vs 4/8, 50% and 4/7, 57%, respectively). Low-volume CBOs had never been directly funded by the CDC for HIV prevention, while nearly 40% of medium- and high-volume CBOs had been funded (3/7, 43% and 3/8, 38%, respectively). Medium- and high-volume CBOs also had more experience implementing CDC-designated EBIs (5/7, 71% and 7/8, 88%, respectively).

CBOs with more years of experience providing HIV services tested more YMSM for HIV on average per year, with high-experience CBOs testing an average of 402.2 YMSM per year for HIV compared to 338.4 YMSM in medium-experience CBOs and 145.3 YMSM in low-experience CBOs. High-experience CBOs also had slightly more staff involved in KIU! delivery than medium- and low-experience CBOs (2.21 vs 1.40 and 1.67, respectively). High- and low-experience CBOs’ project managers were more frequently engaged in KIU! (9/14, 64% and 2/3, 67%, respectively) than medium-experience CBOs (2/5, 40%). Medium-experience CBOs less frequently reported funding from the CDC for HIV prevention than high- and low-experience CBOs (1/5, 20% vs 4/14, 29% and 1/3, 33%, respectively). Finally, high- and medium-experience CBOs more frequently reported prior experience implementing CDC-designated EBIs (9/14, 64% and 3/5, 60%, respectively) than low-experience CBOs (1/3, 33%).

CBOs with prior experience implementing CDC-designated EBIs reported, on average, a higher number of YMSM receiving HIV tests per year (490.5 YMSM) than CBOs without prior experience (153.7 YMSM). CBOs with prior experience implementing EBIs reported a slightly higher average of staff involved in KIU! than CBOs without prior experience (2 and 1.8, respectively). However, CBOs with prior experience implementing EBIs reported more often than those without prior experience that project managers were directly involved in implementation (8/13, 62% and 4/9, 44%, respectively).

To recruit for KIU!, CBOs used four recruitment methods: (1) online recruitment (eg, on their website or on social media); (2) recruitment on hook-up and dating apps (eg, Grindr, Scruff, and Jack’d); (3) recruitment through participant referrals; and (4) recruitment through outreach and community partnerships. The most frequent mode of recruitment was through outreach and community partnerships (21/22, 95%), and the least frequent recruitment occurred via hook-up apps and participant referrals (6/22, 27% and 5/22, 23%, respectively; Table 2). Nearly 70% (15/22, 68%) of CBOs additionally created their own promotional materials to recruit participants in addition to materials provided by the research team. We compared the frequency of engagement in these activities by client volume of the CBOs, years of HIV service provision, and prior experience implementing CDC-designated EBIs (Table 2). We discuss these differences in approaches to recruitment below.

To retain participants, CBOs used four communication methods: (1) phone calls; (2) text messages; (3) emails; and (4) social media. Communication methods were primarily used to send reminders to participants to complete KIU! modules. Communication frequency varied across CBOs from no set schedule to infrequent (ie, monthly or less) and frequent (ie, weekly or biweekly). Phone calls and text messages (14/22, 64% for both) were the most common methods of communication, while social media was less common (2/22, 9%). Moreover, 36% (8/22) of CBOs chose to offer in-house accommodation to complete KIU! modules (eg, space and technology). In addition to CBOs’ individual approaches to retention, they were also able to use an online dashboard developed by our research team to track participant completion of KIU! modules. On average, all CBOs logged into the KIU! dashboard 10.5 times per month. We have compared modes of retention and dashboard utilization by client volume, years of HIV service provision, and prior experience implementing CDC-designated EBIs (Table 2). We discuss these differences in approaches to retention below.

We previously described differences in recruitment and retention activities by CBO client volume, years of HIV service provision, and direct project manager involvement in implementation. In recruiting and retaining participants, CBOs highlighted numerous lessons learned regarding what they felt helped with recruitment and retention and what hampered recruitment and retention. We now describe these lessons learned. In Table 3, lessons learned are compared by years of HIV service provision and project manager involvement. No discernible differences in lessons learned by client volume were found, and thus, these categorizations are not included within the table.

Despite the interest in and uptake of DHIs, little is known about how local organizations, such as CBOs, engage in recruitment and retention in real-world settings. We analyzed phone or Zoom call transcripts and interview data to capture CBO recruitment and retention approaches to KIU! implementation. We also used qualitative analyses to identify lessons learned by CBOs related to recruitment and retention. In future analyses, the research team plans to use coincidence analysis to identify configurations of recruitment and retention activities that promote greater recruitment.

Throughout the implementation of KIU!, CBOs used 4 approaches to recruit participants: outreach and community partnerships, online recruitment (eg, on their website or on social media), hook-up and dating apps, and participant referrals. Overall, CBOs most often used outreach through community partnerships and online recruitment. Most CBOs also developed their own promotional recruitment materials, especially CBOs with prior experience implementing CDC EBIs. During interviews, CBOs acknowledged that their recruitment efforts should have focused on communicating the benefits of KIU!, using recipient-centered language (ie, removing jargon), and emphasizing that KIU! was for anyone and not targeted for individuals deemed at higher risk. CBOs also noted how the recruitment location introduced barriers to discussions with participants (in a waiting room compared to a private room). CBO staff were able to recognize that individual values and preferences were barriers to recruitment efforts. As a result, CBOs had to adjust their approaches by creating customized information about KIU! to communicate the benefits and find safe locations for participants to honestly answer personal questions about themselves in order to enroll in the intervention. CBO recruitment reflections echo the importance of tailoring content for the target audience and considering the perspectives of clients in intervention delivery [38-40]. These findings stress the importance of researchers working closely with their participant audience to anticipate recruitment barriers and collaboratively create facilitators.

For retention, CBOs used a variety of approaches. CBOs called, sent text messages, and emailed participants to remind them about KIU!. Additionally, 1 CBO hosted in-person events with prizes to promote continued engagement; however, these were met with mixed reviews from participants who expected events to be hosted virtually since KIU! is a digital intervention. Overall, CBOs noted that all retention methods had limited success. While reminders represented the most common approach, CBOs may need to consider using other research retention methods such as providing incentives, reducing barriers (providing alternative data collection methods), tracing, and emphasizing participation benefits [41,42]. Further, CBOs may want to work closely with participants to understand which retention strategies are the most meaningful to them [40].

DHIs have the potential to enhance the quality of care and reduce health care costs. Despite evidence that DHIs are effective for improving HIV prevention outcomes [25,30,43], the requirements to effectively scale these programs and bring them to practice are not well-established. The pre-established infrastructure, workflow, and experience of CBOs can create challenges for the scaling up of DHIs. Previous research indicates that the main obstacles to the adoption and scaling of DHIs include a lack of technical support, resistance to change, and financial constraints. Many CBOs lack technical proficiency and experience with implementing EBIs. These barriers make training a necessary component of implementation but can be a burden for CBO staff. CBOs also experience high turnover rates of staff who are involved in the recruitment and retention of participants, which can result in delays and deprioritization of recruitment for interventions that require additional training and cause gaps in retention as new staff undergo the training process. Workflows and strategies for recruiting and retaining participants for testing or in-person intervention modalities differ greatly from the implementation of DHIs, which could promote CBOs’ resistance to change or adapt already existing processes. Some CBOs also lack experience in retaining participants for an extended period of time, which is necessary for completing the learning objectives of online HIV prevention programs (eg, KIU! requires a minimum participant retention period of 3 months). All these challenges can be overcome with a better understanding of the optimal approaches of retention and recruitment activities to achieve DHI implementation success.

Although CBOs have been working with YMSM for many years and have experience implementing other CDC-designated EBIs, there is no consensus on successful recruitment and retention strategies for this population. This suggests that there is a critical need for implementation researchers to identify the best practices to improve the reach and benefits of future EBI implementation.

This study described the approaches real-world CBOs use to recruit and retain participants in a DHI for HIV prevention among YMSM. Although lessons learned are reported through interviews with CBO staff about intervention success, this manuscript does not report recruitment or retention rates, as the evaluation in this study was not part of a prespecified data analysis and was not meant to be a statistical analysis. Instead, a descriptive evaluation of the different recruitment and retention approaches CBOs used to implement a novel DHI for HIV prevention is provided. The results represent staff perceptions of success with different methods and are not associated with participant engagement or retention outcomes. The recruitment and retention methods listed here may not be representative of CBOs that did not choose to be enrolled in this study. CBOs self-selected to offer KIU!, and they may have had more resources and interest to engage in research activities. While the research team aimed to interview 2 staff members at each CBO implementing KIU!, CBO staff were not required in their funding contracts to participate in these interviews, and the research team was not willing to coerce participation. This resulted in 9 CBOs only having 1 staff member who consented to participate and 5 CBOs not having any staff member who consented to participate. Further, the staff who engaged in interviews varied greatly in their position titles across CBOs, which may have resulted in varied perspectives on implementation. This limitation was reduced by requiring those participating in interviews to be directly involved with the implementation of KIU!. Additionally, the retention findings were limited by few interviewee reflections on the approaches in the sample. This study took place during the COVID-19 pandemic. As a result, some of the reflections from CBO staff included challenges unique to that time period and may not be as relevant to recruitment and retention practices during stable economic and epidemiologic periods.

Limited research has described the recruitment and retention practices of CBOs, especially for HIV prevention DHIs. CBOs providing HIV prevention services often have more experience recruiting clients than retaining them, given the nature of point-in-time services such as HIV testing and condom distribution. While some CBOs have experience with behavioral interventions for HIV prevention, DHIs like KIU! rely on CBO staff to ensure ongoing participant engagement and retention, even though the intervention takes place outside the CBO’s physical premises. In this trial of 22 CBOs implementing a DHI for HIV prevention, only a few CBOs had prior experience with DHIs, highlighting the need for dedicated staff and resources to successfully implement such interventions. These resources should include insights and strategies for effective recruitment and retention, as presented in this manuscript.