In 2020, a simulated randomized web-based experiment was programed on SurveyMonkey (SurveyMonkey Inc). The experiment was designed to test the effectiveness of targeted social media messages to increase intentions to participate in a real-world observational case-control study called the Cancer Loyalty Card Study (CLOCS) [37]. CLOCS is an observational case-control study that aims to investigate the self-care behaviors of patients with ovarian cancer before their cancer diagnosis. It seeks to do this by investigating differences in transactional data (such as medication purchasing) between women with and without ovarian cancer (the transactional data are collected through the loyalty cards of 2 UK-based high street retailers). Cases (ie, women with ovarian cancer) were recruited through participating National Health Service sites, while controls were recruited through the study website. Thus, those who were eligible to take part as control participants were recruited through social media and other internet-based sources.

The study sample comprised women aged between 18 and 70 years living in the United Kingdom without an ovarian cancer diagnosis who were potentially eligible for the real-world observational case-control study. Study participants were recruited through a web-based survey vendor, Dynata (Dynata Global UK Ltd).

At the beginning of the experiment, those who were interested in taking part in the web-based experiment were presented with information about the study, including a brief description of CLOCS as well as a consent form. If participants consented and were eligible, they were randomized (in a 1:1:1 ratio) to receive 1 of 3 simulated Twitter posts: a standard advertisement (control condition), an advertisement with patient endorsement (patient endorsement condition), or an advertisement with a descriptive social norms message (social norms condition; Table 1). To generate authentic Twitter messages, real tweets were posted on a dummy Twitter account, alongside an infographic detailing information about CLOCS. Screenshots were taken of these posts for use in the experiment (the messages were immediately deleted after each one was posted; Figures S1, S2, and S3 in Multimedia Appendix 1 contain screenshots of the messages).

The content of the messages is presented in Table 1, along with a Flesch-Kincaid readability score calculated using the web-based software Grammarly (Max Lytvyn, Dmytro Lider, and Alex Shevchenko). This was done to ensure that the message was understandable to the target audience [38]. The standard and patient endorsement messages had the lowest readability scores (15 and 7.9, respectively) and were the easiest to understand (Table 1).

After being presented with the Twitter messages, participants were asked 2 comprehension questions on whether CLOCS only recruits women with ovarian cancer and what kind of data CLOCS are analyzing. Participants could only continue in the survey if they answered the questions correctly [35,39,40]. The primary outcome was participants’ intention to take part in CLOCS, and we asked individuals whether they would participate in the advertised study, adapted from previous literature [34,35,39,41,42]. It featured a fully labeled 4-point response scale (“definitely not,” “probably not,” “yes probably,” and “yes definitely”).

To explore how the messages were perceived by the participants, we included 2 questions on how easy the message was to understand (“very difficult,” “fairly difficult,” “fairly easy,” or “very easy”) and how clearly the aims of the study were communicated (“not at all,” “a little,” “very,” or “extremely”).

In the next step, participants were asked about their past participation in health care research (“yes” or “no”) and whether they had loyalty cards from UK-based high-street retailers. Sociodemographic questions covered age (“18-24,” “25-34,” “35-44,” “45-54,” or “55-70”), education (“no college degree” or “college degree, equivalent, or higher”), employment status (“yes” or “no”), marital status (“single”, “married or living with a partner”, “divorced or separated or widowed”), self-reported health (“poor,” “fair,” “good,” or “excellent”), and history of cancer in themselves, family, or close friends (“yes” or “no”).

Individuals were then given the opportunity to state their thoughts on improving social media messages for the recruitment of study participants in an open-ended question.

The survey concluded with an active interest question on whether participants would be interested in being redirected to the CLOCS website for more information on how to participate [34,40-42]. Those who responded yes were provided with a link to the CLOCS website on the final page of the survey. The website opened in a new tab for participants who clicked on the link. No further data associated with their direct participation in CLOCS were collected in this experiment. The web-based experiment took, on average, 5 minutes to complete.

Ethics approval for this study was obtained from the University College London Research Ethics Committee (17813/001). All participants provided consent to take part in the study. All the data collected as part of the study were anonymized, meaning no identifiable information were collected. Eligible participants who completed the questionnaire received a small financial incentive from Dynata, as per their panelist agreements.

A pilot study was conducted beforehand for sample size calculations. Based on the findings from the initial sample of 359 participants, with a 10 percentage point difference in the intention to take part (“yes, definitely” or “yes, probably” versus “definitely no” or “probably no”), we determined that the number of participants needed to achieve 95% CI and 80% power was 350 per trial arm. Data from participants in both the pilot and final samples were combined for analysis.

Sample characteristics were assessed using descriptive statistics (Table 2). Differences in participants’ intention to take part in CLOCS and perception of the messages were assessed using univariate and multivariate ordinal logistic regression. Willingness to visit the actual website was assessed between groups using univariate and multivariate binary logistic regressions. Adjusted odds ratios (aORs), 95% CIs, and P values are presented in the results, with P values below .05 regarded as statistically significant.

The responses to the open-ended feedback question were categorized into main themes through content analysis [43].

Figure 1 demonstrates the flow of participants through the study. In total, 2500 invitations were sent out on behalf of University College London researchers to women registered on a survey panel (Dynata), and 47.8% (1195/2500) responded to the invitation. Out of these potential participants, 92.6% (1107/1195) were eligible for the study.

Eligible participants were randomized to the experimental conditions: 376 to the control condition, 358 to the patient endorsement condition, and 373 to the social norms condition. Across conditions, 4.2% (47/1107) did not finish the survey after randomization, leaving a final sample of 1060, who were all included in the analysis: 34% (360/1060) in the control condition, 32.5% (345/1060) in the patient endorsement condition, and 33.5% (355/1060) in the social norms condition. Most women in the analytical sample were in paid employment (705/1060, 66.5%), were married or cohabiting (605/1060, 57.1%), did not have a college degree (549/1060, 51.2%), owned at least 1 loyalty card from a pharmacy (783/1060, 73.9%), experienced cancer closely (ie, either themselves or with family or close friends) (772/1060, 72.8%), and reported good or excellent health (701/1060, 66.1%). Post hoc comparisons revealed that sociodemographic variables did not vary significantly across the experimental conditions (Table 2).

Overall, the intention to take part in CLOCS was high, with 60% (636/1060) of women stating that they would probably or definitely participate. Table 3 shows the distribution of intentions after reading the Twitter messages. The ordered logistic regressions in Table S1 in Multimedia Appendix 1 show that the behavioral messages, both patient endorsement (odds ratio [OR] 0.74, 95% CI 0.56-0.98; P=.03 and aOR 0.73, 95% CI 0.55-0.97; P=.03) and social norms (OR 0.74, 95% CI 0.54-0.93; P=.015 and aOR 0.69, 95% CI 0.52-0.92; P=.009), decreased intention to take part in CLOCS. None of the sociodemographic variables were significantly associated with the intention to participate in CLOCS.

Table 4 shows that most study participants stated that the messages were fairly or very easy to understand (796/1060, 75.1%) and that the aims of the study were very or extremely clearly communicated (594/1060, 56%). However, the ordered logistic regression results in Table S2 in Multimedia Appendix 1 show that individuals in the patient endorsement condition perceived the message as more difficult to understand (OR 0.63, 95% CI 0.47-0.84; P=.002 and aOR 0.65, 95% CI 0.48-0.87; P=.004) and the study aims as less clear (OR 0.70, 95% CI 0.53-0.92; P=.01 and aOR 0.72, 95% CI 0.55-0.95; P=.02) than those in the control condition. There were no statistically significant differences in the perceptions of those in the social norms condition and those in the control condition.

Almost half of the study participants (526/1060, 49.6%) indicated that they would like to be redirected to the CLOCS website after the survey. The binary logistic regression in Table S1 in Multimedia Appendix 1 shows that participants who were presented with the social norms message were less interested in being redirected than those in the control condition (157/355, 44.2% vs 191/360, 53.1%; OR 0.70, 95% CI 0.52-0.94; P=.02 and aOR 0.74, 95% CI 0.54-0.99; P=.05). While there were no significant differences between the patient endorsement and control conditions (178/345, 51.6% vs 191/360, 53.1%; OR 0.94, 95% CI 0.70-1.27; P=.70 and aOR 0.96, 95% CI 0.71-1.30; P=.78), women with a loyalty card (117/205, 52.4% vs 58/144, 40.3%; aOR 1.47, 95% CI 1.10-1.95; P=.008), excellent health (70/135, 51.8% vs 22/57, 38.6%; aOR 1.94, 95% CI 1.00-3.76; P=.05), aged between 55 and 70 years (117/205, 57.1% vs n/N, 40.3%; aOR 1.89, 95% CI 1.19-3.00; P=.007) and those who had experienced cancer closely (404/772, 52.34% vs 122/288, 42.4%; aOR 1.37, 95% CI 1.03-1.83; P=.03) were more interested in visiting the study website. Those who had previously participated in health research were less likely to want to be redirected (348/734, 47.4% vs 178/326, 54.6%; aOR 0.75, 95% CI 0.58-0.99; P=.04).

Table 5 shows the main themes of the content analysis per message. While 57.5% (609/1060) of the study participants were willing to provide some feedback, the majority (395/609, 64.8%) stated that the messages did not require changes. Another common theme was clarity (75/609, 12.3%), where participants thought there was too much jargon, that the message should be shorter, adding the hashtags at the end would make it more readable, and message format (59/609, 9.7%), where participants recommended using brighter colors or adding more infographics or a video instead of text.

Moreover, some participants (237/609, 3.8%) stated that the messages were unclear on how the advertised study uses loyalty cards to help with an ovarian cancer diagnosis. To increase the credibility of the message, 3.3% (20/609) of participants suggested including the university’s or sponsor’s logo at the beginning of the message. Some participants also suggested posting the messages on several social media platforms (16/609, 2.6%), as well as using patient or celebrity endorsement (9/609, 1.5%) or advertising an incentive (7/609, 1.1%).

This randomized web-based experiment examined the effectiveness of adding behavioral messages to Twitter advertisements for participant recruitment in a real-world case-control study (CLOCS). The results show that the standard messages yielded the highest intentions compared to the 2 normative behavioral messages. Furthermore, the social norms message decreased the willingness to visit the real study website after the survey. The vast majority of participants stated that the messages did not require changes, but some preferred clearer and shorter advertisements.

Our findings contrast with previous research, which has shown that using behavioral messages, such as social norms [34,35] and patient endorsement [31,32] can be effective methods to engage with the public on the web. The negative effect we found can partially be explained by the reduced readability of the messages, as individuals in the social norms condition perceived the message to be more difficult to understand—and the aims of the study were less clear than those in the standard advertisement.

Individuals who have had experience with a cancer diagnosis, either themselves, with family, or with close friends, were found to be more interested in CLOCS. This is in line with research reporting that familial history of cancer is associated with increased breast and ovarian cancer screenings due to individuals’ increased awareness of cancer-related complications [44]. In this study, participants’ awareness of ovarian cancer and its risks—partially informed by their close experiences with cancer—may explain their increased interest in CLOCS. Therefore, having a personal awareness of or connection to the proposed project can increase individuals’ interest in health-related research. We also found that women aged between 55 and 70 years had increased interest in visiting the CLOCS website. Ovarian cancer is rare in women younger than 30 years, but the risk increases with age, drastically spiking after 50 years—with the average age of diagnosis being between the ages of 50 and 70 years [45]. Thus, the saliency of the risk for ovarian cancer in these age groups may, in part, explain their interest. Final, while individuals recommended including a video in the message, a recent experimental study did not find any effect of adding animated decision aids to a website with the intention to participate in a case-control study [42].

One strength of this study was the use of a randomized experimental design to evaluate the effectiveness of adding behavioral messages to Twitter messages. Additionally, the study used validated questions on intentions and active interest. A final strength of this study is that the statistical analysis included a large number of covariates known to influence participation in health research.

This study has some important limitations, which call for follow-up research. First, the 2 messages were grounded in social norms and patient endorsement, which have mixed and limited evidence supporting the efficacy of these messages in influencing participation in clinical research [46] and may not have been the right theoretical basis for the content of the recruitment messages. This limitation is further exacerbated by the paucity of experimental research testing and reporting different messages on digital and social media platforms and their effectiveness on research recruitment. More theory-based formative research using social media marketing techniques, field experiments, and co-design approaches is needed to improve our understanding of the evidence-based application of social influence on research participation for recruiting participants to health research using social media.

Second, throughout the design and testing of both the social norms and patient endorsement messages, the authors considered whether the messages were suboptimally designed despite having contributions from patient representatives who reviewed the messages, and these have undergone various iterations. This is due to 2 reasons. While previous studies have shown that proximal social norms are more effective in different contexts [30-36], it was not possible to use them in our experiment due to the lack of data supporting the claim at the time of the CLOCS recruitment [37] and the novelty of this case-control study. Additionally, the social norm message had to use a vague verbal quantifier, “many women,” and focus on satisfaction with participation rather than the participation rate to ensure messages were ethical and not coercive. Similarly, the patient endorsement message may not have highlighted the link between motivation and action because the message only referred to the patient representative buying medication for symptoms from retailers before a cancer diagnosis. It is possible that future studies focusing on barriers and facilitators of health research participation in the design of the recruitment messages rather than normative behaviors may demonstrate different outcomes.

The authors aimed to address the aforementioned issues with feedback from the participants. However, this exercise did not lead to clear future recommendations other than the use of the factual message used in the control condition. Nevertheless, the outcomes of this experiment informed the recruitment of the CLOCS participants. The authors gained further understanding of the potential limitations of recruiting participants to CLOCS and successfully recruited 249 participants using Facebook advertisements with the control message [47]. The cost per participant recruited was between US $12 and $19, which is comparable to and less than other health-related studies with a targeted population [48]. This hypothetical experimental study demonstrates the importance of testing messages to be used in internet-based recruitment strategies, the potential limitations, and biases, and not relying only on consensus methods. Embedding process evaluations and pre- and postresearch data collection could have a significant impact on the resources allocated to recruitment as well as whether they reach their intended outcomes. Based on the outcomes of the CLOCS study [37], future studies could emphasize how the participation of women without ovarian cancer in the case-control study can help better early diagnosis of ovarian cancer, use their response rates, and further explore why the existing participants took part in this research to develop effective messages.

Furthermore, in line with previous literature, we measured attitudes toward the simulated Twitter message and CLOCS to capture individuals’ potential reactions to the website [42], which has its limitations, as several studies have reported on the intention-behavior gap [49]. As such, motivational interventions are necessary but often not sufficient to change behavior. It is possible that the hypothetical nature of the web-based experiment may have introduced a potential response bias. Similarly, there might have been a social desirability or agreement bias, where study participants tended to overestimate their intentions.

Last, we did not account for participants’ familiarity with Twitter. Our sample may have contained women who were not used to reading messages containing hashtags (Twitter use was not verified). Moreover, while we tried to include a behavioral outcome by including an option for participants to visit the CLOCS website, our experiment did not formally assess the analytics of the website or investigate how the messages influenced click behavior. Final, this study may have been affected by selection bias, as factors shaping computer use (age, gender, socio-economic status, etc) tend to influence the demographics of the sample in web-based studies [50]. For instance, there are usually similar demographic patterns across social media platforms, where users are primarily made up of young, female, and urban individuals [51,52].

Our findings suggest that researchers conducting health-related studies should focus on using simple messages for participant recruitment through Twitter. To increase engagement with potential participants through social media, recruitment messages should be easy to read, transparent, and appropriately targeted to an audience that could have experience related to or an interest in the proposed study. Future research could test messages involving social proofing, such as sharing the experiences of study participants. Additionally, messages could be tested in field experiments by controlling the date, time, hashtags, and images used.

In conclusion, our results indicate that adding behavioral messages containing patient endorsement or social norms to simulated recruitment messages on Twitter decreased participants’ intention to take part in a real-world case-control study. The social norms message also decreased participant interest in visiting the actual study website. These results can be partially explained by difficulties in reading and understanding the message content, with the addition of normative behavioral components. Future research should continue exploring and optimizing methods that can effectively leverage social media platforms for the engagement of potential participants in health-related research.