Data for this study were retrieved from a previously conducted study on Internet-based relaxation training for people with mild to moderate stress and anxiety symptoms [9]. Participants were recruited from the general population primarily by online advertisement. Inclusion criterion was self-reported stress symptoms, whereas the exclusion criteria were elevated symptom levels of anxiety or depression or other severe psychological or somatic problems that warranted immediate care, younger than 18 years of age, insufficient mastery of the Swedish language, or lacking daily access to computer, Internet, and cell phone. In the study, a total of 162 included participants were presented with an online intervention and were randomized to either normal or enhanced treatment presentation and either normal or enhanced therapist support in a full factorial design [9]. In the normal treatment condition, the treatment was presented as black-and-white text files, whereas in the enhanced presentation condition, the treatment was presented in full color and also in video format. In the normal support condition, participants received support from trained therapists at least once a week, whereas in the enhanced support condition, participants received daily support from therapists based on motivational interviewing. The intervention consisted of a brief 4-week program of applied relaxation similar to what has been used and empirically tested in previous studies [33]. Each week of the program included webpage material, assigned relaxation exercises, and contact with a therapist via the webpage. All participants were asked to answer a battery of self-report instruments before, during, and after the intervention; of the 162 participants, 157 had complete data from the pre- and midtreatment assessments, whereas 96 participants had complete data also from the posttreatment assessment.

Early dropout (yes/no) was assessed by counting the number of participants who discontinued the study before completing the first week of the treatment program; attrition (yes/no) was assessed by counting the total number of participants who discontinued the study before the posttreatment assessment. Treatment adherence was divided into the two variables treatment progress and registered exercises, both on continuous scales. Treatment progress was assessed by measuring how much of the Web-based treatment material (eg, texts, examples, assignments) each participant accessed before dropping out or completing the treatment. Because the treatment consisted of 25 such items, this variable ranged from zero (not accessed the treatment) to 25 (accessed the whole treatment). Registered exercises were measured by the mean number of prescribed exercises of applied relaxation that the participant had registered on the webpage each week. The weekly number of registered exercises ranged from zero (not completed any exercises) to 14 (completed all prescribed exercises). As a measure of stress symptoms and treatment outcome, the Perceived Stress Scale (PSS) [34] was chosen. The short version of the PSS used in this study contains 14 items scored on a scale between zero and 4, which provides a total score between zero and 56 with a higher score indicating more symptoms of stress. In the current study, the PSS had an internal reliability of α=.72. In previous studies, the PSS has showed adequate psychometric properties [35]. To assess reliable improvement (yes/no), the Reliable Change Index for PSS was calculated (described subsequently).

The predictor variables consisted of the background variables age, gender, level of education (primary/secondary/university), occupation (student/unemployed/employed/sick leave/retired), and computer expertise (low/intermediate/high). In order to facilitate interpretation, education and occupation were transformed into dichotomous variables (nonuniversity vs university; employed/student/retired vs unemployed/sick leave). Four psychological predictor variables were collected by self-report instruments at baseline: time perspective, treatment credibility, and internal and external motivation. Intrinsic motivation and therapeutic bond were measured at midtreatment and stress symptoms were measured both at baseline as a predictor variable and at postmeasurement as an outcome variable. Because a previous analysis [9] had shown that enhanced support was associated with higher adherence to the online treatment program, treatment condition (normal vs enhanced therapeutic support) was also included as a predictor variable in the analyses. No other independent variable was significantly associated with any outcome variables.

Time perspective was measured with the Zimbardo Time Perspective Inventory Short Form (ZTPI) [36,37]. The version of the ZTPI used in this study has three subscales—future, hedonistic, and fatalistic—and comprise 22 items scored on a scale from 1 to 5 [38]. The ZTPI has been evaluated for research in health psychology and has shown adequate psychometric properties [39]. The ZTPI subscales had internal reliabilities of α=.69-.73 in this study.

Treatment credibility was measured with the Treatment Credibility Scale (TCS), which is often used in studies of Internet interventions and is an adaptation from Borkovec and Nau [40]. The TCS consists of five items scored on a scale from 1 to 10 with a higher score indicating more trust in the current treatment. The TCS had an internal reliability of α=.83 in this study.

Internal (ie, identified and integrated motivation) and external motivation were measured with the Treatment Self-Regulation Questionnaire (TSRQ) [41]. The TSRQ consists of two subscales—internal motivation (IM) and external motivation (EM)—each measured with six items that were adapted to suit the Internet intervention used in this study. Each subscale provides a score between 6 and 42 with a higher score corresponding to a higher degree of motivation. The TSRQ has been used in studies on motivation and health behaviors and has shown adequate psychometric properties [42]. In the present study, the TSRQ-IM and TSRQ-EM had internal reliabilities of α=.70 and α=.68, respectively.

Intrinsic motivation was measured with the Intrinsic Motivation Inventory (IMI) [43]. The IMI aims at measuring how pleasant, interesting, and meaningful a task is perceived and has nine items scored on a scale between 1 and 7 which provides a total score of 9 to 63 with a higher score indicating a more positive experience of the task. Due to mixed findings concerning the factor structure of the IMI, only the total score was used [44]. The IMI had α=.71 in this study.

Therapist bond was measured with the Working Alliance Questionnaire Short Form (WAI) [45]. The WAI has been widely used in psychotherapy research and has shown adequate psychometric properties [46]. The short form WAI consists of three subscales—Goal, Task, and Bond—each with a score between 4 and 28 where a higher score equals a higher degree of therapeutic alliance. The WAI total scale had an internal reliability of α=.73 in this study.

Before analysis, data were screened for outliers and normality, linearity, and homoscedasticity were evaluated by scrutinizing the residual scatterplots between predicted variables and errors of prediction and found adequate. Because subscales were entered into the analyses, multicollinearity was assessed by analyzing the variance inflation factor for each predictor variable and found to be nonproblematic.

Reliable improvement was computed by dividing the difference between the pretreatment and posttreatment scores by the standard error of the difference between the two scores. If the Reliable Change Index was greater than 1.96, a change of that magnitude would not be expected due to the unreliability of the measure [47]. Using this procedure, the reliable improvement criterion for PSS was a change score of 10 or more in this study.

Modeled after deGraaf et al [48], bivariate regression analysis was first used to identify candidate (P<.10) predictor variables for each outcome variable. All identified predictor variables were included in subsequent multiple regression analyses using a backward deletion process for each outcome variable. Logistic regression was used for the dichotomous outcome variables dropout and attrition and linear regression was used for continuous outcome variables. Cox-Snell R ² and Nagelkerke R ² were used as a measure of overall model fit in the logistic regression analyses and R ² was used in the linear regression analyses. Because some of the variables had distributions that deviated somewhat from normality, the final regression models were confirmed using robust regression analyses with bootstrap and bias correction. The sample size of 157 was deemed adequate for regression analysis of a maximum of eight predictor variables for each outcome variable except stress symptoms and reliable improvement for which the sample size of 96 was deemed adequate for six predictor variables. Single missing values (n<1%) were imputed using expectation-maximization estimates. A P value of .05 was considered the threshold for statistical significance if not stated otherwise; exact P values were reported for the final analyses.

The multivariate logistic regression analyses showed that early dropout could be significantly negatively predicted by the TCS (B=–0.14, χ² ₁=10.5, P=.001), whereas total attrition to postmeasurement was predicted by baseline stress symptoms (B=0.08, χ² ₁=3.2, P=.05), the ZTPI Hedonistic subscale (B=0.32, χ² ₁=10.3, P=.001), and the IMI (B=–0.06, χ² ₁=5.7, P=.02) (see Table 2). Early dropout was associated with a low belief in the treatment model, whereas dropout during the course of treatment was associated with having elevated stress symptoms, being more focused on the immediate consequences of behaviors, and finding the treatment uninteresting or unengaging.

After controlling for level of support, treatment progress was positively predicted by level of education (beta=.24, t ₁₅₃=2.33, P=.02) and by the TCS (beta=.35, t ₁₅₃=3.36, P=.001), whereas registered exercises was significantly predicted only by the IMI (beta=.29, t ₁₅₅=2.43, P=.02) (see Table 3). All other predictor variables were nonsignificant in the multiple regression analyses for treatment adherence. Accessing the online treatment material was associated with a priori belief in the treatment model, whereas complying with the prescribed homework assignments was associated with experiencing interest and engagement in the treatment.

Posttreatment stress symptoms were significantly and positively predicted by the baseline stress symptoms (beta=.47, t ₉₁=4.43, P<.001) and by the TSRQ-EM (beta=.25, t ₉₁=2.40, P=.02) while negatively predicted by the TCS (beta=–.28, t ₉₁=3.53, P=.001) (see Table 4). Reliable improvement was positively predicted by baseline stress symptoms (B=0.11, χ² ₁=4.5, P=.03), by the TCS (B=0.09, χ² ₁=3.3, P=.05) and by the WAI (B=0.14, χ² ₁=3.9, P=.049) (see Table 5). Reporting external pressure to complete the treatment was associated with worse treatment outcome, whereas a good therapeutic bond was associated with a substantial positive treatment effect. Treatment credibility predicted both overall symptom levels and substantial improvement.

In bivariate regression analyses and after controlling for baseline PSS score, posttreatment stress symptoms was significantly negatively predicted by both treatment progress (beta=–.31, t ₉₂=3.29, P=.001) and registered exercises (beta=–.21, t ₉₂=1.98, P=.05). The same pattern was seen for reliable improvement, which was significantly predicted by both treatment progress (B=0.26, χ² ₁=8.0, P=.005; OR 1.29, 95% CI 1.08-1.55) and registered exercises (B=0.20, χ² ₁=7.5, P=.006; OR 1.23, 95% CI 1.06-1.42). Whether treatment effect was mediated through treatment adherence could not be further investigated in this study because the power calculations had not accounted for this type of analyses.

The analyses showed that different ways of operationalizing (ie, measuring) dropout, treatment adherence, and treatment outcomes was related to somewhat different predictor variables, something that may be important to consider in psychotherapy process research. First, there was a difference in that early treatment dropout could be predicted by treatment credibility, whereas further attrition was predicted by a personality pattern of focusing of immediate consequences and by finding the treatment unengaging, two variables that are probably connected. People whose behavior is generally governed by immediate consequences and reward may find an online behavioral treatment largely consisting of abstract instructions and texts unsatisfying and boring. This pattern suggests that some people with low expectations of the treatment start the program, but very soon realize it does not suit them and quit. Whether these participants should receive more motivational support from therapists to stay in the online treatment or efforts should be made to guide them to other forms of treatment needs further investigation. Participants who dropped out later in the treatment reported a lower tendency to focus on future goals and also experienced the treatment as unrewarding. These results are in line with previous results showing that being able to focus on the future and to postpone rewards is associated with higher levels of education and health behaviors [23]. Although the treatment may be designed to be more interesting and engaging (eg, by employing media content or using features of gamification [49]), it may be difficult to counter the fact that most health behaviors rely on an ability to postpone rewards and a focus on future consequences. However, strategies such as repeated reminding of treatment goals and using everyday prompts may be alternatives for further investigation [50].

Treatment adherence showed a somewhat similar pattern to dropout with online treatment progress predicted by education and treatment credibility. Thus, working with the online material was associated with a probable familiarity of working with texts and thinking in abstract terms. Treatment credibility may correspond to a familiarity and interest in using the Internet for learning about health behaviors and perhaps previous positive experiences of online courses. Treatment credibility may also represent participants’ belief in the treatment model and specifically in using relaxation to target stress symptoms. Finally, the association between treatment credibility and treatment progress may represent the different reasons why people undergo Internet-based psychotherapy; for some it is a preferred choice, whereas for others it may be the only available option. Completion of homework assignments, similar to staying in the treatment program, was predicted by finding the treatment engaging. That the IMI was significantly associated with adherence in this study may be partly explained by the nature of relaxation exercises that may actually be pleasant in contrast to exposure exercises, for example. It is worth noting that the instrument used to measure intrinsic motivation, the IMI, is designed to measure different forms of intrinsic reward and comprise items concerning experiences of interest, enjoyment, and meaningfulness [41]. It may capture participants’ experience that the treatment suits them and their needs rather than the treatment being “fun.” Whether results from this study of relaxation exercises can be generalized to other types of assignments used in CBT needs further study. In short, the results from this study suggest that a high level of education and belief in the treatment model may be beneficiary for overcoming the effort needed to work with the online program, but participants who find the treatment suits them work with the prescribed assignments to a higher degree. However, few of the variables in this study could significantly predict adherence to the assignments, so there are probably other unknown variables that could help explain this behavior.

Among participants who did not dropout and remained in the study, treatment outcome, as measured by the posttreatment stress symptoms, was significantly positively predicted by baseline stress symptoms, but also by the TSRQ-EM measuring external motivation. External motivation corresponds to feeling pressured by others, mostly in a negative way, to complete tasks. In previous studies, external motivation has been associated with difficulties in sustaining behavior and a reliance on accountability [19]. That external motivation was associated with worse outcome in this study implies that people who complete the intervention because of feeling obliged to do so benefit less from the treatment. In a recent study of negative effects in online psychotherapy, some participants reported feeling pressured by their online therapist or the demands of the treatment schedule [51], but in this study, external motivation was measured at baseline and corresponded to the reasons why participants took part in the study. Why some participants reported external pressure is unknown and difficult to speculate about because this study was based on self-referral recruitment. It is important to note the different nature of psychotherapy compared to the workplace environment, where most of the research on motivation has been conducted. In studies of psychotherapy, results may rely more on participants being engaged and involved than in most workplace research. For example, unlike work tasks, psychotherapy aims at making participants generalize behaviors from the therapy situation to their everyday life, something that may be more probable if the patient experiences the treatment as meaningful. Also, the Internet format may deemphasize external motivation because of the perceived anonymity of both patients and participants [52,53]. Previous studies have shown that therapists’ prompts and reminders can have a relatively minor impact on treatment adherence among participants who may lack internal motivation [54]. Posttreatment stress symptoms were further negatively predicted by treatment credibility, an effect also seen in previous studies [21,22]. Belief in the treatment format and model seems to be a consistent predictor for positive outcome and may be used as a measure for identifying people for whom the online format and minimal therapist contact may be beneficiary. It is important to note that suggesting or persuading people to try Internet-based psychotherapy or a treatment model they do not believe in may be counterproductive, both given the effect seen for external motivation and the effect of treatment credibility on treatment outcome. However, given the positive results of many online behavioral interventions, it may be beneficiary to better inform skeptical participants about the treatment format (eg, by pretreatment vignettes or examples). After controlling for baseline stress symptoms, reliable improvement after the treatment program was predicted by treatment credibility, but also by therapeutic bond, an effect that has been seen in other studies as well [55]. Interestingly, this effect in this study was seen only among participants who improved substantially, further providing support for the notion that working alliance is as an important variable in online psychotherapy as in face-to-face therapy [56].

In this study, many of the proposed predictor variables could not significantly predict the outcome variables in the multivariate analyses. This suggests that several of the predictor variables covaried to a large degree; therefore, finding the best predictor variables is difficult. The multivariate analyses with backward deletion resulted in models with acceptable levels of model fit except for registered exercises (R ²=.13); so far, there is limited understanding of the processes that may be involved in following prescribed homework in ICBT. That personality traits associated with conscientiousness and delay discounting could predict carrying through with the treatment seemed reasonable, but it was somewhat surprising that working alliance could not. A strong working alliance is often associated with treatment adherence, but in this study this association was overshadowed by the impact of education and treatment credibility. The weak association between working alliance and treatment adherence was unexpected given that therapeutic alliance has shown to be strongly associated with adherence and outcome in several other studies of ICBT [57]. Finding the intervention engaging, interesting, and meaningful were crucial factors for participants in successful ICBT as seen in previous studies [58]. Although intrinsic motivation was a predictor for treatment adherence, it was not associated with treatment outcome; this was surprising given the association between treatment adherence and outcome. This study was not designed to investigate mediation effect; therefore; it was unfortunately underpowered to further analyze these processes.

This study has a number of limitations. First, several of the predictor and outcome variables were difficult to measure accurately. For example, the measurement of registered exercises was constructed to assess the treatment’s effective mechanism, but may have failed to fully capture changes in participants’ everyday behavior. The intervention encouraged participants to conduct relaxation exercises and highly engaged participants may have done so without registering on the webpage. Further, several of the self-report instruments have not been used in psychotherapy research before and their psychometric properties in this context are unknown. For example, the internal reliability of some questionnaires seemed to be somewhat lower in this study compared to previously reported figures. Second, a more complex design with repeated measurements of adherence could have been conducted to show causal mediation, but this would also have demanded a much larger sample size. Third, the lack of a control group or a face-to-face treatment condition limits the generalizability of the results. Therefore, whether the conclusions from this study are valid for other treatment modalities are unknown. Finally, and most importantly, there was large dropout (39%) at postmeasurement and although investigating dropout from Internet-based interventions was one of the aims of this study, it also meant loss of follow-up data. This also means that the conclusions regarding predictors of clinical outcome are only valid for participants who stay in the study or treatment, which limits the generalizability of the results. A more thorough analysis of participants who dropped out may have resulted in a better understanding of this group.

In conclusion, this study confirms the importance of treatment credibility and working alliance in Internet-based psychotherapy and also suggests that experiencing intrinsic reward from participating in the treatment may be important. In contrast, external pressure to try online therapy may be counterproductive and lead to worse outcomes. Apart from identifying people who believe in the online treatment format and continue to explore the best methods for online therapist support, it may also be valuable to further investigate what makes participants engage in an intervention and what features makes the intervention interesting and meaningful [55]. However, given that quite rudimentary online treatment programs seem to be effective, it is possible that Internet-based interventions simply suit some participants better than others and that the focus should be on identifying these participants early [59]. Future studies may investigate both what makes treatments engaging while recognizing that different participants may have different requests and wishes for an intervention [60]. A good match between participant expectations and needs on the one hand and the intervention content and design on the other hand may be one of the reasons clinical assessment before treatment and tailoring treatment to better suit different patients may be beneficiary for treatment outcomes in Internet-based interventions [61]. Believing in the online treatment format and finding working with the online program rewarding may be two aspects of a process that could be investigated further to develop even more effective treatment programs.