Skip to main content

The Appalachia Mind Health Initiative (AMHI): a pragmatic randomized clinical trial of adjunctive internet-based cognitive behavior therapy for treating major depressive disorder among primary care patients



Major depressive disorder (MDD) is a leading cause of disease morbidity. Combined treatment with antidepressant medication (ADM) plus psychotherapy yields a much higher MDD remission rate than ADM only. But 77% of US MDD patients are nonetheless treated with ADM only despite strong patient preferences for psychotherapy. This mismatch is due at least in part to a combination of cost considerations and limited availability of psychotherapists, although stigma and reluctance of PCPs to refer patients for psychotherapy are also involved. Internet-based cognitive behaviorial therapy (i-CBT) addresses all of these problems.


Enrolled patients (n = 3360) will be those who are beginning ADM-only treatment of MDD in primary care facilities throughout West Virginia, one of the poorest and most rural states in the country. Participating treatment providers and study staff at West Virginia University School of Medicine (WVU) will recruit patients and, after obtaining informed consent, administer a baseline self-report questionnaire (SRQ) and then randomize patients to 1 of 3 treatment arms with equal allocation: ADM only, ADM + self-guided i-CBT, and ADM + guided i-CBT. Follow-up SRQs will be administered 2, 4, 8, 13, 16, 26, 39, and 52 weeks after randomization. The trial has two primary objectives: to evaluate aggregate comparative treatment effects across the 3 arms and to estimate heterogeneity of treatment effects (HTE). The primary outcome will be episode remission based on a modified version of the patient-centered Remission from Depression Questionnaire (RDQ). The sample was powered to detect predictors of HTE that would increase the proportional remission rate by 20% by optimally assigning individuals as opposed to randomly assigning them into three treatment groups of equal size.

Aggregate comparative treatment effects will be estimated using intent-to-treat analysis methods. Cumulative inverse probability weights will be used to deal with loss to follow-up. A wide range of self-report predictors of MDD heterogeneity of treatment effects based on previous studies will be included in the baseline SRQ. A state-of-the-art ensemble machine learning method will be used to estimate HTE.


The study is innovative in using a rich baseline assessment and in having a sample large enough to carry out a well-powered analysis of heterogeneity of treatment effects. We anticipate finding that self-guided and guided i-CBT will both improve outcomes compared to ADM only. We also anticipate finding that the comparative advantages of adding i-CBT to ADM will vary significantly across patients. We hope to develop a stable individualized treatment rule that will allow patients and treatment providers to improve aggregate treatment outcomes by deciding collaboratively when ADM treatment should be augmented with i-CBT.

Trial registration NCT04120285. Registered on October 19, 2019.

Peer Review reports

Administrative information

Note: the numbers in curly brackets in this protocol refer to SPIRIT checklist item numbers. The order of the items has been modified to group similar items (see

Title {1}

The Appalachian Mind Health Initiative (AMHI): A pragmatic clinical trial of adjunctive internet-based Cognitive Behavior Therapy for treating major depressive disorder among primary care patients

Trial registration {2a and 2b}. Identifier: NCT04120285

Protocol version {3}

Initial version date: 10/15/20

Funding {4}

This trial is funded by the Patient Centered Outcomes Research Institute (PCORI).

Author details {5a}

Robert M. Bossarte, PhD; West Virginia University Injury Control Research Center and Department of Behavioral Medicine and Psychiatry, West Virginia University School of Medicine, Morgantown, WV

Ronald C. Kessler, PhD; Department of Healthcare Policy, Harvard Medical School, Boston, MA

Andrew A. Nierenberg, MD; The Dauten Family Center for Bipolar Treatment Innovation, Massachusetts General Hospital and Harvard Medical School, Boston, MA

Pim Cuijpers, PhD; Department of Clinical, Neuro and Developmental Psychology, Amsterdam Public Health research institute, Vrije Universiteit Amsterdam, Van der Boechorststraat 7-9, 1081 BT, Amsterdam, The Netherlands

Angel Enrique, PhD; E-mental Health Research Group, School of Psychology, University of Dublin, Trinity College Dublin and Clinical Research & Innovation, SilverCloud Health, Dublin, Ireland

Phyllis M. Foxworth, BS; Depression and Bipolar Support Alliance, Chicago, IL

Sarah M. Gildea, BS; Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts

Bea Herbeck Belnap, PhD; Center for Behavioral Health, Media, and Technology, University of Pittsburgh School of Medicine

Marc W. Haut, PhD; Department of Behavioral Medicine and Psychiatry, West Virginia University School of Medicine, Department of Neurology, West Virginia University School of Medicine and Department of Radiology, West Virginia University School of Medicine, Morgantown, WV

Kari B. Law, MD; Department of Behavioral Medicine and Psychiatry, West Virginia University School of Medicine, Morgantown, WV

William D. Lewis, MD; Department of Family Medicine, West Virginia University School of Medicine and West Virginia University Clinical and Translational Science Institute, Morgantown, WV

Howard Liu, SD; Department of Health Care Policy, Harvard Medical School, Boston, MA and Center of Excellence for Suicide Prevention, Canandaigua VA Medical Center, Canandaigua, NY

Alexander R. Luedtke, PhD; Department of Statistics, University of Washington and Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, Seattle, WA

Wilfred R. Pigeon, PhD; Center of Excellence for Suicide Prevention, Canandaigua VA Medical Center, Canandaigua, NY and Department of Psychiatry, University of Rochester Medical Center, Rochester, NY 14642 USA

Larry A. Rhodes, MD; Department of Pediatrics, West Virginia University School of Medicine and West Virginia University Institute for Community and Rural Health, Morgantown, WV

Derek Richards, PhD; E-mental Health Research Group, School of Psychology, University of Dublin, Trinity College Dublin and Clinical Research & Innovation, SilverCloud Health, Dublin, Ireland

Bruce L. Rollman, MD, MPH; Center for Behavioral Health, Media and Technology, University of Pittsburgh, Pittsburgh, PA

Nancy A. Sampson, BA; Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts

Cara M. Stokes, PhD; West Virginia University Injury Control Research Center and Department of Behavioral Medicine and Psychiatry, West Virginia University School of Medicine, Morgantown, WV

John Torous, MD; Department of Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts

Tyler D. Webb, MSW; West Virginia University Injury Control Research Center, Morgantown, WV

Jose R. Zubizarreta, PhD; Department of Health Care Policy, Harvard Medical School, Boston, MA, Department of Statistics, Harvard University and Department of Biostatistics, Harvard University, Cambridge, MA,

Name and contact information for the trial sponsor {5b}

Patient Centered Outcomes Research Institute (PCORI)

1828 L Street NW

Suite 900

Washington, DC 20036


Role of sponsor {5c}

PCORI funds this trial. The funder is not involved in study design, study execution, writing of reports or the decision to submit reports for publication.


Background and rationale {6a}

Major depressive disorder (MDD) is one of the most burdensome of all disorders [1]. Indeed, the Global Burden of Disease study ranks MDD as the 2nd top cause of disease morbidity in the USA [2] due to the combination of its high prevalence and high impairment. MDD is associated with high work disability, absenteeism, and lost work productivity [3] and is also a powerful risk factor for suicide [4]. Based on these results, the annual economic burden of MDD in the USA is estimated to be $210 billion [5], but this estimate omits indirect costs such as associations of MDD with increased risk of subsequent onset [6] of chronic physical disorders and increased persistence severity of such secondary physical disorders when they occur [7]. Other important indirect costs associated with reduction in quality of life [8], social role functioning [9], and burdens experienced by family members [10]. MDD also has substantial and diverse negative effects on the health and well-being of the children of parents with depression that can be reversed with successful treatment of parental MDD [11].

Estimates from the most recent US Medical Expenditures Panel Surveys suggest that 8% of US adults receive MDD treatment over a 12-month time period, with 87% receiving antidepressant medication (ADM), 23% psychotherapy, and 10% combined ADM-psychotherapy [12]. A number of ADM classes exist, but none is consistently superior to others, resulting in ADM treatment recommendations being based largely on tolerability and safety [13]. A number of evidence-based psychotherapies also exist, with little evidence of differences in effects, but cognitive behavioral therapy (CBT) has the most consistent evidence of effectiveness because it has most often been studied and can most reliably be implemented [14, 15]. Controlled trials comparing ADM to face-to-face CBT find generally comparable aggregate effects [16]. However, controlled trials typically examine only aggregate effects and do not consider the possibility that patients might differ in the treatment that is most helpful to them. The growing amount of research that investigates this possibility of heterogeneity of treatment effects (HTE) finds considerable evidence that HTE exists for MDD treatment [17]. For example, one small trial studying MDD HTE estimated that a clinically significant advantage of either CBT or ADM over the other exists for more than 60% of primary care patients with MDD, suggesting that optimal treatment selection when both CBT and ADM are available would result in a substantial increase in treatment response [18]. Another group of trials randomized patients either to ADM, CBT, or combined ADM-CBT and found that combined treatment had a roughly 50% higher aggregate MDD symptom remission rate than either of the monotherapies [19]. HTE analyses in these studies documented over two dozen consistently significant predictors of HTE across the broad categories of ADM only, CBT-only, and combined treatment, but only a handful of these predictors were included in any single trial and no attempt has ever been made to develop a comprehensive HTE model with all these predictors [20].

Other trials compared guided internet-based CBT (i-CBT) and found aggregate effects generally comparable to those of face-to-face CBT, but at much lower cost [21, 22]. Guided i-CBT is internet-based CBT completed by the patient with a remote guide or coach who communicates with the patient via email, text, or telephone. Coaches also provide elements of remote collaborative care case management [23, 24], such as encouraging ADM adherence, monitoring ADM side effects and treatment response, coordinating with the primary care physician, and facilitating specialty referral. A limitation of these trials, though, was that no HTE analyses were carried out, making it impossible to determine if i-CBT is helpful in all cases or if its value is limited to a subset of patients that can be well-defined before the beginning of treatment [25]. A final relevant group of trials found that self-guided i-CBT had worse aggregate effects than guided i-CBT [25, 26], but significantly better effects than a waiting list control group [27,28,29]. HTE analyses were not carried out, although the effect of self-guided i-CBT was comparable across levels of baseline MDD symptom severity [30]. Self-guided i-CBT is CBT completed by the user on the internet with computerized feedback but no clinician involvement after an initial orientation meeting.

Six major gaps in evidence exist in the above trials. First, few of them evaluated patient-centered outcomes. Second, the trials that compared ADM only and CBT-only with combined ADM-CBT lack external validity. That is, their results are limited to the small proportion of patients that agreed in advance to be treated either with ADM only, CBT-only, or combined ADM-CBT. This makes it impossible from these trials to evaluate the incremental effect of CBT over ADM only among patients with a strong preference for ADM who would be willing to try CBT in addition to ADM even though they would not be willing to try CBT in the absence of ADM. Trials that add CBT to ADM among patients receiving ADM would be needed to do that [31]. This is what we are doing in our trial. Third, prior trials comparing combined ADM-CBT to ADM only were limited largely to the evaluation of face-to-face CBT, which is an unrealistic option for the great majority of patients with depression due to the limited and declining number of psychotherapists in the population [32] and the fact that only a minority of psychotherapists are fully trained in CBT. Fourth, HTE analyses were limited to only a handful of prescriptive predictors in any single study [33]. Fifth, when more than a handful of prescriptive predictors were considered [18, 34], the sample sizes were too small to generate stable multivariate HTE estimates, leading to a focus on dimensional outcomes and using statistical methods that almost certainly over-fit the data. Sixth, HTE analyses usually used suboptimal analysis methods.

We plan to address all these gaps in the Appalachian Mind Health Initiative (AMHI), a pragmatic trial of the comparative effectiveness of two levels of remote i-CBT added to ADM compared to ADM only. The sample will consist of 3360 patients seeking primary care MDD treatment in West Virginia. AMHI is being carried out by researchers at West Virginia University (WVU) School of Medicine. An evaluation of the incremental benefit of i-CBT is of special importance in West Virginia for several reasons. First, combined ADM-psychotherapy is recommended in some treatment guideline for patients with moderate-severe MDD [35] and comorbidity [36], both of which are elevated in West Virginia due to the state having the 2nd lowest per capita income in the country [37], the highest proportion of residents covered by Medicaid in the country [38], and the highest opioid death rate in the country [39]. West Virginia is also the 2nd most rural state in the country [40]. This confluence of factors results in the proportion of patients with MDD in West Virginia receiving psychotherapy being only about half the national average. This is part of a larger pattern in which West Virginia ranks only 42nd across the 50 states in overall mental health care [41], with the great majority of treatment occurring in primary care settings and consisting of ADM only. Patients who access psychotherapy typically do so only after being on long waiting lists (often 3+ months) and traveling substantial distances to receive treatment. Access to telephone or videoconference psychotherapy is limited. Yet 75% of primary care patients with depression express a desire for psychotherapy either alone (40%) or in combination with ADM (35%) [42]. This mismatch between treatment availability and preference is important because MDD remission increases substantially when patients are not treated with their preferred type of treatment [43,44,45]. There is thus good reason to believe that providing access to i-CBT will improve MDD treatment outcomes in our trial.

Objectives {7}

The first objective of AMHI is to evaluate the aggregate incremental effects of combining either best practices guided or self-guided i-CBT to ADM among patients seeking primary care treatment for MDD in West Virginia. The second objective is to determine whether stable predictors can be found of HTE in order to develop a clinical decision support system that generates an individualized treatment rule (ITR) to help patients and clinicians decide whether to add either self-guided or guided i-CBT to ADM primary care treatment of MDD. The ITR will also help identify patients for whom ADM only and combined ADM-i-CBT delivered in primary care both have low probabilities of resulting in MDD remission. A third (exploratory) objective is to use nonexperimental methods to investigate HTE with respect to two major uncontrolled aspects of MDD treatment: type of ADM; and i-CBT versus live psychotherapy (the latter obtained by 12% of primary care MDD patients in West Virginia). We hypothesize that substantial additional HTE will be documented in these nonexperimental analyses and that treatment selection across both randomized and major non-randomized aspects of treatment based on knowledge of this HTE could increase the MDD remission rate significantly.

Trial design {8}

AMHI will be a three-arm single-blind individually randomized equal allocation controlled pragmatic trial. It will compare aggregate superiority of ADM plus either self-guided or guided i-CBT over ADM only. It will also evaluate the extent to which HTE exists. An extensive baseline internet-based patient self-report questionnaire (SRQ) will be administered prior to randomization. Results will be used to randomize eligible patients across study arms using the finite selection model [46]. As detailed below, i-CBT will typically be completed within 3 months. Brief SRQs will be administered at 2, 4, 8, and 13 weeks to monitor intervention uptake and continued engagement. SRQs at 16 weeks will be used to determine remission (the primary outcome) and various aspects of treatment response (secondary outcomes). Subsequent SRQs at 26, 39, and 52 weeks will be used to monitor maintenance of remission and longer-term outcomes among patients that did not remit previously. Phone calls will be used to obtain patient-reported outcomes data when the internet-based SRQs are not completed. Telephone assessors will be blinded. Informed consent will be obtained to merge electronic medical records (EMR) with self-report data to enrich the dataset and adjust for SRQ loss to follow-up.

Methods: participants, interventions, and outcomes

Study setting {9}

Primary care facilities from three established networks throughout the state of West Virginia are being recruited to participate in AMHI: The West Virginia Practice Based Research Network (WVPBRN); the West Virginia Primary Care Association (WVPCA) network; and the anticipated inclusion of the Veterans Health Administration (VHA) system. The WVPBRN is a group of primary care practices with the majority considered Federally Qualified Health Centers. The WVPCA is a private, non-profit membership association that represents West Virginia safety-net health care providers. The WVPCA is also the federally designated primary care association for the state and is the link between federal, state, and local entities providing healthcare for 25% of the state’s residents. The VHA is the largest health care system in the nation and has a strong presence in West Virginia [47] due to the fact that West Virginia has a much higher concentration of Veterans than most states [48]. Recruitment of sites is still under way. and the final set of participating sites is to be determined. The target is to recruit 50 practices with a total of 100 participating clinicians.

Eligibility criteria {10}

Table 1 lists inclusion and exclusion criteria. The criteria were chosen to recruit a broadly representative sample of primary care patients under treatment for MDD in primary care settings throughout the state. Patients seeking primary care treatment for MDD will need to be in treatment for depression the first time in the past 6 months to be eligible, as the focus is on new episodes of treatment. They will need to be adults (aged 18+) and have a level of severity that does not require hospitalization. They will differ widely in severity, course of illness, and comorbidities. Given that the trial will focus on patient-reported assessments, patients will be required to be literate in English, have access to a telephone, and either have access to a smart phone or computer or be willing to travel to access a tablet computer at their doctor’s office for periodic SRQs. In addition to the exclusion of patients with treatment of MDD within the prior 6 months or current need for inpatient treatment, patients will be excluded if they have an impairment that would interfere with completing the study tasks (i.e., hearing or vision loss), a history of either bipolar disorder or psychosis (based on either EMRs or baseline self-report), or acute serious suicide risk based on self-report of suicide ideation with active suicide intent.

Table 1 Appalachian Mind Health Initiative (AMHI) inclusion and exclusion criteria

Who will take informed consent? {26a}

Potential study participants will be identified either at the clinics on the day of initial treatment contact or upon review of records by clinic staff at the end of the workday. If identified while at the clinics, a study fact brochure will be provided to the patient after obtaining preliminary consent for study staff to call them at home to explain the study in more detail. The brochure will contain the study website URL and an 800 number for additional questions or to opt out prior to receiving a call from a study staff member. Potential participants missed during the day will be determined by clinic staff based on review of records at the end of each workday. A letter will be sent to these patients along with the study fact brochure explaining the study and informing these patients that a study staff member will call to explain the study. The letter will also include the study 800 number for patients that want to opt out. Contact information for potential participants will then be sent to the study team using a secure web application in the patient’s electronic medical record. This information will be password protected and access will be limited to those with a need to know. Trained WVU research staff will then contact consented patients within 24 h of their clinic visit to explain the study, answer questions, and obtain verbal informed consent. All research staff involved in consenting participants will complete the Collaborative Institutional Training Initiative training. Only those who have satisfactorily demonstrated the ability to follow consenting protocols and procedures will be approved to consent participants. Electronic informed consent will then be obtained prior to the beginning of the baseline internet-based patient SRQ.


Explanation for the choice of comparators {6b}

As noted in the introduction, the great majority of patients with MDD prefer psychotherapy either instead of or in addition to ADM [42]. Yet only a small minority (12% in West Virginia; 23% in the total USA) of MDD patients receive psychotherapy [12]. This is a problem both because MDD treatment preference is a strong predictor of treatment response [43,44,45] and because combined ADM-psychotherapy is known to be more effective than ADM only, especially among patients with moderate-severe depression and psychiatric comorbidity [19]. Face-to-face CBT, the most widely studied evidence-based psychotherapy, is not a realistic option for most MDD patients, especially in a poor rural state like West Virginia, making it important to know whether less expensive and easily scalable i-CBT would improve treatment outcomes if added to ADM compared to ADM only. We consequently decided to focus on a three-arm trial comparing treatment outcomes associated with ADM only versus ADM combined with either self-guided or guided i-CBT. Following a review, the specific version of i-CBT we selected is SilverCloud [49], a leading evidence-based digital i-CBT program. SilverCloud was selected based on its extensive evidence base [50,51,52,53,54,55,56,57,58] and the fact that it can be delivered in either self-guided or guided forms.

Intervention description {11a}

SilverCloud is a transdiagnostic guided i-CBT platform with 30 programs that can be tailored to the specific needs of users. Programs can be repeated if the user (or, in the case of the guided version, the coach) feels that this would be useful [55]. All programs are available 24/7. Participants in the trial assigned to i-CBT (either self-guided or guided) will all receive the SilverCloud program designed for patients with depression, which is described in Tables 2 and 3. This program is designed to relieve symptoms of depression by teaching more flexible ways of thinking, increasing awareness and understanding of emotions, and increasing activity and motivation in daily life. The program consists of 8 modules, each taking 45–60 min to complete. Users are recommended to complete one module per week and to break up each module into 3–4 sessions of 10–20 min each. In the case of the guided version, coaches provide asynchronous post-session feedback on the work patients have completed on the platform and provide personalized recommendations of content for the users. Coaches can also suggest that users revisit some sections within the module or prior modules within the week based on open-ended text provided by users, whereas this kind of tailoring relies on user selections from menus provided in the self-guided version of the program in addition to weekly email messages sent through the platform for up to 10 weeks. Coaches will be BA-level graduates of behavioral health programs who have been trained in the SilverCloud platform and in how to deliver feedback. In addition to the core depression program, patients can be provided with unlockable modules depending on concomitant issues and difficulties they may be experiencing, such as with sleep, self-esteem, and communication.

Table 2 SilverCloud Space from Depression i-CBT 8-module overview
Table 3 SilverCloud Space from Depression i-CBT modules, topics, goals, and activities

Criteria for discontinuing or modifying allocated interventions {11b}

Patients randomized into the trial will be monitored via 8 internet-based tracking SRQs at 2, 4, 8, 13, 16, 26, 39, and 52 weeks after randomization regarding important changes in their symptomology and the need for communication with their primary care provider or implementation of a crisis management strategy (i.e., a participant expresses intent to harm themselves or others). Each tracking survey will contain questions assessing suicidality within the past 2 weeks. Participants who report thinking of suicide or death several times a day in some detail with at least some intent of acting on these thoughts will receive a closing statement at the end of their survey encouraging them to contact the National Suicide Prevention Lifeline (1-800-273-TALK) and/or online chat services (via We will also provide the participant’s contact information to the National Suicide Prevention Lifeline for outreach, and we will generate a report to the participant’s primary care treatment provider. The Principal Investigator (PI) or designee will then notify the participant’s primary care treatment provider about the incident. The reviewing primary care treatment provider will then determine whether the participant should remain in the study. Other reasons for discontinuation from the study will be a patient’s need for hospitalization as assessed by their primary care treatment provider, clinically significant adverse events not consistent with continuation in the study as determined by the study team or the participant’s primary care treatment provider.

Strategies to improve adherence to the interventions {11c}

i-CBT users often exhibit low levels of engagement, especially in disadvantaged populations [59]. We will address this problem using several strategies. First, patients assigned to i-CBT will both be (i) sent email messages notifying them of these assignments and (ii) receive a phone call from study staff using motivational interviewing techniques to encourage engagement [60]. Second, the first two brief SRQs, at 2 and 4 weeks after randomization, will be used as additional occasions to monitor intervention engagement. Patients that either fail to respond or (in the case of the i-CBT arms) report that they have not yet started the intervention will receive additional motivational interviewing contacts to encourage engagement with the assessments and (in the i-CBT arms) the interventions. Consistent with previous research [61, 62], we anticipate that these frequent contacts will increase engagement. Third, we will attempt to build on a recent process analyses of meta-data from over 50,000 SilverCloud users, which identified five early longitudinal user engagement profiles that predict treatment response [63]. To the extent possible, we will score these clusters for each patient assigned to SilverCloud and use the profile scores to target the subset of patients identified as non-engagers in weeks 3–4 of the trial for additional motivational outreach email messages.

Relevant concomitant care permitted or prohibited during the trial {11d}

Eligible patients will be treated for major depression by their primary care physician. We will not control type or dose of ADM prescribed, but we will track dose, titration, augmentation, and switching through EMRs. Some patients will also be treated with other psychotropic medications for comorbid conditions, such as anxiolytics for comorbid anxiety disorders, stimulants for ADHD, or addiction medications for comorbid substance use disorders, and these will be allowed. However, patients treated with anti-mania medications or antipsychotics will be ineligible for the trial due to the exclusion of patients with a history of bipolar disorder or psychosis.

Provision for post-trial care {30}

Access to SilverCloud will continue for 12 months after randomization for patients randomized to the two i-CBT arms.

Outcomes {12}

The primary outcome will be MDD remission at 16 weeks as defined by a revised version of the composite patient-centered Remission from Depression Questionnaire (RDQ) [64, 65]. The RDQ is a patient self-report scale that assesses the 7 dimensions found in extensive research to be the ones most important to patient-centered definitions of MDD recovery: [66,67,68] remission from depressive and non-depressive (e.g., anxiety) symptoms, positive mental health, coping ability, productive and social role functioning, life satisfaction, and global sense of well-being. The RDQ is the best validated patient-reported outcome measure of these 7 dimensions. It has excellent psychometric properties, is as sensitive to change as symptom-based scales, but captures additional information [64, 69]. Importantly, patients consistently say that the RDQ represents their treatment goals more than standard symptom scales [70]. Based on evidence of a strong second-order factor across the 7 RDQ dimensions [64], an aggregate score can also be derived. Our primary outcome will be a dichotomous definition of remission based on this aggregate score at 16 weeks and of maintenance of remission at 26, 39, and 52 weeks. The cutoff for this designation in the RDQ was calibrated by the RDQ developers to balance false positives and false negatives in using composite RDQ scores to predict patient reports of “being completely back to normal.” This calibration exercise will be replicated in AMHI to guarantee the internal validity of results. In addition, the depression symptom severity scale used in AMHI will be different from the one in the RDQ: the Quick Inventory of Depressive Symptomatology Self-Report (QIDS-SR) [71]. The QIDS-SR will be used because of its strong association with the gold standard Hamilton Rating Scale for Depression (HRSD) [72] and the existence of a validated crosswalk between the QIDS-SR and HRSD [73]. The focus on a dichotomous measure of remission as the primary outcome is based on evidence that MDD remission is critical for reducing recurrence, leading treatment guidelines to call for MDD to be treated to remission [35, 74]. The 7 dimensional RDQ and QIDS-SR scores will be secondary outcomes. Given the prominence of substance use disorders in West Virginia [75] and the high comorbidity of MDD with these disorders in the population [76], we will also include a substance use disorder symptom scale [77] as a secondary outcome. Anxiety comorbidity, while also of interest, is already addressed in one of the RDQ dimensions. Finally, a measure of ADM treatment compliance based on EMR data, measures of treatment engagement based on both self-reports and administrative records [78], and a patient-reported measure of shared decision-making [79] will be additional secondary outcomes.

Participant timeline {13}

The participant timeline is outlined in Fig. 1. Potential study participants identified at participating clinics during their initial visit will be informed of the study by their treatment provider. Additionally, they will be provided with a study fact brochure, which contains highlighted study information as well as the study website URL and an 800 number for additional questions. Participating clinics will then gain permission from their potentially eligible participants to provide AMHI study staff with their contact information for enrollment. Potentially eligible patients who were not informed of the study during their initial treatment visit will be sent a letter with the signature of their treatment provider along with the study fact brochure explaining the study and informing these patients that a study staff member will call to further explain the study and discuss possible enrollment. The letter will also include the study 800 number for patients that want more information or to opt out. AMHI study staff at WVU will then contact, gain informed consent, and subsequently enroll each participant into the study. Upon receipt of informed consent, potentially eligible participants will be emailed a link to complete the baseline SRQ and an online web challenge to evaluate cognitive performance. Given the nature of eligibility criteria, a final decision about eligibility will be made only after completion of the baseline SRQ and online web challenge. Eligible participants will then be randomized to one of the three treatment arms. The finite selection model will be used to increase balance in baseline covariates across treatment arms [46]. Email will be used to notify patients of these assignments. Participants in the i-CBT arms will then receive a phone call from study staff using motivational interviewing techniques to encourage engagement with the intervention [55]. Regardless of the arm the participant is randomized to, subsequent SRQs will then be administered at 2, 4, 8, 13, 16, 26, 39, and 52 weeks post randomization. Participants that fail to complete SRQs will receive a set of reminder emails encouraging completion. Additionally, participants that fail to complete SRQs in weeks 2, 16, 26, and 52 will receive phone calls using motivational interviewing techniques to complete SRQs.

Fig. 1
figure 1

Participant timeline

Sample size {14}

We will enroll a target enrollment sample of 3360 patients who complete the baseline SRQ and are randomized across the three treatment arms (i.e., 1120 per arm). The sample was powered to detect predictors of HTE that would increase the proportional remission rate by 20% by optimally assigning individuals as opposed to randomly assigning them into three treatment groups of equal size. A much smaller sample size would be adequate to address the first study objective of evaluating the significance of differences in aggregate remission rates across the 3 treatment arms. Meta-analyses of previous trials show that the MDD symptom remission rate averages about 25% among MDD patients randomized to ADM only and 37.5% among patients randomized to combined ADM-psychotherapy [20]. Although there were no meta-analyses of remission rates for self-guided i-CBT at the time AMHI was planned, other meta-analyses showed that effects of self-guided i-CBT were significantly better than controls [30] but worse than guided i-CBT [25, 26]. We consequently assumed, for purposes of power calculations, that the patient-centered remission rate for ADM plus self-guided i-CBT in AMHI would be 31%, which is roughly midway between the extremes. Power to detect each of the 6% differences between the extremes and the middle category using a 0.05-level 1-sided test and assuming 30% loss to follow-up (which is typical of i-CBT trials) is .86, whereas power to detect the 12.5% difference between the two extremes using the same specifications is .99.

Evaluating power to detect HTE is more complex and requires simulation. We did this by beginning with data on the observed distributions and exogenous associations among the baseline predictors in the STAR*D trial [80] and assumed that each patient was randomized across three treatment arms. We then specified a series of relatively complex nonlinear-interactive multivariate classification models for associations of these predictors with MDD remission, assuming plausible prognostic and prescriptive coefficient values that generated population distributions with the same aggregate outcome prevalence as assumed above [81]. We then modified the prescriptive coefficients to retain the aggregate remission rates while embedding HTE in the population that would result in a 20% proportional increase in the aggregate remission rate under randomization with equal allocation (i.e., from 31% to 1.2 × 31% = 37.5%). We then drew 500 pseudo-samples of different sizes from this simulated population and applied our HTE analysis method (see below) to estimate HTE in these sample data. We assumed that all the significant prognostic and prescriptive predictors were measured in the samples in addition to 20 noise predictors. Power was calculated as the proportion of replicates in which the lower bound of the 95% CI of the HTE estimate was greater than 0.0%. We also calculated proportional regret (i.e., downward bias in the estimated versus true proportional increase in remission under optimization vs. randomization). The sample size was set to yield power greater than .80 and proportional regret less than .20.

Recruitment {15}

We anticipate a 24-month recruitment period. As outlined in Table 4, initial eligibility will be determined during the patient’s clinic visit. Potentially eligible participants will be provided with a study fact brochure, which contains highlighted study information as well as the study website URL and an 800 number for additional questions. Participating clinics will then gain permission from eligible participants to provide AMHI study staff with their contact information. Eligible participants who were not informed (i.e., missed in recruitment) will be sent a letter explaining the study along with a study fact brochure under the signature of their treating clinician. This letter will contain the study 800 number for patients who would prefer to opt out. Patients who do not opt out will receive a telephone call from AMHI study staff to reassess eligibility, review material in the study fact brochure, and answer patient questions before seeking electronic informed consent to participate in the study. Patients who prefer to physically sign the informed consent will be mailed two hard copies (one for their records) along with a pre-addressed pre-stamped return envelope.

Table 4 Participant recruitment

Assignment of interventions: allocation

Sequence generation {16a}

Baseline SRQ results will be used to stratify randomization of eligible patients across study arms. This will be done initially using the 3-way cross-classification of depression symptom severity, chronicity, and comorbidity based on a SAS macro that will be run daily by the WVU study data manager until 100 participants are assigned per arm. The finite selection model [46] will be used subsequently to randomly assign patients and balance on a larger set of variables, again with the WVU study data manager implementing the procedure daily.

Implementation {16c}

Initial stratification by the 3-way cross-classification of depression symptom severity, chronicity, and comorbidity will be implemented by a SAS macro that is independent of the WVU data manager that implements the assignment. The subsequent stratification based on the finite selection model will also be implemented by computer, again independent of the WVU data manager that implements the assignment.

Assignment of interventions: blinding

Who will be blinded {17a}

Clinical staff at primary care facilities and telephone recruiters will be blinded at the time of recruitment to treatment assignment, which will take place after recruitment is complete. Telephone interviewers that obtain self-report information from participants who did not complete their SRQ will also be blinded to treatment arm.

Procedure for unblinding if needed {17b}

If concerning symptomology emerges as defined in the Data Safety and Monitoring plan, the participant will be flagged and reported to the PI for review. If a participant’s SRQ indicates acute serious suicide risk, a closing statement in the SRQ will encourage the participant to contact the National Suicide Prevention Lifeline and will inform the participant that both their treating physician and the National Suicide Prevention Lifeline are being informed of their suicidality.

Data collection and management

Plans for assessment and collection of outcomes {18a}

Assessments will be carried out with patient-reported SRQs augmented by EMRs that will provide information about relevant baseline information (e.g., information about treatment history prior to the intervention that might be relevant in predicting outcomes or HTE) as well as treatment information over the course of the trial involving both the ADM (type, dose, titration, switching, augmentation) and other treatments that might be relevant to outcome assignment among patients lost to follow-up in the SRQ assessments (e.g., psychiatric hospitalizations, emergency department visits for mental health crises, suicide attempts, suicide deaths, other deaths by external cause). Additionally, assessments will be gathered via smartphone with consenting participants capturing ecological momentary assessments as well as sensor data from the smartphone app mindLAMP [82, 83] that can be used to derive estimations for daily physical activity, sedentary activity, sleep duration, screen time exposure, and social behaviors. Participants will be asked to download an Apple or Android version of mindLAMP onto their personal smartphones for these measures.

The SRQs will be collected remotely by a health survey firm that has extensive experience implementing mixed-mode web-phone surveys. Computerized procedures will be used to automate skip logic, flag missing values for completion, disallow out-of-range and inconsistent responses, and discourage superficial responses. This survey firm will also carry out the telephone interviews with SRQ nonrespondents. Each SRQ will be sent by email. Reminders to initial nonrespondents will be sent 3 and 6 days later. Options will be provided for participants who want to break up an SRQ for completion over multiple sessions. Telephone follow-up calls and interviews with initial SRQ nonrespondents will be carried out by the survey firm in conjunction with the SRQs at 2, 16, 26, and 52 weeks. The survey firm will also maintain an 800 number for technical assistance.

As noted above, more than two dozen consistently significant baseline patient-reported predictors of MDD HTE have been documented in the literature [33]. These are outlined in Table 5. We developed a baseline self-report SRQ to assess these predictors by carrying out a systematic literature review of the best available short-form patient-reported measures of each construct. We also worked with statisticians and psychometricians to develop optimal short-form scales in secondary analyses to create new short-form versions of existing scales when none already existed [84,85,86]. We made extensive use of intelligent skip logic in designing the baseline SRQ to shorten the assessment once scale scores in a relevant range could be inferred from partial responses.

Table 5 Baseline predictors

We also noted above that the primary outcome in AMHI will be remission defined by a modified version of the composite RDQ [64] that substitutes the QIDS-SR [71] for the RDQ depression symptom severity scale. Secondary outcomes in addition to the component dimensional RDQ and QIDS-SR scores will include a substance use disorder symptom scale [77], a measure of ADM treatment compliance, measures of patient engagement in treatment based on both self-reports and administrative records [78], and a patient-reported measure of shared decision-making [79], all of which are based on widely-used self-report scales that have good psychometric characteristics as shown in Table 6.

Table 6 Self-report questionnaire (SRQ) outcomes

Plans to promote participant retention and complete follow-up {18b}

It will be made clear to participants at the onset of the recruitment process that they will be expected to complete a series of 10 study tasks (baseline SRQ and online challenge tasks to evaluate cognitive performance; and follow-up SRQs at 2, 4, 8, 13, 16, 26, 39, and 52 weeks). We will use a citizen-scientist model to appeal to participants to complete as many of these assessments as possible, emphasizing the special importance of the 16-week and 52-week assessments. Participants will be paid for their time completing all assessments, including $50 for the baseline SRQ and $50 for the cognitive challenge task, $50 for the 52-week SRQ, and $20 for each of the other SRQs. Reminder emails and texts will be used to increase response. In the case of the 2-, 16-, 26-, and 52-week assessments, we will also use telephone reminder calls both to encourage SRQ completion and to collect this information via telephone interview when we cannot do so by SRQ.

Data management {19}

The WVU study team will maintain a master dataset for all patients who were referred to the project for recruitment along with their dispositions (i.e., withdrew from participation before or after completing the baseline SRQ or after initiating the intervention, were judged ineligible before or after completing the baseline SRQ, were terminated from the study after initiating the intervention, continued throughout the study). Research ID numbers will be assigned separately to this disposition file and to a file containing all personally identifying information (PII) but the PII will not be linked directly to the disposition file. The WVU team will also create a master EMR data file that contains research ID numbers in addition to EMR data for all participants but contains no PII. The master EMR data file will contain information abstracted directly from the EMRs of participants who provide consent to share this information. AMR data will be managed as a SAS data file stored on a HIPPA compliant server maintained by WVU. The survey firm will maintain a separate consolidated master SRQ data file for each participant that contains the same research ID numbers as those used by the WVU team but contains no PII. A separate secure datafile will be maintained by the survey firm that contains only the research ID and the PII of each participant. The de-identified master EMR data file and the master SRQ data file will be merged for data processing by the WVU study team. All data analyses will be carried out with this de-identified consolidated data file jointly by the WVU and Harvard Medical School (HMS) collaborators. Access to participant data will be restricted to members of the study team listed on the IRB-approved protocol.

Confidentiality {27}

Access to PII will be restricted to study staff identified on the IRB-approved protocol. When appropriate, information on risk for harm to self or others or significant worsening of symptomology will be reported to the patient’s primary care clinician and, in the case of acute serious suicidality, to the National Suicide Prevention Lifeline. A Certificate of Confidentiality has been obtained for this study from National Institutes of Health.

Statistical methods

Statistical methods for primary and secondary outcomes {20a}

The WVU analysis team will construct summary EMR variables for patients and treatment providers behind the WVU firewall and transfer these data to the survey firm via secure file transfer for linkage with the SRQ dataset behind the survey firm’s firewall. These data will be de-identified before returning to WVU and HMS for analysis. Objective 1 analyses will evaluate aggregate differences across the treatment arms. We will use logistic regression to estimate binary outcomes and report adjusted prevalence ratios with design-adjusted 95% confidence intervals (CIs). We will calculate number needed to treat (NNT) for each comparison. Generalized linear models will be used to estimate effects on continuous outcomes, making use of standard visual diagnostics to choose appropriate link functions and error structures [181]. We will report adjusted mean differences with design-adjusted 95% CIs [182].

Objective 2 analyses will evaluate HTE across the three randomized treatment arms using a special case of the super learner (SL) algorithm [183], an ensemble machine learning approach that uses cross-validation (CV) to select a weighted combination of predicted outcome scores across a collection of candidate algorithms that yields an optimal weighted combination according to a pre-specified criterion that performs at least as well as the best component algorithm. The candidate algorithms in SL can either be parametric or flexible machine learning algorithms, making SL less prone to model misspecification than traditional parametric approaches. The guarantee that SL performs at least as well as the best candidate algorithm allows a rich library of parametric and flexible candidate algorithms to be included.

In the conventional approach to estimating HTE, a model with main effects and interactions between prescriptive predictors and dummy variables for treatment indicators is estimated. Predicted values based on this model are then used to estimate the expected individual-level outcome conditional on the values of the prescriptive predictors for each patient in each treatment condition (e.g., the estimated outcomes of patient p under treatment arm a, arm b, and arm c). An estimate of the optimal treatment strategy for patient p is then obtained by comparing predicted values of the outcome across all treatment arms. It is important to appreciate that the accuracy of this approach requires correct specification of both the (possibly nonlinear) main effects and the (possibly complex nonlinear and higher-order) interaction terms. SL has two advantages over this conventional approach [184]. First, it requires only correct specification of the interactions. It does not require correct specification of main effects, as it directly estimates contrasts that allow the correct specification of the main effects to be circumvented. Second, unlike earlier approaches to estimating HTE that share this desirable feature [185, 186], SL uses a flexible set of component machine learning algorithms that maximize chances of capturing complex nonlinear and higher-order interactions correctly.

Objective 3 (exploratory) will examine two aspects of treatment that were not randomized: treatment with one of three broad types of ADM (SSRIs, SNRIs, bupropion) for which there is some evidence of HTE, and live psychotherapy combined with ADM rather than i-CBT with ADM. As noted earlier, 12% of West Virginia primary care MDD patients currently receive live psychotherapy. In these analyses, we will estimate a model to predict selection into each nonrandomized type of treatment as a function of baseline covariates to determine if these uncontrolled aspects of treatment are nonrandom with respect to baseline predictors. We will balance on these baseline covariates before estimating the SL model to evaluate the aggregate effects of these aspects of treatment as well as baseline predictors of HTE with respect to these aspects of treatment.

To quantify the potential value of developing a precision treatment rule for i-CBT, we will use an approach that is roughly equivalent to the calculation of NNT in aggregate analyses of treatment effects. Specifically, we will use a cross-validated targeted minimum loss-based estimator (CV-TMLE) [187] of the attained improvement of the mean outcome under a treatment selection scheme that always selects the treatment option with the best predicted outcome compared to the mean outcome under balanced randomization. This CV-TMLE yields an estimator of the attained improvement with minimal bias because it uses CV to separate the estimation of the optimal treatment strategy from the assessment of the estimated strategy’s performance and also by allowing for the incorporation of flexible estimation approaches for the regressions and conditional probabilities needed to define the attained improvement. Given that we will be evaluating the effects of expanding of treatment options rather than deciding between two alternative options (i.e., adding i-CBT to ADM rather than choosing between i-CBT and ADM as alternative monotherapies), we will evaluate a range of decision margins; that is, the expected aggregate effects on overall remission rates associated with patient decisions about adding i-CBT to ADM when individual-level increases in predicted probability of remission are in a given range. In addition, we will quantify the uncertainty in aggregate estimates in CIs.

Interim analyses {21b}

Interim analyses will be conducted only to assess patterns and predictors of attrition for purposes of improving the targeting of motivational interviewing contacts to improve intervention uptake and continued engagement. Ongoing data monitoring will also be used if requested by the Data Safety Monitoring Board to facilitate recommending changes to study activities. Attending clinicians will be notified when reports of suicide thoughts or behaviors are reported and will inform study staff if there is a recommendation to discontinue study activities. Participants who report acute suicidal thoughts or behaviors will be flagged for follow-up. The study team will inform the participant’s treatment provider when there is a patient-reported suicide attempt or ideation with a plan and intent.

Methods for additional analyses (e.g., subgroup analyses) {20b}

As noted above, objective 2 is to carry out subgroup analyses that evaluate the significance of HTE and attempt to develop an ITR to make treatment assignments under balanced allocation that optimize the aggregate remission rate.

Methods in analysis to handle protocol non-adherence and any statistical methods to handle missing data {20c}

Objective 1 analyses will be intent-to-treat [188] analyses using inverse probability weights (IPW) to deal with loss to follow-up [189]. For each time t (weeks 2, 4, 8, 13, 16, 26, 39, 52), we will compute the probability of a participant in the study at time t remaining in the study up through time t+1 conditional on information collected as of time t. We will use flexible, nonparametric estimation methods with variable selection for confounder control [190]. The IPW as of time t+1 will be based on the product of these conditional probabilities up through t+1. The treatment-specific mean outcome will be estimated using these weights for the subjects whose outcomes were observed at t+1. Under a coarsening at random assumption, this estimator converges with increasing sample size to the treatment-arm-specific mean outcome that would have been observed had all subjects remained in the study up through t+1 [191]. When data are missing, we will provide thorough summaries of reasons for missingness, proportions of missing data, and test for differences in the characteristics of participants with and without missing data and we will describe these and the implications of missing data for interpretation when we report the trial’s results. As our IPW approach to missing outcomes data will be based on a missing-at-random assumption, sensitivity analysis will be carried out based on the weaker missing-not-at-random assumption using pattern mixture modeling [192] in a generalized mixed model framework [193]. The predictors of success in obtaining outcome data at the time we evaluate remission (13 weeks after baseline) and maintenance of remission (in later assessments) will include information obtained in prior waves of data collection along with data on intensity of efforts needed to obtain outcome data in a discrete-time survival framework (i.e., in response to the 1st, 2nd, or 3rd e-requests, a subsequent telephone call appeal for response, and later telephone interviews). A range of assumptions about the distribution of the missing data will be made in this approach to investigate sensitivity of results [194]. We will record and report distributions and correlates of dropout and missing data and account for all patients in reports.

Plans to give access to the full protocol, participant-level data, and statistical code {31c}

Only research team members from WVU and Harvard will have access to the final analytic data file. Requests for access to the full study protocol or statistical code should be directed to the PI.

Oversight and monitoring

Composition of the coordinating center and trial steering committee {5d}

The project Scientific Advisory Committee (SAC) will include as members PI Bossarte (Chair) and Co-PI Kessler along with collaborating researchers, clinical care providers, payers, and patient partners with lived experience. The SAC will meet by telephone and Zoom at least monthly for the first 6 months of the project and quarterly thereafter. The roles and decision-making authority of SAC members will be defined collaboratively and clearly stated, such that the first set of meetings will be devoted to establishing roles and expectations, giving each member equal time to describe what they hope to achieve or learn through the study and participation in the SAC and what they hope to offer. The SAC’s main role will be to ensure that a broad spectrum of patients and other stakeholders advise and assist the research team with refining the study questions, outcomes, and protocols.

The project Implementation Monitoring Committee (IMC) will have a similar composition as the SAC but will have the separate task of providing ongoing quality control (QC) monitoring to guarantee that recruitment of participants, intervention implementation, data collection, and report preparation-dissemination follow PCORI principals of being patient-centered. The IMC will meet by telephone and Zoom as needed and at least monthly for the first 9 months of the project and as needed thereafter to review technical assistance calls received by the project 800 number as well as any confusions or complaints to consider opportunities for improving participant experiences.

The SAC and IMC will both monitor authenticity of engagement by using and discussing PCORI’s (the funding agency) Ways of Engaging: ENgagement ACtivity Tool (WE-ENACT) Inventory during meetings, once in the initial months and at least twice annually subsequently. To uphold the PCORI Engagement Principle of co-learning, all researcher and clinician SAC and IMC members will seek to better understand patient populations’ needs and priorities by reviewing the commentaries of patients with lived experience created by our partners in the Depression and Bipolar Support Alliance of West Virginia.

Composition of the data monitoring committee, its role and reporting structure {21a}

The PI along with co-investigators will have overall responsibility for monitoring the integrity of study data and participant safety. In addition, an independent monitoring committee, the Data and Safety Monitoring Board (DSMB), will be established. DSMB members will consist of (1) an expert in mental health research; (2) a clinical researcher experienced in conducting randomized clinical trials for depression; (3) three experts in assessing and treating depression; and (4) a stakeholder with immediate family members diagnosed with mood disorders. All members of the DSMB will either be established PIs, have DSMB experience, lived experience with mood disorders, and/or will be intimately familiar with the safety and ethical concerns related to human subjects in clinical research. The DSMB will review the progress of the trial and safety of participants bi-annually (i.e., two times per year), discuss any safety concerns that have arisen, and make recommendations to improve safety procedures if indicated. At each meeting, the DSMB will evaluate the progress of this project, review data quality, recruitment, and study retention and examine other factors that may affect outcome. The DSMB will review reports of any serious adverse events and/or unanticipated problems that occurred within the past study period. They will review the rates of adverse events to determine any changes in participant risk. The chair of the DSMB will report back to the AMHI investigators and will generate a brief report regarding each meeting for the study record and forwarded for review to the Institutional Review Board (IRB).

Adverse event reporting and harms {22}

The project 800 number will be continually monitored for reports of adverse events and harms. Participants will be informed that the 800 number should be used for this purpose and solicitation of such reports will be sought as part of ongoing contacts with participants in completing SRQs. Messages reported to the study team using the 800 number are immediately delivered to designated members of the study team as an email attachment. Follow-up contact with study participants to assess safety and provide referral to crisis services if needed will occur within 24 h of message notification. Adverse events anticipated in this study include indicators of imminent risk for suicide (e.g., ideation, plans or recent attempt) or need for inpatient psychiatric hospitalization. A log of all reported adverse events, outcomes, and impact on study participation (if any) will be maintained by the study team. Adverse events will be categorized by type (behavioral, psychiatric, social, medical, etc.) and outcomes. In addition, as guided i-CBT coaches and telephone interviewers can be informed about adverse events and harms, these individuals will be instructed to notify the WVU study manager immediately of any such reports. The PI, IMC, DSMB, and IRB will all be notified immediately of each such report. Based on the judgment of the PI in consultation with the Chairs of the IMS, DSMB, and IRB, the IMC will meet as needed to discuss management strategies based on such reports.

Frequency and plans for auditing trial conduct {23}

Study activities, including those related to consent, data collection, and participation in the interventions will be monitored on an ongoing basis by the study IMC and DSMB as well as by the WVU IRB. As noted above, the IMC will have regularly scheduled meetings by telephone and Zoom at least monthly for the first 9 months of the project and as needed thereafter to review technical assistance calls received by the project 800 number as well as any confusions or complaints to consider opportunities for improving participant experiences. The DSMB will meet twice a year and more frequently as necessary to review trial progress and participant safety.

Plans for communicating important protocol amendments to relevant parties {25}

Changes to the study protocol will be communicated during SAC and IMC meetings and as needed to clinical partners and participants.

{26b} This study will not involve collecting biological specimens for storage.

Dissemination plans {31a}

As noted above, the trial is being carried out in collaboration with the Depression and Bipolar Support Alliance of West Virginia (DBSAWV) [195] and the West Virginia Practice Based Research Network (WVPBRN) [196], both of which will collaborate in project dissemination activities. DBSA is the leading peer-directed organization in America focused on depression and bipolar disorder. With nearly 650 peer support groups and 250 chapters nationally, including an active chapter in West Virginia, DBSA reaches millions of people each year, offering support, referrals, and understandable information about the nature of and treatments for these disorders. DBSAWV Executive Director Diana Thompson and 4 DBSAWV members with lived experience of MDD will be members of SAC and IMC and will work closely with project researchers to disseminate results to patients and their familiar in West Virginia and, through the national DBSA, throughout the country. The PBRN has the goal of finding “real solutions for the health problems facing the people of West Virginia” and, to that end, disseminates the results of PBRN projects to healthcare providers throughout the network. In addition, the project team will prepare scientific reports of study results for publication in high-impact journals. In addition, to honor a promise we make to participants to inform them of study results, project staff will prepare and disseminate print materials summarizing study results to participants and will host a series of webinars to present and discuss results with participants.


The AMHI trial has the potential to be of considerable importance in addressing the problem of suboptimal treatment of MDD. We know several things that lead us to this view. First, although combined ADM-psychotherapy yields aggregate MDD remission rates about 50% higher than ADM only [19], only about 10% of MDD patients in the USA receive combined ADM-psychotherapy compared to 77% receiving ADM only [197]. Second, we know that this under-use of combined treatment is not because ADM is preferred over psychotherapy, as 75% of primary care patients with depression express a desire for psychotherapy either alone (40%) or in combination with ADM (35%) [42]. The mismatch is instead due to the low availability of psychotherapy. Given that MDD remission increases substantially when patients are not treated with their preferred type of treatment [43,44,45], there is good reason to believe that providing access to psychotherapy in addition to ADM will improve MDD treatment response substantially. But, third, we recognize that it is not realistic to think that this will happen in the short term due to the limited number of psychotherapy treatment providers in the country coupled with the substantially rising need and demand for treatment associated with COVID-19 [198, 199].The only realistic option is for combined treatment to be implemented using guided i-CBT added to ADM, as guided i-CBT is scalable and inexpensive. Specifically, guided i-CBT can be delivered by BA-level lay coaches, each full-time equivalent of whom can guide the treatment of well over 100 patients. Furthermore, a team of a dozen lay coaches can be supervised by a single psychotherapist, leveraging the skills of the psychotherapist to reach well over 1000 patients per week rather than the 30 a full-time psychotherapist typically treats each week in one-on-one psychotherapy. Importantly, guided i-CBT has aggregate MDD treatment effects that are comparable to those of face-to-face CBT [21, 22].

And it is clear that combined ADM-CBT is not helpful for all patients, given that the number of patients with MDD that remit with combined treatment is only proportionally 50% higher than the number that remit with ADM only. This means that at least two-thirds of patients with MDD are as likely to remit with ADM only as combined ADM-psychotherapy. Knowing which patients are which could be valuable in allocation of guided i-CBT to maxmize cost-effectiveness. In addition, if probability of remission among some patients is lower for combined treatment than for ADM only, the proportion of patients that remit with optimal allocaton would be even greater than 1.5 times the number that remit with ADM only. A similar line of thinking applies to self-guided i-CBT, which might be equally or perhaps even more effective than guided i-CBT for some patients and could be delivered at a much lower cost. All these possibilities will be examined in the AMHI trial. Results will have great potential to provide actionable information to help patients, clinicians, and payers know when to use i-CBT and at what level of intensity. It also has the potential to increase awareness of i-CBT across the USA.

Trial status

IRB Approval of Protocol Version 1.0; 3/13/2020. Recruitment began 11/1/2020. Recruitment is tentatively scheduled to be completed 4/31/2022.

Availability of data and materials {29}

Trial materials can be obtained from the first author upon request.



Antidepressant medication


Appalachian Mind Health Initiative


Cognitive behavioral therapy


Confidence interval




Cross-validated target minimum loss-based estimator


Depression and Bipolar Support Alliance of West Virginia


Data Safety Monitoring Board


Electronic medical record


Harvard Medical School


Hamilton Rating Scale for Depression


Heterogeneity of treatment effects


Internet-based cognitive behavioral therapy


Implementation Monitoring Committee


Inverse probability weighting


Individualized treatment rule


Major Depressive Disorder


Number needed to treat


Patient-Centered Outcomes Research Institute


Principal investigator


Personally identifying information


Quick Inventory of Depressive Symptomatology Self-Report


Remission from Depression Questionnaire


Scientific Advisory Committee


Super learner


Self-report questionnaire


Veterans Health Administration


West Virginia Practice Based Research Network


West Virginia Primary Care Association


West Virginia University


  1. Cuijpers P, Reynolds CF 3rd, Donker T, Li J, Andersson G, Beekman A. Personalized treatment of adult depression: medication, psychotherapy, or both? A systematic review. Depress Anxiety. 2012;29(10):855–64.

    Article  PubMed  Google Scholar 

  2. Kessler RC. The costs of depression. Psychiatr Clin North Am. 2012;35(1):1–14.

    Article  PubMed  Google Scholar 

  3. Ferrari AJ, Charlson FJ, Norman RE, Patten SB, Freedman G, Murray CJ, et al. Burden of depressive disorders by country, sex, age, and year: findings from the global burden of disease study 2010. PLoS Med. 2013;10(11):e1001547.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Ferrari AJ, Norman RE, Freedman G, Baxter AJ, Pirkis JE, Harris MG, et al. The burden attributable to mental and substance use disorders as risk factors for suicide: findings from the Global Burden of Disease Study 2010. PLoS One. 2014;9(4):e91936.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  5. Greenberg PE, Fournier AA, Sisitsky T, Pike CT, Kessler RC. The economic burden of adults with major depressive disorder in the United States (2005 and 2010). J Clin Psychiatry. 2015;76(2):155–62.

    Article  PubMed  Google Scholar 

  6. Patten SB, Williams JV, Lavorato DH, Modgill G, Jetté N, Eliasziw M. Major depression as a risk factor for chronic disease incidence: longitudinal analyses in a general population cohort. Gen Hosp Psychiatry. 2008;30(5):407–13.

    Article  PubMed  Google Scholar 

  7. Deschênes SS, Burns RJ, Schmitz N. Associations between depression, chronic physical health conditions, and disability in a community sample: a focus on the persistence of depression. J Affect Disord. 2015;179:6–13.

    Article  PubMed  Google Scholar 

  8. Ishak WW, Balayan K, Bresee C, Greenberg JM, Fakhry H, Christensen S, et al. A descriptive analysis of quality of life using patient-reported measures in major depressive disorder in a naturalistic outpatient setting. Qual Life Res. 2013;22(3):585–96.

    Article  PubMed  Google Scholar 

  9. Druss BG, Hwang I, Petukhova M, Sampson NA, Wang PS, Kessler RC. Impairment in role functioning in mental and chronic medical disorders in the United States: results from the National Comorbidity Survey Replication. Mol Psychiatry. 2009;14(7):728–37.

    Article  CAS  PubMed  Google Scholar 

  10. Viana MC, Gruber MJ, Shahly V, Alhamzawi A, Alonso J, Andrade LH, et al. Family burden related to mental and physical disorders in the world: results from the WHO World Mental Health (WMH) surveys. Braz J Psychiatry (Sao Paulo, Brazil: 1999). 2013;35(2):115–25.

    Article  Google Scholar 

  11. Weissman MM, Wickramaratne P, Pilowsky DJ, Poh E, Hernandez M, Batten LA, et al. The effects on children of depressed mothers’ remission and relapse over 9 months. Psychol Med. 2014;44(13):2811–24.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Olfson M, Blanco C, Marcus SC. Treatment of adult depression in the United States. JAMA Intern Med. 2016;176(10):1482–91.

    Article  PubMed  Google Scholar 

  13. American Psychiatric Association. Practice guideline for the treatment of patients with major depressive disorder (revision). Am J Psychiatry. 2000;157(4 Suppl):1–45.

    Google Scholar 

  14. Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry. 2014;13(3):288–95.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Cuijpers P, Noma H, Karyotaki E, Vinkers CH, Cipriani A, Furukawa TA. A network meta-analysis of the effects of psychotherapies, pharmacotherapies and their combination in the treatment of adult depression. World Psychiatry. 2020;19(1):92–107.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Weitz ES, Hollon SD, Twisk J, van Straten A, Huibers MJ, David D, et al. Baseline depression severity as moderator of depression outcomes between cognitive behavioral therapy vs pharmacotherapy: an individual patient data meta-analysis. JAMA Psychiatry. 2015;72(11):1102–9.

    Article  PubMed  Google Scholar 

  17. Cohen ZD, DeRubeis RJ. Treatment selection in depression. Annu Rev Clin Psychol. 2018;14:209–36.

    Article  PubMed  Google Scholar 

  18. DeRubeis RJ, Cohen ZD, Forand NR, Fournier JC, Gelfand LA, Lorenzo-Luaces L. The Personalized Advantage Index: translating research on prediction into individualized treatment recommendations. A demonstration. PLoS One. 2014;9(1):e83875.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  19. Karyotaki E, Smit Y, Holdt Henningsen K, Huibers MJ, Robays J, de Beurs D, et al. Combining pharmacotherapy and psychotherapy or monotherapy for major depression? A meta-analysis on the long-term effects. J Affect Disord. 2016;194:144–52.

    Article  CAS  PubMed  Google Scholar 

  20. Craighead WE, Dunlop BW. Combination psychotherapy and antidepressant medication treatment for depression: for whom, when, and how. Annu Rev Psychol. 2014;65:267–300.

    Article  PubMed  Google Scholar 

  21. Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlöf E. Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther. 2018;47(1):1–18.

    Article  PubMed  Google Scholar 

  22. Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med. 2010;40(12):1943–57.

    Article  CAS  PubMed  Google Scholar 

  23. Van den Broeck K, Remmen R, Vanmeerbeek M, Destoop M, Dom G. Collaborative care regarding major depressed patients: a review of guidelines and current practices. J Affect Disord. 2016;200:189–203.

    Article  PubMed  Google Scholar 

  24. Rollman BL, Herbeck Belnap B, Abebe KZ, Spring MB, Rotondi AJ, Rothenberger SD, et al. Effectiveness of online collaborative care for treating mood and anxiety disorders in primary care: a randomized clinical trial. JAMA Psychiatry. 2018;75(1):56–64.

    Article  PubMed  Google Scholar 

  25. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther. 2009;38(4):196–205.

    Article  PubMed  Google Scholar 

  26. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012;32(4):329–42.

    Article  PubMed  Google Scholar 

  27. Wells MJ, Owen JJ, McCray LW, Bishop LB, Eells TD, Brown GK, et al. Computer-assisted cognitive-behavior therapy for depression in primary care: systematic review and meta-analysis. Prim Care Companion CNS Disord. 2018;20(2):24454.

    Article  Google Scholar 

  28. Wright JH, Owen JJ, Richards D, Eells TD, Richardson T, Brown GK, et al. Computer-assisted cognitive-behavior therapy for depression: a systematic review and meta-analysis. J Clin Psychiatry. 2019;80(2):3573.

    Article  Google Scholar 

  29. Cuijpers P, Noma H, Karyotaki E, Cipriani A, Furukawa T. Individual, group, telephone, self-help and internet-based cognitive behavior therapy for adult depression: a network meta-analysis of delivery methods. JAMA Psychiatry. 2019;76:700–7.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry. 2017;74(4):351–9.

    Article  PubMed  Google Scholar 

  31. Cuijpers P, Sijbrandij M, Koole SL, Andersson G, Beekman AT, Reynolds CF 3rd. Adding psychotherapy to antidepressant medication in depression and anxiety disorders: a meta-analysis. World Psychiatry. 2014;13(1):56–67.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Grohol J. Mental health professionals: US statistics 2017: Psych Central; 2019. Accessed 15 Oct 2020

    Google Scholar 

  33. Kessler RC, van Loo HM, Wardenaar KJ, Bossarte RM, Brenner LA, Ebert DD, et al. Using patient self-reports to study heterogeneity of treatment effects in major depressive disorder. Epidemiol Psychiatr Sci. 2017;26(1):22–36.

    Article  CAS  PubMed  Google Scholar 

  34. Kraemer HC. Discovering, comparing, and combining moderators of treatment on outcome after randomized clinical trials: a parametric approach. Stat Med. 2013;32(11):1964–73.

    Article  PubMed  Google Scholar 

  35. National Collaborating Centre for Mental Health. Depression: the treatment and management of depression in adults (updated edition): British Psychological Society; 2010.

    Google Scholar 

  36. Gelenberg A, Freeman M, Markowitz J, Rosenbaum J, Thase M, Trivedi M, et al. American Psychiatric Association practice guideline for the treatment of patients with major depressive disorder. Am J Psychiatry. 2010;167(suppl 10):9–118.

    Google Scholar 

  37. United States Census Bureau. American FactFinder 2015. Accessed 15 Oct 2020.

  38. The Henry J. Kaiser Family Foundation. Medicaid fact sheets. In: Percent of people covered by Medicaid Accessed 15 Oct 2020.

  39. National Institute on Drug Abuse. Opioid summaries by state 2020. Accessed 15 Oct 2020.

  40. United States Census Bureau. 2010 census urban and rural classification and urban area criteria. Accessed 15 Oct 2020.

  41. Mental Health America. The state of mental health in America 2017. Accessed 15 Oct 2020.

  42. van Schaik DJ, Klijn AF, van Hout HP, van Marwijk HW, Beekman AT, de Haan M, et al. Patients’ preferences in the treatment of depressive disorder in primary care. Gen Hosp Psychiatry. 2004;26(3):184–9.

    Article  PubMed  Google Scholar 

  43. Lin P, Campbell DG, Chaney EF, Liu CF, Heagerty P, Felker BL, et al. The influence of patient preference on depression treatment in primary care. Ann Behav Med. 2005;30(2):164–73.

    Article  PubMed  Google Scholar 

  44. Waltz TJ, Campbell DG, Kirchner JE, Lombardero A, Bolkan C, Zivin K, et al. Veterans with depression in primary care: provider preferences, matching, and care satisfaction. Fam Syst Health. 2014;32(4):367–77.

    Article  PubMed  Google Scholar 

  45. Dunlop BW, Kelley ME, Aponte-Rivera V, Mletzko-Crowe T, Kinkead B, Ritchie JC, et al. Effects of patient preferences on outcomes in the Predictors of Remission in Depression to Individual and Combined Treatments (PReDICT) study. Am J Psychiatry. 2017;174(6):546–56.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Morris C. A finite selection model for experimental design of the health insurance study. J Econom. 1979;11(1):43–61.

    Article  Google Scholar 

  47. U.S. Department of Veterans Affairs. West Virginia 2018. Accessed 1 Oct 2020.

  48. Harrington J. There are 18.2 million veterans in the US. Which state is home to the most of them? 2019. Accessed 20 Sept 2020.

    Google Scholar 

  49. SilverCloud. SilverCloud digital mental health platform. 2019. Accessed 30 Oct 2020.

    Google Scholar 

  50. Duffy D, Enrique A, Connell S, Connolly C, Richards D. Internet-delivered cognitive behaviour therapy as a prequel to face-to-face therapy for depression and anxiety: a naturalistic observation. Front Psychiatry. 2019;10:902.

    Article  PubMed  Google Scholar 

  51. Enrique A, Palacios JE, Ryan H, Richards D. Exploring the relationship between usage and outcomes of an internet-based intervention for individuals with depressive symptoms: secondary analysis of data from a randomized controlled trial. J Med Internet Res. 2019;21(8):e12775.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Richards D, Enrique A, Eilert N, Franklin M, Palacios J, Duffy D, et al. A pragmatic randomized waitlist-controlled effectiveness and cost-effectiveness trial of digital interventions for depression and anxiety. NPJ Digit Med. 2020;3:85.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Coyle D, Doherty G, Matthews M, Sharry J. Computers in talk-based mental health interventions. Interact Comput. 2007;19(4):545–62.

    Article  Google Scholar 

  54. Earley C, Joyce C, McElvaney J, Richards D, Timulak L. Preventing depression: qualitatively examining the benefits of depression-focused i-CBT for participants who do not meet clinical thresholds. Internet Interv. 2017;9:82.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Palacios JE, Richards D, Palmer R, Coudray C, Hofmann SG, Palmieri PA, et al. Supported internet-delivered cognitive behavioral therapy programs for depression, anxiety, and stress in university students: open, non-randomised trial of acceptability, effectiveness, and satisfaction. JMIR Ment Health. 2018;5(4):e11467.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Doherty G, Coyle D, Sharry J. Engagement with online mental health interventions: an exploratory clinical study of a treatment for depression. In: Proceedings of the SIGCHI conference on human factors in computing systems; 2012.

    Google Scholar 

  57. Richards D, Timulak L, O’Brien E, Hayes C, Vigano N, Sharry J, et al. A randomized controlled trial of an internet-delivered treatment: its potential as a low-intensity community intervention for adults with symptoms of depression. Behav Res Ther. 2015;75:20–31.

    Article  CAS  PubMed  Google Scholar 

  58. Enrique A, Duffy D, Lawler K, Richards D, Jones S. An internet-delivered self-management program for Bipolar Disorder in mental health services in Ireland. Results and learnings from a feasibility trial. Clin Psychol Psychother. 2020;27(6):925–39.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. 2019;21(9):e14567.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Beck CD, Soucy JN, Hadjistavropoulos HD. Mixed-method evaluation of an online motivational intervention as a pre-treatment to internet-delivered cognitive behaviour therapy: immediate benefits and user feedback. Internet Interv. 2020;20:100311.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Addington EL, Cheung EO, Bassett SM, Kwok I, Schuette SA, Shiu E, et al. The MARIGOLD study: feasibility and enhancement of an online intervention to improve emotion regulation in people with elevated depressive symptoms. J Affect Disord. 2019;257:352–64.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Cheung EO, Cohn MA, Dunn LB, Melisko ME, Morgan S, Penedo FJ, et al. A randomized pilot trial of a positive affect skill intervention (lessons in linking affect and coping) for women with metastatic breast cancer. Psychooncology. 2017;26(12):2101–8.

    Article  PubMed  Google Scholar 

  63. Chien I, Enrique A, Palacios J, Regan T, Keegan D, Carter D, et al. A machine learning approach to understanding patterns of engagement with internet-delivered mental health interventions. JAMA Netw Open. 2020;3(7):e2010791.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Zimmerman M, Martinez JH, Attiullah N, Friedman M, Toba C, Boerescu DA, et al. A new type of scale for determining remission from depression: the Remission from Depression Questionnaire. J Psychiatr Res. 2013;47(1):78–82.

    Article  PubMed  Google Scholar 

  65. Montoya A, Lebrec J, Keane KM, Fregenal I, Ciudad A, Moríñigo Á, et al. Broader conceptualization of remission assessed by the remission from depression questionnaire and its association with symptomatic remission: a prospective, multicenter, observational study. BMC Psychiatry. 2016;16(1):352.

    Article  PubMed  PubMed Central  Google Scholar 

  66. DiBenedetti DB, Danchenko N, François C, Lewis S, Davis KH, Fehnel SE. Development of a family functioning scale for major depressive disorder. Curr Med Res Opin. 2012;28(3):303–13.

    Article  PubMed  Google Scholar 

  67. Lasch KE, Hassan M, Endicott J, Piault-Luis EC, Locklear J, Fitz-Randolph M, et al. Development and content validity of a patient reported outcomes measure to assess symptoms of major depressive disorder. BMC Psychiatry. 2012;12:34.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Vaccarino AL, Evans KR, Kalali AH, Kennedy SH, Engelhardt N, Frey BN, et al. The depression inventory development workgroup: a collaborative, empirically driven initiative to develop a new assessment tool for major depressive disorder. Innov Clin Neurosci. 2016;13(9-10):20–31.

    PubMed  PubMed Central  Google Scholar 

  69. Zimmerman M, Martinez JH, Attiullah N, Friedman M, Toba C, Boerescu DA. The remission from depression questionnaire as an outcome measure in the treatment of depression. Depress Anxiety. 2014;31(6):533–8.

    Article  PubMed  Google Scholar 

  70. Zimmerman M, Galione JN, Attiullah N, Friedman M, Toba C, Boerescu DA, et al. Depressed patients’ perspectives of 2 measures of outcome: the Quick Inventory of Depressive Symptomatology (QIDS) and the Remission from Depression Questionnaire (RDQ). Ann Clin Psychiatry. 2011;23(3):208–12.

    PubMed  Google Scholar 

  71. Trivedi MH, Rush AJ, Ibrahim HM, Carmody TJ, Biggs MM, Suppes T, et al. The Inventory of Depressive Symptomatology, Clinician Rating (IDS-C) and Self-Report (IDS-SR), and the Quick Inventory of Depressive Symptomatology, Clinician Rating (QIDS-C) and Self-Report (QIDS-SR) in public sector patients with mood disorders: a psychometric evaluation. Psychol Med. 2004;34(1):73–82.

    Article  CAS  PubMed  Google Scholar 

  72. Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry. 1960;23(1):56–62.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  73. Inventory of Depressive Symptomatology (IDS) and Quick Inventory of Depressive Symptomatology (QIDS). Accessed 1 Oct 2020.

  74. Jobst A, Brakemeier EL, Buchheim A, Caspar F, Cuijpers P, Ebmeier KP, et al. European Psychiatric Association Guidance on psychotherapy in chronic depression across Europe. Eur Psychiatry. 2016;33:18–36.

    Article  CAS  PubMed  Google Scholar 

  75. Statista. Drug overdose death rate in the United States in 2015, by state (per 100,000 population). 2017. Accessed 15 Oct 2020.

    Google Scholar 

  76. de Jonge P, Wardenaar KJ, Lim CCW, Aguilar-Gaxiola S, Alonso J, Andrade LH, et al. The cross-national structure of mental disorders: results from the World Mental Health Surveys. Psychol Med. 2018;48(12):2073–84.

    Article  PubMed  Google Scholar 

  77. PROMIS. Short form v1.0 - severity of substance use (past 30 days) 7a. 2017. Accessed 30 Oct 2020.

    Google Scholar 

  78. Castel AD, Tang W, Peterson J, Mikre M, Parenti D, Elion R, et al. Sorting through the lost and found: are patient perceptions of engagement in care consistent with standard continuum of care measures? J Acquir Immunue Defic Syndr. 2015;69 Suppl 1(0 1):S44–55.

    Article  Google Scholar 

  79. Barr PJ, Thompson R, Walsh T, Grande SW, Ozanne EM, Elwyn G. The psychometric properties of CollaboRATE: a fast and frugal patient-reported measure of the shared decision-making process. J Med Internet Res. 2014;16(1):e2.

    Article  PubMed  PubMed Central  Google Scholar 

  80. Fava M, Rush AJ, Trivedi MH, Nierenberg AA, Thase ME, Sackeim HA, et al. Background and rationale for the sequenced treatment alternatives to relieve depression (STAR* D) study. Psychiatr Clin North Am. 2003;26(2):457–94.

    Article  PubMed  Google Scholar 

  81. Newman MG, Szkodny LE, Llera SJ, Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy? Clin Psychol Rev. 2011;31(1):89–103.

    Article  PubMed  Google Scholar 

  82. Vaidyam A, Halamka J, Torous J. Actionable digital phenotyping: a framework for the delivery of just-in-time and longitudinal interventions in clinical healthcare. mHealth. 2019;5:25.

    Article  PubMed  PubMed Central  Google Scholar 

  83. Henson P, Torous J. Feasibility and correlations of smartphone meta-data toward dynamic understanding of depression and suicide risk in schizophrenia. Int J Methods Psychiatr Res. 2020;29(2):e1825.

    Article  PubMed  PubMed Central  Google Scholar 

  84. Yarkoni T. The abbreviation of personality, or how to measure 200 personality scales with 200 items. J Res Pers. 2010;44(2):180–98.

    Article  PubMed  PubMed Central  Google Scholar 

  85. Ustun B, Rudin C. Supersparse linear integer models for optimized medical scoring systems. Mach Learn. 2016;102(3):349–91.

    Article  Google Scholar 

  86. Germine L, Nakayama K, Duchaine BC, Chabris CF, Chatterjee G, Wilmer JB. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychon Bull Rev. 2012;19(5):847–57.

    Article  PubMed  Google Scholar 

  87. Hamilton CM, Strader LC, Pratt JG, Maiese D, Hendershot T, Kwok RK, et al. The PhenX Toolkit: get the most from your measures. Am J Epidemiol. 2011;174(3):253–60.

    Article  PubMed  PubMed Central  Google Scholar 

  88. Department of Homeland Security. Cybersecurity and Infrastructure Security Agency. Identifying critical infrastructure during COVID-19. Accessed 30 Oct 2020.

  89. Rush AJ, Trivedi MH, Ibrahim HM, Carmody TJ, Arnow B, Klein DN, et al. The 16-Item Quick Inventory of Depressive Symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol Psychiatry. 2003;54(5):573–83.

    Article  PubMed  Google Scholar 

  90. Rush AJ, Gullion CM, Basco MR, Jarrett RB, Trivedi MH. The inventory of depressive symptomatology (IDS): psychometric properties. Psychol Med. 1996;26(3):477–86.

    Article  CAS  PubMed  Google Scholar 

  91. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®): American Psychiatric Pub; 2013.

    Book  Google Scholar 

  92. Day CV, John Rush A, Harris AW, Boyce PM, Rekshan W, Etkin A, et al. Impairment and distress patterns distinguishing the melancholic depression subtype: an iSPOT-D report. J Affect Disord. 2015;174:493–502.

    Article  PubMed  Google Scholar 

  93. Zimmerman M, Chelminski I, Young D, Dalrymple K, Martinez JH. A clinically useful self-report measure of the DSM-5 mixed features specifier of major depressive disorder. J Affect Disord. 2014;168:357–62.

    Article  PubMed  Google Scholar 

  94. Kessler RC, Ustün TB. The World Mental Health (WMH) Survey Initiative Version of the World Health Organization (WHO) Composite International Diagnostic Interview (CIDI). Int J Methods Psychiatr Res. 2004;13(2):93–121.

    Article  PubMed  Google Scholar 

  95. Rizvi SJ, Quilty LC, Sproule BA, Cyriac A, Michael Bagby R, Kennedy SH. Development and validation of the Dimensional Anhedonia Rating Scale (DARS) in a community sample and individuals with major depression. Psychiatry Res. 2015;229(1-2):109–19.

    Article  PubMed  Google Scholar 

  96. Llerena K, Park SG, McCarthy JM, Couture SM, Bennett ME, Blanchard JJ. The Motivation and Pleasure Scale-Self-Report (MAP-SR): reliability and validity of a self-report measure of negative symptoms. Compr Psychiatry. 2013;54(5):568–74.

    Article  PubMed  PubMed Central  Google Scholar 

  97. Stewart JW, McGrath PJ, Fava M, Wisniewski SR, Zisook S, Cook I, et al. Do atypical features affect outcome in depressed outpatients treated with citalopram? Int J Neuropsychopharmacol. 2010;13(1):15–30.

    Article  CAS  PubMed  Google Scholar 

  98. Ursano RJ, Colpe LJ, Heeringa SG, Kessler RC, Schoenbaum M, Stein MB, et al. The Army study to assess risk and resilience in servicemembers (Army STARRS). Psychiatry. 2014;77(2):107–19.

    Article  PubMed  Google Scholar 

  99. Kessler RC, Üstün TB. The world mental health (WMH) survey initiative version of the world health organization (WHO) composite international diagnostic interview (CIDI). Int J Methods Psychaitr Res. 2004;13(2):93–121.

    Article  Google Scholar 

  100. Sotsky SM, Glass DR, Shea MT, Pilkonis PA, Collins JF, Elkin I, et al. Patient predictors of response to psychotherapy and pharmacotherapy: findings in the NIMH Treatment of Depression Collaborative Research Program. Am J Psychiatry. 1991;148(8):997–1008.

    Article  CAS  PubMed  Google Scholar 

  101. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia–Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266–77.

    Article  PubMed  PubMed Central  Google Scholar 

  102. Nock MK, Holmberg EB, Photos VI, Michel BD. Self-Injurious Thoughts and Behaviors Interview: development, reliability, and validity in an adolescent sample. Psychol Assess. 2007;19(3):309–17.

    Article  PubMed  Google Scholar 

  103. Dube P, Kurt K, Bair MJ, Theobald D, Williams LS. The p4 screener: evaluation of a brief measure for assessing potential suicide risk in 2 randomized effectiveness trials of primary care and oncology patients. Prim Care Companion J Clin Psychiatry. 2010;12(6).

  104. Blevins CA, Weathers FW, Davis MT, Witte TK, Domino JL. The posttraumatic stress disorder checklist for DSM-5 (PCL-5): development and initial psychometric evaluation. J Trauma Stress. 2015;28(6):489–98.

    Article  PubMed  Google Scholar 

  105. Zuromski KL, Ustun B, Hwang I, Keane TM, Marx BP, Stein MB, et al. Developing an optimal short-form of the PTSD Checklist for DSM-5 (PCL-5). Depress Anxiety. 2019;36(9):790–800.

    Article  PubMed  PubMed Central  Google Scholar 

  106. Gibbons LE, Fredericksen R, Merrill JO, McCaul ME, Chander G, Hutton H, et al. Suitability of the PROMIS alcohol use short form for screening in a HIV clinical care setting. Drug Alcohol Depend. 2016;164:113–9.

    Article  PubMed  PubMed Central  Google Scholar 

  107. Spitzer RL, Williams JB, Gibbon M, First MB. User’s guide for the structured clinical interview for DSM-III-R: SCID: American Psychiatric Association; 1990.

    Google Scholar 

  108. Joyce PR, McKenzie JM, Carter JD, Rae AM, Luty SE, Frampton CM, et al. Temperament, character and personality disorders as predictors of response to interpersonal psychotherapy and cognitive-behavioural therapy for depression. Br J Psychiatry. 2007;190(6):503–8.

    Article  PubMed  Google Scholar 

  109. Weissman MM, Wickramaratne P, Adams P, Wolk S, Verdeli H, Olfson M. Brief screening for family psychiatric history: the family history screen. Arch Gen Psychiatry. 2000;57(7):675–82.

    Article  CAS  PubMed  Google Scholar 

  110. Andreasen NC, Endicott J, Spitzer RL, Winokur G. The family history method using diagnostic criteria. Reliability and validity. Arch Gen Psychiatry. 1977;34(10):1229–35.

    Article  CAS  PubMed  Google Scholar 

  111. Axelsson E, Andersson E, Ljótsson B, Wallhed Finn D, Hedman E. The health preoccupation diagnostic interview: inter-rater reliability of a structured interview for diagnostic assessment of DSM-5 somatic symptom disorder and illness anxiety disorder. Cogn Behav Ther. 2016;45(4):259–69.

    Article  PubMed  Google Scholar 

  112. Limburg K, Sattel H, Radziej K, Lahmann C. DSM-5 somatic symptom disorder in patients with vertigo and dizziness symptoms. J Psychosom Res. 2016;91:26–32.

    Article  PubMed  Google Scholar 

  113. Toussaint A, Murray AM, Voigt K, Herzog A, Gierk B, Kroenke K, et al. Development and validation of the Somatic Symptom Disorder-B Criteria Scale (SSD-12). Psychosom Med. 2016;78(1):5–12.

    Article  PubMed  Google Scholar 

  114. Herzog A, Voigt K, Meyer B, Wollburg E, Weinmann N, Langs G, et al. Psychological and interactional characteristics of patients with somatoform disorders: validation of the Somatic Symptoms Experiences Questionnaire (SSEQ) in a clinical psychosomatic population. J Psychosom Res. 2015;78(6):553–62.

    Article  PubMed  Google Scholar 

  115. Morin CM, Belleville G, Bélanger L, Ivers H. The Insomnia Severity Index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep. 2011;34(5):601–8.

    Article  PubMed  PubMed Central  Google Scholar 

  116. Krebs EE, Lorenz KA, Bair MJ, Damush TM, Wu J, Sutherland JM, et al. Development and initial validation of the PEG, a three-item scale assessing pain intensity and interference. J Gen Intern Med. 2009;24(6):733–8.

    Article  PubMed  PubMed Central  Google Scholar 

  117. Von Korff M, Ormel J, Keefe FJ, et al. Grading the severity of chronic pain. Pain. 1992;50:133–49.

    Article  Google Scholar 

  118. Leon A, Olfson M, Portera L, Farber L. Assessing impairment in primary care: a psychometric analysis of the Sheehan Disability Scale. Int J Psychiatry Med. 1997;27:93–105.

    Article  CAS  PubMed  Google Scholar 

  119. Motrico E, Moreno-Küstner B, de Dios Luna J, Torres-González F, King M, Nazareth I, et al. Psychometric properties of the List of Threatening Experiences—LTE and its association with psychosocial factors and mental disorders according to different scoring methods. J Affect Disord. 2013;150(3):931–40.

    Article  PubMed  Google Scholar 

  120. Health Do, Cox A, Bentovim A. Recent life events questionnaire. London: The Stationery Office, Ltd.; 2000. p. 35–8. Accessed 30 Oct 2020

    Google Scholar 

  121. Bernstein DP, Stein JA, Newcomb MD, Walker E, Pogge D, Ahluvalia T, et al. Development and validation of a brief screening version of the Childhood Trauma Questionnaire. Child Abuse Negl. 2003;27(2):169–90.

    Article  PubMed  Google Scholar 

  122. Dube SR, Anda RF, Felitti VJ, Chapman DP, Williamson DF, Giles WH. Childhood abuse, household dysfunction, and the risk of attempted suicide throughout the life span: findings from the Adverse Childhood Experiences Study. JAMA. 2001;286(24):3089–96.

    Article  CAS  PubMed  Google Scholar 

  123. Stein MB, Campbell-Sills L, Ursano RJ, Rosellini AJ, Colpe LJ, He F, et al. Childhood maltreatment and lifetime suicidal behaviors among new soldiers in the US Army: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). J Clin Psychiatry. 2018;79(2):2567.

    Article  Google Scholar 

  124. Williams LM, Debattista C, Duchemin AM, Schatzberg AF, Nemeroff CB. Childhood trauma predicts antidepressant response in adults with major depression: data from the randomized international study to predict optimized treatment for depression. Transl Psychiatry. 2016;6(5):e799.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  125. Parker G, Tupling H, Brown L. Parental bonding instrument (PBI). Br J Med Psychol. 1979;52(1):1–10.

    Article  Google Scholar 

  126. Lin LY, Sidani JE, Shensa A, Radovic A, Miller E, Colditz JB, et al. Association between social media use and depression among U.S. young adults. Depress Anxiety. 2016;33(4):323–31.

    Article  PubMed  PubMed Central  Google Scholar 

  127. Schuster TL, Kessler RC, Aseltine RH. Supportive interactions, negative interactions, and depressed mood. Am J Community Psychol. 1990;18(3):423–38.

    Article  CAS  PubMed  Google Scholar 

  128. Van Orden KA, Cukrowicz KC, Witte TK, Joiner TE Jr. Thwarted belongingness and perceived burdensomeness: construct validity and psychometric properties of the Interpersonal Needs Questionnaire. Psychol Assess. 2012;24(1):197.

    Article  PubMed  Google Scholar 

  129. Besse RD. Loneliness among college students: examining potential coping strategies and the influence of targeted messages on the likelihood of befriending: Kansas State University; 2016. Accessed 30 Oct 2020

    Google Scholar 

  130. Robinson JP, Shaver PR. Measures of social psychological attitudes. Revised. 1973. Accessed Oct 15 2020.

    Google Scholar 

  131. (SMART). SMART. Best practices for asking questions about sexual orientation on surveys: The Williams Institute, UCLA School of Law; 2009. Accessed 15 Oct 2020

    Google Scholar 

  132. Iwamoto DK, Brady J, Kaya A, Park A. Masculinity and depression: a longitudinal investigation of multidimensional masculine norms among college men. Am J Mens Health. 2018;12(6):1873–81.

    Article  PubMed  PubMed Central  Google Scholar 

  133. Thompson E Jr, Pleck JH, Ferrera DL. Men and masculinities: scales for masculinity ideology and masculinity-related constructs. Sex Roles. 1992;27(11-12):573–607.

    Article  Google Scholar 

  134. Bagby RM, Parker JD, Taylor GJ. The twenty-item Toronto Alexithymia Scale--I. Item selection and cross-validation of the factor structure. J Psychosom Res. 1994;38(1):23–32.

    Article  CAS  PubMed  Google Scholar 

  135. Judah MR, Grant DM, Mills AC, Lechner WV. Factor structure and validation of the Attentional Control Scale. Cognit Emot. 2014;28(3):433–51.

    Article  Google Scholar 

  136. Bartholomew K, Horowitz LM. Attachment styles among young adults: a test of a four-category model. J Pers Soc Psychol. 1991;61(2):226–44.

    Article  CAS  PubMed  Google Scholar 

  137. Medrano LA, Trógolo M. Construct validity of the difficulties in emotion regulation scale: further evidence using confirmatory factor analytic approach. Abnorm Behav Psychol. 2016;2:2.

    Article  Google Scholar 

  138. Garnefski N, Kraaij V. Relationships between cognitive emotion regulation strategies and depressive symptoms: a comparative study of five specific samples. Pers Individ Dif. 2006;40(8):1659–69.

    Article  Google Scholar 

  139. Tedeschi RG, Calhoun LG. The Posttraumatic Growth Inventory: measuring the positive legacy of trauma. J Trauma Stress. 1996;9(3):455–71.

    Article  CAS  PubMed  Google Scholar 

  140. Roberti JW, Harrington LN, Storch EA. Further psychometric support for the 10-item version of the perceived stress scale. JOCC. 2006;9(2):135–47.

    Google Scholar 

  141. Campbell-Sills L, Stein MB. Psychometric analysis and refinement of the Connor-davidson Resilience Scale (CD-RISC): validation of a 10-item measure of resilience. J Trauma Stress. 2007;20(6):1019–28.

    Article  PubMed  Google Scholar 

  142. Campbell-Sills L, Kessler RC, Ursano RJ, Sun X, Taylor CT, Heeringa SG, et al. Predictive validity and correlates of self-assessed resilience among U.S. Army soldiers. Depress Anxiety. 2018;35(2):122–31.

    Article  PubMed  Google Scholar 

  143. Stein MB, Kessler RC, Heeringa SG, Jain S, Campbell-Sills L, Colpe LJ, et al. Prospective longitudinal evaluation of the effect of deployment-acquired traumatic brain injury on posttraumatic stress and related disorders: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Am J Psychiatry. 2015;172(11):1101–11.

    Article  PubMed  PubMed Central  Google Scholar 

  144. Schlotz W, Yim IS, Zoccola PM, Jansen L, Schulz P. The perceived stress reactivity scale: Measurement invariance, stability, and validity in three countries. Psychol Assess. 2011;23(1):80.

    Article  PubMed  Google Scholar 

  145. Aish AM, Wasserman D, Renberg ES. Does Beck’s Hopelessness Scale really measure several components? Psychol Med. 2001;31(2):367–72.

    Article  CAS  PubMed  Google Scholar 

  146. Treynor W, Gonzalez R, Nolen-Hoeksema S. Rumination reconsidered: a psychometric analysis. Cognit Ther Res. 2003;27(3):247–59.

    Article  Google Scholar 

  147. Akiskal HS, Mendlowicz MV, Jean-Louis G, Rapaport MH, Kelsoe JR, Gillin JC, et al. TEMPS-A: validation of a short version of a self-rated instrument designed to measure variations in temperament. J Affect Disord. 2005;85(1-2):45–52.

    Article  PubMed  Google Scholar 

  148. Nakato Y, Inoue T, Nakagawa S, Kitaichi Y, Kameyama R, Wakatsuki Y, et al. Confirmation of the factorial structure of the Japanese short version of the TEMPS-A in psychiatric patients and general adults. Neuropsychiatr Dis Treat. 2016;12:2173–9.

    Article  PubMed  PubMed Central  Google Scholar 

  149. Fossati A, Somma A, Borroni S, Markon KE, Krueger RF. The personality inventory for DSM-5 brief form: evidence for reliability and construct validity in a sample of community-dwelling Italian adolescents. Assessment. 2017;24(5):615–31.

    Article  PubMed  Google Scholar 

  150. Aluja A, Jm R, García LF, Angleitner A, Kuhlman M, Zuckerman M. A cross-cultural shortened form of the ZKPQ (ZKPQ-50-cc) adapted to English, French, German, and Spanish languages. Pers Individ Dif. 2006;41(4):619–28.

    Article  Google Scholar 

  151. Zuckerman M, Kuhlman DM, Joireman J, Teta P, Kraft M. A comparison of three structural models for personality: the big three, the big five, and the alternative five. J Pers Soc Psychol. 1993;65(4):757.

    Article  Google Scholar 

  152. John OP, Naumann LP, Soto CJ. Paradigm shift to the integrative big five trait taxonomy. Personal Disord. 2008;3(2):114–58.

    Google Scholar 

  153. Costa PT, McCrae RR. Professional manual: revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI). Odessa: Psychol Assess; 1992. p. 61.

    Google Scholar 

  154. Cyders MA, Littlefield AK, Coffey S, Karyadi KA. Examination of a short English version of the UPPS-P Impulsive Behavior Scale. Addict Behav. 2014;39(9):1372–6.

    Article  PubMed  PubMed Central  Google Scholar 

  155. Mullins-Sweatt SN, Jamerson JE, Samuel DB, Olson DR, Widiger TA. Psychometric properties of an abbreviated instrument of the five-factor model. Assessment. 2006;13(2):119–37.

    Article  PubMed  Google Scholar 

  156. Nock MK, Wedig MM, Holmberg EB, Hooley JM. The emotion reactivity scale: development, evaluation, and relation to self-injurious thoughts and behaviors. Behav Ther. 2008;39(2):107–16.

    Article  PubMed  Google Scholar 

  157. Gosling SD, Rentfrow PJ, Swann WB Jr. A very brief measure of the Big-Five personality domains. J Res Pers. 2003;37(6):504–28.

    Article  Google Scholar 

  158. Sato T. The Eysenck Personality Questionnaire Brief Version: factor structure and reliability. J Psychol. 2005;139(6):545–52.

    Article  PubMed  Google Scholar 

  159. Pearlin LI, Schooler C. The structure of coping. J Health Soc Behav. 1978;19(1):2–21.

  160. Tambs K, Røysamb E. Selection of questions to short-form versions of original psychometric instruments in MoBa. Nor Epidemiol. 2014;24(1-2):195–201.

    Google Scholar 

  161. Dreer LE, Berry J, Rivera P, Snow M, Elliott TR, Miller D, et al. Efficient assessment of social problem-solving abilities in medical and rehabilitation settings: a Rasch analysis of the Social Problem-Solving Inventory-Revised. J Clin Psychol. 2009;65(7):653–69.

    Article  PubMed  PubMed Central  Google Scholar 

  162. Langbehn DR, Pfohl BM, Reynolds S, Clark LA, Battaglia M, Bellodi L, et al. The Iowa Personality Disorder Screen: development and preliminary validation of a brief screening interview. J Personal Disord. 1999;13(1):75–89.

    Article  CAS  Google Scholar 

  163. Safran DG, Kosinski M, Tarlov AR, Rogers WH, Taira DH, Lieberman N, et al. The Primary Care Assessment Survey: tests of data quality and measurement performance. Med Care. 1998;36(5):728–39.

    Article  CAS  PubMed  Google Scholar 

  164. Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med. 2004;36(8):588–94.

    PubMed  Google Scholar 

  165. Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res. 2006;8(4):e27.

    Article  PubMed  PubMed Central  Google Scholar 

  166. McHorney CA, Spain CV, Alexander CM, Simmons J. Validity of the adherence estimator in the prediction of 9-month persistence with medications prescribed for chronic diseases: a prospective analysis of data from pharmacy claims. Clin Ther. 2009;31(11):2584–607.

    Article  PubMed  Google Scholar 

  167. Unni EJ, Olson JL, Farris KB. Revision and validation of Medication Adherence Reasons Scale (MAR-Scale). Curr Med Res Opin. 2014;30(2):211–21.

    Article  PubMed  Google Scholar 

  168. Vik SA, Maxwell CJ, Hogan DB. Measurement, correlates, and health outcomes of medication adherence among seniors. Ann Pharmacother. 2004;38(2):303–12.

    Article  PubMed  Google Scholar 

  169. Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the Working Alliance Inventory. Psychother Res. 2006;16(1):12–25.

    Article  Google Scholar 

  170. Baumann M, Baumann C, Le Bihan E, Chau N. How patients perceive the therapeutic communications skills of their general practitioners, and how that perception affects adherence: use of the TCom-skill GP scale in a specific geographical area. BMC Health Serv Res. 2008;8(1):244.

    Article  PubMed  PubMed Central  Google Scholar 

  171. Bieber C, Müller KG, Nicolai J, Hartmann M, Eich W. How does your doctor talk with you? Preliminary validation of a brief patient self-report questionnaire on the quality of physician-patient interaction. J Clin Psychol Med Settings. 2010;17(2):125–36.

    Article  PubMed  Google Scholar 

  172. Steidtmann D, Manber R, Arnow BA, Klein DN, Markowitz JC, Rothbaum BO, et al. Patient treatment preference as a predictor of response and attrition in treatment for chronic depression. Depress Anxiety. 2012;29(10):896–905.

    Article  PubMed  PubMed Central  Google Scholar 

  173. Milosevic I, Levy HC, Alcolado GM, Radomsky AS. The treatment acceptability/adherence scale: moving beyond the assessment of treatment effectiveness. Cogn Behav Ther. 2015;44(6):456–69.

    Article  PubMed  Google Scholar 

  174. Wisniewski SR, Rush AJ, Balasubramani G, Trivedi MH, Nierenberg AA, Investigators SD. Self-rated global measure of the frequency, intensity, and burden of side effects. J Psychiatr Pract. 2006;12(2):71–9.

    Article  PubMed  Google Scholar 

  175. Curry J, Rohde P, Simons A, Silva S, Vitiello B, Kratochvil C, et al. Predictors and moderators of acute outcome in the Treatment for Adolescents with Depression Study (TADS). J Am Acad Child Adolesc Psychiatry. 2006;45(12):1427–39.

    Article  PubMed  Google Scholar 

  176. Saffer BY, Lanting SC, Koehle MS, Klonsky ED, Iverson GL. Assessing cognitive impairment using PROMIS(®) applied cognition-abilities scales in a medical outpatient sample. Psychiatry Res. 2015;226(1):169–72.

    Article  PubMed  Google Scholar 

  177. Byerly MJ, Nakonezny PA, Rush AJ. The Brief Adherence Rating Scale (BARS) validated against electronic monitoring in assessing the antipsychotic medication adherence of outpatients with schizophrenia and schizoaffective disorder. Schizophr Res. 2008;100(1-3):60–9.

    Article  PubMed  Google Scholar 

  178. Nease DE Jr, Aikens JE, Klinkman MS, Kroenke K, Sen A. Toward a more comprehensive assessment of depression remission: the Remission Evaluation and Mood Inventory Tool (REMIT). Gen Hosp Psychiatry. 2011;33(3):279–86.

    Article  PubMed  Google Scholar 

  179. Bushey MA, Kroenke K, Baye F, Lourens S. Assessing depression improvement with the remission evaluation and mood inventory tool (REMIT). Gen Hosp Psychiatry. 2019;60:44–9.

    Article  PubMed  Google Scholar 

  180. Son E, Halbert A, Abreu S, Hester R, Jefferson G, Jennings K, et al. Role of Google Glass in improving patient satisfaction for otolaryngology residents: a pilot study. Clin Otolaryngol. 2017;42(2):433–8.

    Article  CAS  PubMed  Google Scholar 

  181. Faraway JJ. Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models. 2nd ed. Boca Raton: CRC Press, Taylor & Francis Group; 2016.

    Book  Google Scholar 

  182. Jiménez FJR. Acceptance and commitment therapy versus traditional cognitive behavioral therapy: a systematic review and meta-analysis of current empirical evidence. Rev Int Psicol Ter Psicol. 2012;12(3):333–58.

    Google Scholar 

  183. van der Laan MJ, Polley EC, Hubbard AE. Super learner. Stat Appl Genet Mol Biol. 2007;6:Article25.

    PubMed  Google Scholar 

  184. Luedtke AR, van der Laan MJ. Super-learning of an optimal dynamic treatment rule. Int J Biostat. 2016;12(1):305–32.

    Article  PubMed  Google Scholar 

  185. Murphy SA. Optimal dynamic treatment regimes. J R Stat Soc Series B Stat Methodol. 2003;65(2):331–55.

    Article  Google Scholar 

  186. Robins JM, editor. Optimal structural nested models for optimal sequential decisions. Proceedings of the second seattle Symposium in Biostatistics: Springer; 2004.

  187. Van Der Laan MJ, Rubin D. Targeted maximum likelihood learning. Int J Biostat. 2006;2(1):19.

  188. Gupta SK. Intention-to-treat concept: a review. Perspect Clin Res. 2011;2(3):109–12.

    Article  PubMed  PubMed Central  Google Scholar 

  189. Robins JM, Rotnitzky A, Zhao LP. Estimation of regression coefficients when some regressors are not always observed. J Am Stat Assoc. 1994;89(427):846–66.

    Article  Google Scholar 

  190. Schnitzer ME, Lok JJ, Gruber S. Variable selection for confounder control, flexible modeling and collaborative targeted minimum loss-based estimation in causal inference. Int J Biostat. 2016;12(1):97–115.

    Article  PubMed  Google Scholar 

  191. Robins JM, Rotnitzky A. Recovery of information and adjustment for dependent censoring using surrogate markers. In: AIDS epidemiology: Springer; 1992. p. 297–331.

    Chapter  Google Scholar 

  192. Bunouf P, Molenberghs G. Implementation of pattern-mixture models in randomized clinical trials. Pharm Stat. 2016;15(6):494–506.

    Article  CAS  PubMed  Google Scholar 

  193. Stroup WW. Generalized linear mixed models: modern concepts, methods and applications: CRC press; 2012.

    Google Scholar 

  194. Daniels MJ, Jackson D, Feng W, White IR. Pattern mixture models for the analysis of repeated attempt designs. Biometrics. 2015;71(4):1160–7.

    Article  PubMed  PubMed Central  Google Scholar 

  195. West Virginia State Organization. Depression and Bipolar Support Alliance. Accessed 30 Oct 2020.

  196. West Virgnia Clinical and Translational Science Institute. West Virginia practice-based research network. Accessed 30 Oct 2020.

  197. Olfson M, Blanco C, Marcus SC. Treatment of adult depression in the United States. JAMA Intern Med. 2016;176(20):1482–91.

    Article  PubMed  Google Scholar 

  198. Czeisler MÉ, Lane RI, Petrosky E, Wiley JF, Christensen A, Njai R, et al. Mental health, substance use, and suicidal ideation during the COVID-19 pandemic—United States, June 24–30, 2020. MMWR Morb Mortal Wkly Rep. 2020;69(32):1049.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  199. Twenge JM, Joiner TE. US Census Bureau-assessed prevalence of anxiety and depressive symptoms in 2019 and during the 2020 COVID-19 pandemic. Depress Anxiety. 2020;37:954-6.

Download references


We gratefully acknowledge the entire study team at the West Virginia University Injury Control Research Center (Mary Holleran, Tiffany Davis, Aneva Holleran, Samantha Pierson, Maria Martik) and Harvard Medical School (Marrena Lindberg, Casper Kessler, Masha Petukhova); our Scientific Advisorsy Committee; (Phyllis Foxworth; Pamela Greenburg, MPP; Sally Hodder, MD; Kari Beth Law, MD; William D. Lewis, MD, FAACP; Ron Manderscheid, PhD; Jennifer McVey; Emily Moore; Thomas Parry, PhD; Larry A. Rhodes, MD; H. David Sanders; Diana Thompson; Bruce Rollman, MD, MPH; Christina Schreiber, LPC; Joshua Austin, MA, MSc; Pim Cuijpers, PhD; Andrew A. Nierenberg, PhD); our Implemenetation Monitoring Committee; (Bea Herbeck Belnap, PhD; William D. Lewis, MD, FAACP; Samantha Mann, MD; Jennifer McVey; Emily Moore; H. David Sanders; Christina Schreiber; Diana Thompson); our Data Safety Monitoring Board (Wilfred Pigeon, PhD; Michael Thase, MD; Jeff Huffman, MD; Robert Ursano, MD; John Campo, MD); partnership with the West Virginia Practice Based Research Network and the West Virginia Primary Care Association for participant recruitment; partnership with the Depression and Bipolar Support Alliance; and the West Virginia Clinical and Translational Science Institute. We would also like to acknowledge continued partnership with the SilverCloud team (Derek Richards, PhD; Angel Enrique, PhD; Jorge Palacios, MD, PhD; Olivia Cerf, BS).


The trial is funded by the Patient-Centered Outcomes Research Institute, contract number PCS-2017C3-9252, awarded to Robert M. Bossarte and Ronald C. Kessler. The funder was not involved in study design and will not be involved in study execution, writing of reports, or decisions to submit reports for publication.

Author information

Authors and Affiliations



RMB and RCK drafted the protocol. All other coauthors provided critical revisions and read as well as approved the final manuscript.

Authors’ information

Authors RMB, WRP, HL, and SW are employees of the United States (U.S.) Department of Veterans Affairs (VA). The views or opinions expressed herein do not necessarily reflect those of the U.S. Government or the VA.

Corresponding author

Correspondence to Robert M. Bossarte.

Ethics declarations

Ethics approval and consent to participate {24}

This study was approved by the West Virginia University Institutional Review Board, protocol number 1812375609. Every participant will be required to provide informed consent prior to enrolling in the study.

Consent for publication {32}

Not applicable.

Competing interests {28}

In the past 3 years, RCK has been a consultant for Datastat, Inc, Sage Pharmaceuticals, and Takeda. WRP consulted for CurAegis Technologies and received clinical trial support from Pfizer, Inc and Abbvie, Inc. AE and DR are employees of SilverCloud Health, developers of computerized psychological interventions for depression, anxiety, stress, sleep, resilience, and comorbid long-term conditions. In the past 3 years, JT has received research support from Otsuka. MWH has received funding from Insightec and has been a paid grand rounds speaker for Medtronic and Atrium Health. AAN serves on scientific advisory boards for Alkermes, Jazz Pharmaceuticals, Sage Pharmaceuticals, Otsuka, and Neuronetics; was a consultant for Acadia Pharmaceuticals, Esai, Myriad, Merck, Ginger, Protogenics, Neurogenics and Clexio; and also reports receiving honoraria from Sunovion and Neurostar.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bossarte, R.M., Kessler, R.C., Nierenberg, A.A. et al. The Appalachia Mind Health Initiative (AMHI): a pragmatic randomized clinical trial of adjunctive internet-based cognitive behavior therapy for treating major depressive disorder among primary care patients. Trials 23, 520 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: