In the ‘Tailored Implementation for Chronic Diseases (TICD)’ project, five tailored implementation programs to improve healthcare delivery in different chronic conditions have been developed. These programs will be evaluated in distinct cluster-randomized controlled trials. This protocol describes the process evaluation across these trials, which aims to identify determinants of change in chronic illness care, to examine the validity of the tailoring methods that were applied, and to analyze the association of implementation activities and the effectiveness of the program.
A multilevel approach was used to develop five tailored implementation interventions. In order to guide the process evaluation in five distinct trials, the study protocols for the cluster randomized trials and the related process evaluations were developed simultaneously and iteratively.
The process evaluation comprises three main components: a structured survey with health professionals in the trials, semi-structured interviews with a purposeful sample of this study population, and standardized documentation of organizational practice characteristics. Norway will only conduct the qualitative part of the analysis because the survey and documentation of practice characteristics are considered to be not feasible. The evaluation is guided by ‘logic models’ of the implementation programs: frameworks that specify the linkages between the strategies used, the determinants addressed by tailoring, and the anticipated outcomes. Standardization of measures across trials is sought to facilitate analysis of aggregated data from the trials.
This process evaluation will need to find a balance between standardization of methods across trials and the tailoring of measures to the specificities of each trial.
‘Tailored interventions’ are commonly defined as interventions that are designed to address previously identified determinants of practice (also called ‘barriers to and enablers of change’) . A range of different methods has been used for tailoring, but there is little insight into how tailoring is best conducted in order to optimize the effectiveness of interventions . The overall aim of the ‘Tailored Implementation for Chronic Disease (TICD)’ project is to provide insight into methods for tailoring implementation programs to determinants of implementation of evidence-based healthcare for patients with chronic illness . Research teams from five European countries participate in the TICD project, each focusing on a specific clinical condition: chronic obstructive pulmonary disease (COPD) in Poland, obesity in the United Kingdom, cardiovascular diseases in The Netherlands, depression in the elderly in Norway, and multimorbidity in Germany.
Addressing the identified determinants of practice in chronic illness care requires complex interventions, meaning interventions that have multiple components which interact with each other. In the TICD project, five tailored implementation interventions were developed, each of which will be evaluated in a cluster-randomized trial [4–8].
A frequent criticism of evaluations of complex interventions is their lack of detail in both description and analysis on what components of the intervention and which contextual factors contributed to the (lack of) effectiveness . We planned process evaluations in the TICD trials aiming to address this by exploring which factors are associated with change [10, 11]. This knowledge is essential for understanding the potential causal mechanisms underlying the effectiveness of interventions as well as for ensuring generalizability of intervention effectiveness across populations and settings . At the same time, this will provide insight into the validity of tailoring methods that were applied in earlier phases of the TICD project.
A factor of specific interest is the degree to which the offered interventions were applied as planned by the participants (depending on the context this has been labeled ‘fidelity’ or ‘integrity’ of the intervention ). This knowledge is important to draw valid conclusions about intervention effectiveness and the underlying change mechanisms. Potentially useful interventions may appear ineffective due to low or mistaken application as compared to the original plan . On the other hand, low fidelity or integrity may reflect the adaptation of interventions in the delivery phase to optimize their effectiveness.
Here, we aim to describe the planned process evaluations that are intended to answer the following questions:
To what extent did the practitioners or patients use the various components of the planned intervention programs?
What is the validity of the methods used to tailor the implementation program to determinants of practice?
Which factors are associated with effective implementation of evidence-based practice in chronic illness care, after use of a tailored implementation program?
How consistent are the findings regarding the previous questions across the five trials?
Tailoring concept within the Tailored Implementation for Chronic Diseases project
The overall approach in the TICD project has been described elsewhere  and is summarized here. As depicted in Figure 1, the tailoring process in the TICD project is divided into five phases. In the first phase, a checklist of determinants of practice was developed to support the design and evaluation of implementation strategies . Additionally, each team identified evidence-based recommendations and relevant literature for the targeted conditions. In the second phase, determinants of practice were identified in all countries. A combination of methods was used for this purpose: brainstorming and group interviews with health professionals, individual interviews with health professionals and patients, and a written survey for health professionals. The identified determinants were evaluated according to their plausible importance and the degree to which they could be addressed in an intervention study. In the third phase, different methods (open group interviews and structured group work with health care professionals, researchers and other stakeholders) were used to identify strategies to overcome the identified barriers for the specific recommendations. In the fourth phase, the identified strategies were prioritized on the basis of the criteria ‘feasibility’ and ‘assumed impact’. These four phases have been conducted in all countries, and based on this prior work, five tailored implementation interventions were elaborated, each of which will be evaluated in a cluster randomized controlled trial [4–8]. In the delivery phase, implementation interventions may be adapted by adjusting interventions to individual or practice-related factors.
Development of a process evaluation protocol
The study protocols for the cluster randomized trials and the related process evaluations were developed simultaneously and iteratively, using email and telephone conversations among research teams to provide feedback and discuss controversial aspects. The project coordinator monitored and protected the coherence of the protocols and their fit with the overall aims of the TICD project.
To facilitate the measurement and analysis all TICD implementation programs will be operationalized into a ‘logic’ (or ‘explanatory’) model comprising an anticipated model of change mechanisms based on the results of the tailoring process of each implementation intervention. These models describe the hypothesized linkages between determinants, interventions, and outcomes (Figure 2).
Setting/target groups of the process evaluation
The intervention programs in the TICD project are situated in primary care settings and target physicians, nurses, and patients and their relatives. The study population in the process evaluation comprises the healthcare professionals of the intervention group, however implementation activities (whether planned or not) will be documented in the control arms as well.
The process evaluation has three main items: a survey with two components (A1 and A2), an interview study (B), and documentation of practice characteristics (C).
A1 - Survey on perceived change of determinants of practice
We will perform written surveys in participating health professionals at the primary follow-up measurement in the trials (except in the Norwegian trial, where this survey is considered to be not feasible). As the content of the studies differs, we have developed a frame of the questionnaire that needs to be adapted to the relevant content (Table 1). The questionnaire lists the determinants of practice, which were identified and prioritized in an earlier phase of the TICD project (regardless of whether they are addressed by the implementation program), with the question to assess whether the program successfully targeted them. A free text field is used to identify other determinants that are lacking from this list and are perceived to have major impact.
A2 - Survey to document implementation activities
The survey of participants also contains questions on the actual implementation activities. We intend to record the extent to which the target group used the offered interventions and adaptions on it in the delivery phase of the implementation programs. Table 2 shows the frame of the questionnaire used in all studies to document implementation activities. The design of the questionnaire is based on a published framework , which distinguishes four aspects of intervention fidelity:
Content - Was the content of the intervention delivered as planned?
Duration - Was the intervention fully implemented across the intended time period?
Frequency - Was the intervention conducted as frequent as planned?
Coverage - Was the intervention applied to all individuals in the targeted group?
For each implementation program the core components will be specified. For each core component, content, duration, frequency and coverage will be recorded in a structured way (Table 2). The content of the items will be specific for the different trials. A free text field is added to identify strategies that have been missed in the tailoring process so far, allowing the evaluation of the methods used for tailoring.
B - Interviews with health professionals of the intervention group
The third component of the process evaluation comprises of semi-structured interviews with participants in the intervention arms of the trials. The face-to-face interviews (or, alternatively, telephone interviews) will be done with a purposive sample of health professionals. The exact phrasing of questions will differ across trials, but the following interview format will serve as a starting point for all interviews:
What made you participate in the project in the first place? What where your reasons and what were your expectations?
Did the implementation program help you to adhere to the recommendations?
If yes, what components did you find helpful and why?
If no, why not and what strategies would have been more helpful?
Were there any other factors or developments which made it difficult for you or helped you to adhere to the recommendations? You can think of changes in regional or national healthcare, developments in the practice organization and team or events in your life and work.
Having experienced the program, what would you recommend for the future? You may think of further development, wider implementation or perhaps research.
Interim analysis after 5 to 10 interviews will be done to adapt the interview format and purposeful sampling scheme as required.
C - Practice characteristics and other contextual factors
The success of an implementation program may be influenced by factors, which cannot, or at least cannot easily, be addressed, such as the structure of the health system, laws or the organizational structure of the targeted settings . The TICD studies are focused on primary care. In four TICD studies a number of practice characteristics will be recorded in both study arms (Table 3). This is not feasible in the Norwegian study, where municipalities are the unit of randomization. In Norway, information regarding relevant characteristics of the municipalities and specialist health care that may modify the effectiveness of the interventions will be collected instead.
We will use the following analytical approach to answer the research questions addressed by this process evaluation:
Research question 1 - To what extent did the practitioners or patients use the various components of the planned intervention programs? and
Research question 2 - What is the validity of the methods used to tailor the implementation program to determinants of practice?In order to answer these questions a descriptive analysis of qualitative and quantitative data will be done separately in each country. The descriptive analysis of questionnaires will result in tables with frequency distributions. The qualitative interviews will be transcribed verbatim in the original languages. An iterative thematic analysis will be done, involving two or more researchers in order to identify key themes relevant for implementation.
Research question 3 - Which factors are associated with implementation of evidence-based practice in chronic illness care, after use of a tailored implementation program?
To answer this research question a summary outcome measure will be constructed in each trial, expressing professional performance by aggregating items reflecting implementation of evidence-based recommendations. This outcome will be standardized and taken as dependent variable in a regression analysis approach. The independent variables in this model will be a limited number of questionnaire items (to reduce chance capitalization), selected according to hypothesis informed by relevant key themes of the interviews. A special focus will be given to the association between professional performance and the questionnaire items concerning the application and adaption of the implementation program by the target group.
Research question 4 - How consistent are the findings across the trials?
A meta-analysis of quantitative and a meta-ethnography of qualitative data from all five studies will be conducted.
A qualitative meta-ethnography will be conducted using the data from the interviews. For this purpose a description of subcategories with significant quotations from all studies will be provided in English. This material will be analyzed according to the principles of Noblit and Hare’s reciprocal translational analysis, which is recommended to be used when analyzing studies with similar topics [17, 18]. The result of this analysis will be a set of key themes relevant in all studies. Interviews will be re-analyzed to explore how each single study contributes to the overall key themes. In a last step third order interpretation across the key themes will be considered if supported by a ‘line of argument’ across all studies.
A quantitative meta-analysis will be conducted, focused on the summary outcome measure in each trial, using the predictors that are identified as result from the analysis for question 3. While the items composing this aggregate outcome obviously differ among the trials, the assumption is that the resulting measures are reasonably comparable to allow pooling. Based on hypotheses derived from phase 1 and 2, individual data and/or clustered data from all trials will be included in a meta-regression analysis. This last step will efficiently make use of the total number of cases included across all studies and the shared instruments as described in this protocol.
Process evaluation is essential to understand the effects and to explore potential causal mechanisms of complex interventions. The planned process evaluation in the TICD project needs to find a balance between standardization of methods across trials and adaptation of measures to the specific characteristics of each trial. Despite obvious limitations, such as heterogeneity and suboptimal statistical power, we believe that the planned study will contribute to our knowledge on determinants of change in chronic illness care and methods for tailoring implementation programs to barriers and facilitators of change.
The trial is currently in the planning phase.
Tailored Implementation for Chronic Diseases.
Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, Robertson N: Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010, 3: CD005470-
Bosch M, van der Weijden T, Wensing M, Grol R: Tailoring quality improvement interventions to identified barriers: a multiple case analysis. J Eval Clin Pract. 2007, 13: 161-168. 10.1111/j.1365-2753.2006.00660.x.
Krause J, Agarwal S, Bodicoat DH, Ring A, Shepherd D, Rogers S, Wensing M, Baker R: Evaluation of a tailored intervention to improve management of overweight and obesity in primary care; study protocol of a cluster randomised controlled trial. Trials. 2014, (submitted)
Godycki-Cwirko M, Zakowska I, Kosiek K, Wensing M, Krawczyk J, Kowalczyk A: Evaluation of a tailored implementation strategy to improve the management of patients with chronic obstructive pulmonary disease in primary care – study protocol of a cluster randomized trial. Trials. 2014, (submitted)
Aakhus E, Granlund I, Odgaard-Jensen J, Wensing M, Oxman AD, Flottorp SA: Tailored interventions to implement recommendations for elderly patients with depression in primary care: a study protocol for a pragmatic cluster randomised controlled trial. Trials. 2014, 15: 16-10.1186/1745-6215-15-16.
Huntink E, Heijmans N, Wensing M, van Lieshout J: Effectiveness of a tailored intervention to improve cardiovascular risk management in primary care: study protocol for a randomised controlled trial. Trials. 2013, 14: 433-10.1186/1745-6215-14-433.
Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B: Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013, 14: 15-10.1186/1745-6215-14-15.
Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ: Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011, 89: 167-205. 10.1111/j.1468-0009.2011.00625.x.
Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, Baker R, Eccles MP: A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013, 8: 35-10.1186/1748-5908-8-35.
Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill R, Yardley L, Pope C, Donovan J: Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess. 2011, 15: 1-164.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.
Jäger, C., Freund, T., Steinhäuser, J. et al. Tailored Implementation for Chronic Diseases (TICD): a protocol for process evaluation in cluster randomized controlled trials in five European countries.
Trials15, 87 (2014). https://doi.org/10.1186/1745-6215-15-87