Lessons learned from Integrated Management Program Advancing Community Treatment of Atrial Fibrillation (IMPACT-AF): a pragmatic clinical trial of computerized decision support in primary care

Abstract

Background

Integrated Management Program Advancing Community Treatment of Atrial Fibrillation (IMPACT-AF) was a pragmatic, cluster randomized trial assessing the effectiveness of a clinical decision support (CDS) tool in primary care, Nova Scotia, Canada. We evaluated if CDS software versus Usual Care could help primary care providers (PCPs) deliver individualized guideline-based AF patient care.

Methods

Key study challenges including CDS development and implementation, recruitment, and data integration documented over the trial duration are presented as lessons learned.

Results

Adequate resources must be allocated for software development, updates and feasibility testing. Development took longer than projected. End-user feedback suggested network access and broadband speeds impeded uptake; they felt further that the CDS was not sufficiently user-friendly or efficient in supporting AF care (i.e., repetitive alerts).

Integration across e-platforms is crucial. Intellectual property and other issues prohibited CDS integration within electronic medical records and provincial e-health platforms. Double login and data entry were impediments to participation or reasons for provider withdrawal. Data integration challenges prevented easy and timely data access, analysis, and reporting.

Primary care study recruitment is resource intensive. Altogether, 203 PCPs and 1145 of their patients participated, representing 25% of eligible providers and 12% of AF patients in Nova Scotia, respectively. The most effective provider recruitment strategy was in-office, small group lunch-and-learns. PCPs with past research experience or who led patient consent were top recruiters. The study office played a pivotal role in achieving patient recruitment targets.

Conclusions

A rapid growth in healthcare data is leading to widespread development of CDS. Our experience found practical issues to address for such applications to succeed. Feasibility testing to assess the utility of any healthcare CDS prior to implementation is recommended. Adequate resources are necessary to support successful recruitment for future pragmatic trials. CDS tools that integrate multiple co-morbid guidelines across eHealth platforms should be pursued.

Trial registration

ClinicalTrials.gov NCT01927367. Registered on August 22, 2013

Background

Atrial fibrillation (AF) is a common chronic condition associated with increased mortality, substantial morbidity, and high health care costs [1]. It is also an independent risk factor for stroke and, as such, clinical guidelines recommend antithrombotic therapy for stroke prevention in most patients with AF [2]. Despite national guidelines, gaps in provider knowledge and patient care have been documented [3, 4]_.

Integrated Management Program Advancing Community Treatment of Atrial Fibrillation (IMPACT-AF) was a pragmatic, cluster randomized trial assessing the clinical relevance and effectiveness of a clinical decision support (CDS) tool in the primary care setting of Nova Scotia, Canada [5]. The IMPACT-AF study methods and main results have been published [5, 6]. Briefly, clinical and health informatics researchers developed and evaluated if CDS software could help primary care providers (PCPs) better deliver individualized AF patient care based on national guidelines, thereby improving process of care and patient health outcomes at 12 months. No significant effects on the primary efficacy or safety outcomes were observed at follow-up [6].

Need for documentation of lessons learned

The use of electronic medical records (EMRs) has been increasing among community-based clinicians. CDS tools have begun to appear as well, although few of those available in Canada and elsewhere have been validated through randomized controlled trials [7,8,9,10]. As such, limited information has been published on the potential challenges that might arise when developing and implementing a point-of-care decision support software application in a clinical trial setting, whether for atrial fibrillation or any other medical condition. The purpose of this paper is to offer insight into the key challenges faced in the development, implementation, and assessment of a CDS software, with lessons learned that would support the success of future work in this area of practice. We also provide suggestions for other researchers based on our experience and learnings.

Methods

Following ethics approval by the Nova Scotia Health Authority Research Ethics Board, software design began in 2013. The goal was to create a fully integrated application that computerized national clinical guidelines into decision rules to support AF patient management at point-of-care (Fig. 1). A key CDS feature was prioritized automated alerts signaling material changes in patient clinical or biochemical profiles requiring expedited treatment modifications. Users, both providers and patients, could interact with the software by entering and receiving health-related information and or alerts. Where possible, relevant patient data could be captured in real time (Fig. 1) [11].

Fig. 1

CDS design. AF, atrial fibrillation; ECG, electrocardiogram

From 2014 to 2016, PCPs and their patients with AF were identified, recruited, and randomized 1:1 to CDS access (n = 104 PCPs, n = 597 patients) or usual care (n = 99 PCPs, n = 548 patients). Consented patients were followed for 12 months with assessments of process of care and health outcomes conducted. Throughout the study duration, the research team documented key challenges regarding CDS development and implementation (2013–2018), data integration (2013 to 2020), and study recruitment (2014–2016).

Results

Lessons learned

CDS-related challenges

Key lessons: Sufficient resources must be allocated for software development, feasibility testing, and software updates, and the integration across e-platforms is crucial for CDS use and study data (timely access, analysis, and reporting).

Bringing together clinical experts with a largely non-clinical health informatics team exposed many challenges (Table 1) and, partly as a result, CDS deployment took longer than projected. The most significant challenge concerned software development, which was quite complex given the objective of providing more than unidimensional functionality, such as simply supporting prescribing of antithrombotic therapy. A technology advisory committee was created to provide guidance and input into software design, and a small convenience sample of PCPs (n = 6) provided practical input based on pilot testing over 3 months. Figures 1 and 2 illustrate the CDS design and information flow. Programming was intricate, and the CDS needed to undergo considerable modification in response to user feedback, including updates to screen layouts and the addition of new functionalities. Despite such efforts, user assessments gathered during and at the end of the study indicated that the CDS was not user-friendly and did not create sufficient efficiencies in the management of their AF patients. Reasons cited included too many “clicks” to accomplish a given task, or repeated alerts for tasks already addressed that needed to be manually cleared, contributing to “alert fatigue” [12]. Although the clinical research team anticipated the ability to modify the CDS features and functions over time, the health informatics team favored a non-iterative approach, limiting system modification of the product once it was launched. This hampered the ability to have the software enhanced, updated, or modified to best meet end-user needs as identified overtime with an increasing number of users and ongoing CDS use.

Table 1 CDS development and implementation challenges and strategies

Fig. 2

CDS information flow. DB, database; CDS, computer decision support; VPN, virtual private network

During the time of CDS development, there were three popular EMRs in Nova Scotia primary care. While the CDS software was built using standardized computer programming language, it could neither be integrated within desired provincial e-health systems nor, perhaps more importantly, embedded within any of the EMRs especially due to proprietary issues, but also cost and time constraints. The lack of EMR integration meant that a greater burden of responsibility was placed on PCPs to enter relevant AF patient data such as changing medications or bloodwork results themselves in order to ensure the intended operational functionality of the CDS. The time and effort required to do this ran contrary to the intended purpose of the tool.

As noted earlier, a key CDS feature was prioritized automated alerts signaling material changes in patient clinical or biochemical profiles requiring expedited treatment modifications. Since EMR integration was not feasible, for bloodwork essential in the monitoring AF patient care, a work-around to capture test results from provincial laboratories and have them available in the CDS was created. For context, bloodwork results from provincial laboratories, the most commonly used blood collection services, are routinely available electronically to ordering physicians in Nova Scotia. Test results from the limited number of private laboratories however are not integrated into the Provincial electronic system. PCPs in the intervention arm were instructed to “copy” the study (using a dedicated identification code) on all test requisitions submitted to provincial laboratories with data relevant to study patients. The applicable CDS fields could then be auto-populated with the electronic results once posted. For those providers using private laboratories, manual entry of test results was required, creating yet another burden for CDS use. Remembering to indicate on test requisitions for consented patients that results be copied to the study was problematic, with only 29.2% (n = 26) doing so. An additional 10 PCPs used the manual data entry option. Thus, in total, 36 (40.4%) of 89 PCPs eligible for the intervention had recorded labs in the CDS.

The lack of EMR integration also meant that providers had to log into two separate platforms, their EMR and the CDS, to record relevant details for consented study patients. The study office guided providers on the minimum dataset required (e.g., patient blood pressure, medication or weight changes) to maximize the benefit of the CDS (run the decision rules) while minimizing provider workload. For many, double login and data entry were notably cited as impediments to optimal CDS use and as reasons for CDS provider withdrawal. Internet access and slow network broadband speeds also proved to be key challenges for successful CDS uptake, particularly in rural areas. Such areas contain just under half of the province’s population and the study purposefully sought to sample and recruit so as to reflect this.

The challenges with data integration across e-health platforms also prevented easy and timely data access, analysis and reporting. As noted, initial design plans (Fig. 1) were for the CDS to receive real-time health data (e.g., labs, echocardiogram reports, cardiovascular [CV] hospitalization and or AF-related emergency department [ED] visit data) from various sources. After ethical and privacy impact assessment, and in discussion with relevant stakeholders including provincial information technology specialists, this was not deemed to be achievable for this research initiative. The study then strove for quarterly data transfers, but ultimately this was also unfeasible due to the inability to readily link electronic provincial health datasets with the study database. As noted previously, this limited the operational functionality of the CDS (i.e., lack of new data to run the decision rules and trigger alerts based upon changing medical biochemical parameters of consented patients). In the end, the study team had to access and manually record baseline data directly into the study database from the provider’s EMRs. This activity (recording baseline data into the study database) did pre-populate each intervention provider’s CDS with historical patient data from which to begin their active PCP study phase.

In order to access patient charts and or data remotely, many steps had to be identified then taken to ensure both the ability to access and maintain the privacy and security of the required information (see Table 2). While the set-up procedure was a one-time occurrence, the processes and obstacles faced were challenging. However, having the ability to access data remotely offered several benefits, with reduced travel for study staff and less in-office disruptions for providers. In addition, the need to utilize and securely store vast amounts of paper-based study records (e.g., case report forms) was eliminated.

Table 2 Steps necessary for remote primary care data collection

Delays were experienced in retrieving 12-month patient health outcomes, specifically ED visits, hospitalizations, and especially mortality. Some of these key health outcome variables were only available from on-site reviews of hospital charts. This required considerable travel across the province, which has an area slightly larger than that of Vermont and New Hampshire combined, at the cost of human and financial resources as well as time expenditures that had not been anticipated. Thus, while the last patient completed their 12-month follow-up in January 2018, the analysis of study outcomes was delayed until the Fall of that year.

Collecting data from multiple sources (provincial systems, provider EMRs and patients themselves) created challenges of data integration and analysis. Patient-reported event timelines often did not overlap with those recorded in the PCP charts, limiting the ability to compare patient self-reported events with administrative health data. The primary source for health outcomes data were hospital-based records, accessed remotely where feasible and at times in person, as noted above. These data were cross-referenced with provincial health datasets, providers’ EMRs, and patient self-reported data for completeness.

Study recruitment and engagement

Key lesson: Primary care participant recruitment was resource intensive. Significant effort (time and human resources) was required to visit medical clinics and build rapport with PCPs’ offices, including administrative staff who often play critical roles in supporting providers’ research activities.

Based upon sample size calculations, it was estimated that upwards of 200 PCPs across the province would be required in order to meet patient recruitment goals. Looking back, this was an aggressive target considering that there were only approximately 1000 PCPs within the province at the time. Once ineligible providers were excluded (those without high-speed internet and or primarily involved in speciality work such as pediatrics, palliative care, addiction services or ED coverage), the recruitment pool was 827 (including primary care nurse practitioners). While it took 2 years to achieve (from start to final enrollment, 2014–2016), the IMPACT-AF study was successful in recruiting and randomizing 203 PCPs, representing approximately one quarter of all eligible providers in the province [6]. The recruitment strategies employed, along with their respective impact levels, are found in Table 3. The most successful strategy was resource-intensive, involving in-office small group lunch-and-learns, organized and implemented across the province. Up to 10 points of contact (including email, faxes, telephone calls, webinars, in-office visits and continuing medical education events) were required to recruit an individual provider. The most common reason for non-participation of potentially eligible providers was time constraints (due to a lack of staff, staff turnover, current participation in other research, or the provider practicing in multiple locations). Other reasons included practice size deemed to be too small, a simple lack of interest in the research topic, concerns with “double-data entry” (given that the intervention tool was not contained within the providers’ EMR) and or privacy concerns (since the study abstractors had to access and review the PCP’s charts of consented patients for relevant study data). Reasons cited for provider withdrawal over the duration of the study included lack of time or human resources to conduct planned study activities such as dual EMR/CDS login and data entry, too much effort to recruit patients and retirement or departure from the practice [6, 13].

Table 3 Provider identification, recruitment, and engagement strategies

Patient recruitment and engagement

In order to meet sample size calculations, the proposed patient recruitment target was 1075. With a provincial population of just under one million, and an estimated AF prevalence of 1%, the study required approximately 10% of all patients living with AF to participate [14, 15]. For ethical reasons, PCPs were required to be the initial point of contact for patient identification. Providers were shown methods they could utilize to identify their own eligible AF patients (e.g., review of patient records or billings). The specific strategies used for patient recruitment and their impact are listed in Table 4. Traditional resources for advertising the study to patients, such as posters, pamphlets, and standardized invitation letters, were produced and shared. Early on, the research team deemed that this more passive approach would not be sufficient given the project timeline and milestone deliverables. The team also realized the critical role that front-office administrative staff can play in supporting research activities, including patient recruitment. Accordingly, primary care clinic staff were provided with training and supports in order to identify potentially eligible patients. Unplanned study resources, such as the hiring of additional study staff to provide assistance and consent patients were required to accelerate these efforts.

Table 4 Patient identification, recruitment, and engagement strategies

In our study, the average cluster size for active providers at 12 months (n = 77 usual care, n = 89 CDS, as per the Consort flow diagram [6]) was 6.8 patients. There were 46 PCPs (27.7%) who recruited between six and seven patient participants. Another 49 (29.5%) PCPs recruited more than six patients, while 71 (42.8%) enrolled fewer than this. Ultimately, the 1133 study patients included in the outcomes analysis represented an estimated 12% of all Nova Scotia residents with AF [6]. PCPs with past research experience or who led patient consent were top recruiters.

In this study, there was no single patient recruitment strategy that was clearly superior at maximizing patient recruitment, but rather a combination of approaches whose assortment often differed between practices. In regard to retention, there was a single issue viewed negatively by both patients and providers that led to study withdrawal. This related to what was felt to be too-frequent contact by the study office, whether by phone or email, in order to obtain follow-up information or solicit study feedback. Once this became clear, the study office attempted to limit the volume of communications as much as possible and to focus on collecting only the information deemed most essential. The aspect of the study affected was the qualitative one, with less information that aimed to assess the personal burden and economic costs of AF being passed on by patients over time.

Discussion

Despite the many challenges faced by the IMPACT-AF research team, there were some successes, such as recruitment of highly representative samples of primary care providers and their patients with AF from within the small province of Nova Scotia, Canada, along with the rigorous assessment of a CDS tool. Disappointingly, however, clinically significant improvements in the primary study outcome (a composite of unplanned CV hospitalizations and AF-related ED visits) in favor of the CDS were not observed at 12-month follow-up [6].

Based upon our learnings, we would propose several suggestions for any investigators planning to undertake a pragmatic clinical trial of clinical decision support tools. As a fundamental first step, appropriate expertise and sufficient resources should be allocated for CDS software development, adequate feasibility testing, software updates as required, and technology support. A dedicated and robust testing and feasibility assessment phase, with an iterative design process whereby end-user feedback can be readily incorporated to improve the usefulness of the application would be highly recommended. For point-of-care CDS tools to be useful, they must meet the needs of the end users, be they physicians and or patients. Ideally, individuals with a background in both Medicine and Health Informatics should be engaged at an early stage in CDS development as they will have especially important insights to provide in regard to clinical content and software algorithms. However sophisticated its software, the success of any CDS will ultimately depend on its value to its intended users. Accordingly, effort should be invested into ensuring that it is user friendly, provides work efficiency, and has a clear and attractive interface that is easy to navigate. To this end, a representative sample of the targeted end-users (e.g., practicing primary care providers and their patients) needs to be consulted throughout the design process, allowed sufficient time to test it, and have their feedback carefully considered. Software updates should be provided as soon as needed to address any “bugs,” glitches, or other unforeseen problems negatively impacting the application. Information technology support, preferably by phone, but also via on-line means, with an easy-to-understand user manual, or a combination of some or all of the above, should be made available. Such comprehensive and timely software updates and technology support could not be provided in the IMPACT-AF study.

We also recommend that any CDS be fully integrated with, or capable of seamlessly interacting with, any related applications used by physicians or patients, such as EMRs or patient health tracking devices, to increase the likelihood of use. Some studies have shown that even when CDS software or automated alerts are fully embedded within EMRs, these features can be under-utilized [15,16,17]. Based upon our experience, any platform that requires double login and data entry is simply not worth pursuing as it is unlikely to be much used, if at all. This point cannot be over stressed. Despite PCPs interest, desire and willingness to participate in the research, duplicating work in the real-world setting was not sustainable for most busy clinicians. Another requirement for CDS software to function optimally is to have them integrated with as many clinical datasets as possible across a given healthcare system. Comprehensive patient care requires consideration of the individual as a whole, not just one medical condition (e.g., AF) or a component of that one condition (prescription of antithrombotic medication). In the future, CDS tools capable of integrating multiple disease guidelines as required to manage multi-morbid patients, and that can seamlessly integrate with patient health monitoring devices and eHealth platforms including EMRs, are those most worth pursuing.

The ability to have timely access, analysis and reporting of data for quality assurance and improvement initiatives, as well as for clinical research, is imperative. A field documenting patient consent for contact regarding clinical trial involvement or to allow deidentified use of their data could readily be included in the CDS. With heightened concerns over potential privacy breaches however, considerable security and precautions are required to protect patient confidentiality.

Emphasizing the potential benefits accruable to PCPs for their participation in a study of a CDS tool, rather than focusing simply on the potential to improve patient outcomes, seemed to enhance provider recruitment. In particular, highlighting the possibility that use of such decision support can result in a more efficient workflow might be an effective inducement for study enrolment. We would recommend engaging well-respected peer champions to help promote the benefits of CDS applications in enabling patient care and workflow and as well in helping with physician recruitment. Ultimately, the study office had to enhance significantly its efforts to better support providers to achieve patient recruitment targets, thus allocating adequate resources—time and human—during the study planning phase would be prudent. Given the critical role clinic administrative staff play in supporting PCPs with patient identification and recruitment, engaging and supporting these individuals is also strongly recommended.

Limitations

The observations presented here are based upon our experiences and challenges faced within the IMPACT-AF study over time (2013–2020) conducted in Nova Scotia, Canada. Other CDS development teams may not have similar experiences, instead utilizing an iterative design process with a robust testing phase and incorporating sufficient modifications based upon end-user feedback. Certainly, the lack of facile data integration across and within health systems is common. The impacts of the recruitment strategies may vary in other jurisdictions and or be dependent upon the resources available to support them, timing of their deployment, and ongoing implementation during a clinical trial.

Conclusion

A rapid growth in healthcare data is leading to widespread development of CDS software to analyze it, better support evidence-informed patient management and thereby improve health outcomes. Our experience of developing and implementing a pragmatic, cluster randomized controlled trial of a CDS in the real-world setting found a variety of practical issues to address if such applications are to succeed. There is a need for the allocation of adequate resources for CDS software development and updates, robust feasibility testing, as well as for primary care study recruitment. Most critical is a need for the integration of applications across e-health platforms. We hope that by sharing our experiences others can learn and achieve greater success in their future pragmatic clinical trials of CDS tools.