Skip to main content

The effect of automated audit and feedback on data completeness in the electronic health record of the general physician: protocol for a cluster randomized controlled trial

Abstract

Background

The electronic health record (EHR) of the general physician (GP) is an important tool that can be used to assess and improve the quality of healthcare. However, there are some problems when (re) using the data gathered in the EHR for quality assessments. One problem is the lack of data completeness in the EHR. Audit and feedback (A&F) is a well-known quality intervention that can improve the quality of healthcare. We hypothesize that an automated A&F intervention can be adapted to improve the data completeness of the EHR of the GP, more specifically, the number of correctly registered diagnoses of type 2 diabetes and chronic kidney disease.

Methods

This study is a pragmatic cluster randomized controlled trial with an intervention at the level of GP practice. The intervention consists of an audit and extended electronically delivered feedback with multiple components that will be delivered 4 times electronically to general practices over 12 months. The data will be analyzed on an aggregated level (per GP practice). The primary outcome is the percentage of correctly registered diagnoses of type 2 diabetes. The key secondary outcome is the registration of chronic kidney disease. Exploratory secondary outcomes are the registration of heart failure, biometric data and lifestyle habits, and the evolution of 4 different EHR-extractable quality indicators.

Discussion

This cluster randomized controlled trial intends to primarily improve the registration of type 2 diabetes in the EHR of the GP and to secondarily improve the registration of chronic kidney disease. In addition, the registration of heart failure, lifestyle parameters, and biometric data in the EHR of the GP are explored together with 4 EHR-extractable quality indicators. By doing so, this study aims to improve the data completeness of the EHR, paving the way for future quality assessments.

Trial registration

ClinicalTrials.gov NCT04388228. Registered on May 14, 2020.

Peer Review reports

Background

The electronic health record (EHR) is an important instrument that, when properly implemented, can be used as a tool to improve the quality of healthcare [1]. However, many challenges need to be overcome to use the vast amount of EHR-stored data for quality assessment [2, 3]. One of these challenges is the lack of data completeness of the EHR and, in particular, the varying quality of the correct and complete registration of diagnoses [4,5,6]. After all, when the quality of care for certain diseases needs to be evaluated, the number of registered diagnoses of these diseases in the problem list of the EHR needs to be as accurate as possible to have reliable results in our quality measurement. In the past, data quality feedback tools and frameworks have been developed to address the problem of the lack of data completeness [7, 8].

Chronic diseases such as diabetes mellitus type 2 (DM 2) and chronic kidney disease (CKD) have a high prevalence, and the quality of primary care for these conditions needs improvement [9, 10]. For example, CKD can be identified in the EHR with the help of the electronic CKD phenotype, which has proven to be an accurate method to detect patients likely to have CKD based on data stored in the EHR [11]. However, there is still the need for effective organizational and technical strategies to achieve data completeness [12]. The electronic CKD phenotype is thus an important first step, but additional pragmatic trials to explore implementation strategies incorporating this electronic CKD phenotype are necessary [13]. An implementation strategy that can be useful for this purpose in primary care could be audit and feedback (A&F).

Audit and feedback is a well-known quality intervention that, according to the last Cochrane review, leads to “small but potentially important improvements in professional practice” [14]. However, the important features of A&F and their influence on the effect of an A&F intervention are still the subject of debate [15]. Previous work identified some testable and theory-informed hypotheses for designing an A&F intervention, and suggestions to improve the interventions’ effectiveness are available in the literature [16, 17]. Some features of the feedback are known to be effective, for example, the frequency of the feedback provision (more than once), but other features, such as the use of benchmarks as a comparison, the evidence-based quality of the feedback, and a low cognitive load of the feedback, need further investigation [14, 16,17,18,19]. Therefore, the aim of this study is to investigate whether an automated A&F intervention, using different features of feedback, can be effective in improving the data completeness of the EHR in primary care.

Methods

Study design and objectives

Study design

The study is a pragmatic cluster randomized controlled trial (CRCT) with a superiority framework and an intervention at the level of the general physician (GP) practice. The protocol is reported according to the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklist [20] (see additional file 1 and Fig. 1).

Fig. 1
figure1

Time schedule. DM 2, diabetes mellitus type 2; CKD, chronic kidney disease; EHR, electronic health record

Primary objective

The primary objective of this study is to improve the quality of registration of DM 2 in GP EHR.

Key secondary objective

  • Improve the registration of CKD

Exploratory secondary objectives

  • Evaluate the registration of heart failure, biometric data, and lifestyle habits (smoking, exercise, weight, height, alcohol use)

  • Two different EHR-extractable quality indicators for DM 2 and two different EHR-extractable quality indicators for CKD will be evaluated to investigate whether an improved registration of these two diseases has an impact on EHR-extractable quality indicators.

Study population, eligibility criteria, and data storage

This trial has GP practices as the level of allocation. All GPs in the Intego network will be asked to participate in this trial. For GPs working in a group practice, the whole group will be asked to collaborate. GP practices are excluded when participation is not unanimous. Intego is a Belgian general practice-based morbidity registration network at the Center of General Practice of the University of Leuven [21]. Intego collects data on health parameters, incidence and prevalence rates, laboratory results, and prescribed drugs for all relevant subgroups. As of 2020, 395 GPs of 106 practices spread throughout Flanders, Belgium, collaborated in Intego. In order to ensure enough GP practices take part in the trial, all the GP practices will be contacted and/or visited individually and asked to take part in this trial. An Intego researcher will explain the study and request informed consent. The researcher will also emphasize that the provision of the feedback is an incentive on its own. Therefore both groups will receive feedback although the feedback in the control group will be basic. In addition, the Intego researcher will also inform the participants that the intervention will not be intrusive since it will be integrated (automated) in the normal workflow of the GP (to optimize the use of the EHR-system).

The inclusion criteria for GP practices participating in this trial are the same criteria as for inclusion in the Intego network, which are explained in more detail elsewhere [21, 22]. Only the aggregated data of patients 40 years or older who had at least one consultation the past year will be analyzed for both primary and secondary outcomes.

The data will be hosted on the Healthdata.be platform, which is described in more detail elsewhere [22].

Intervention

The intervention consists of an extended electronically delivered feedback report with multiple components that will be delivered 4 times electronically into GP practices over 12 months. The extended feedback report is generated based on an automated audit that is built into the EHR of the GP. This audit will be executed on the Intego database in an automated fashion 4 times and is available in both the control and intervention groups. The feedback report for both trial arms will be provided on the independent Healtstat.be platform. The intervention group will also receive the extended feedback report through a push system into their e-health inbox to minimize the effort to consult it. The aim of the push system is to involve participating GPs and to actively direct their attention to the task at hand, namely, the registration of diagnosis and parameters in the EHR.

The extended feedback report will be delivered in the form of action plans and goals consisting of multiple components:

  • Benchmarking of the results of the audit with peers on disease-specific laboratory results or medication prescriptions. The results of the audit will be compared with the mean of the 10% best performers according to the Achievable Benchmarks of Care (ABC) method [23].

  • A low cognitive load of the feedback, such that the results will be presented with the help of graphs and without too many in-depth elements.

  • Guidance on which codes to use for which cases or where to put the information in a structured way.

  • Links to disease-specific guidelines.

  • A push system to minimize the effort the GP must make to consult the feedback.

The aim of the automated audit is to identify the unregistered population, and this built-in audit is available in both study arms. The aim of the extended feedback report is to improve the registration level in the EHR and thus improve the data completeness of the EHR.

The control group will also have access to the automated audit since it is built in the EHR and will receive basic feedback on the level of registration in their EHR. The feedback will only be available on HealtStat.be through a pull system, so GPs will need to actively check Healtstat.be. For a detailed comparison of control versus intervention, please see Table 1.

Table 1 Intego feedback trial: control versus intervention group

Allocation concealment and random sequence generation

The list of participating practices will be known at the start of the trial. The allocation sequence will be performed by a statistician (independent from the research group) and it will be generated using covariate-based constrained randomization [24]. More specifically, out of a list of all acceptable allocations one allocation scheme will be drawn at random. To balance the mean size of the practice between both groups, a list of all possible allocations (restricted to those allocations with equal number of practices in each group) with a maximal difference of 5% in mean number of patients per practice and a maximal difference of 5% in total number of patients will be created and one allocation will be drawn at random from this list [24]. The covariate-based constrained randomization procedure will be implemented using a SAS-macro [25]. If the trial would start before all practices are known, practices with “late entry” will be allocated using minimization [26], categorizing the number of patients per practice into 2 or 3 categories.

The control versus intervention group will be allocated in a 1:1 manner. GPs are not blinded to the intervention. The research facility will be blinded to the allocations until all data are collected.

Outcomes

Our primary outcome is the registration level of DM 2. This registration level will be expressed as the percentage of registered diagnoses of DM 2 in the EHR. The numerator will be the number of registered diagnoses of DM 2 in the EHR, and the denominator will be the number of patients with a prescription of diabetes medication (ATC classification A10) or with elevated levels of HbA1c (at least once > 53 mmol/mol) or with abnormal fasting glycemia (2 independent measurements > 126 mg/dl). For the primary outcome, we hypothesize that the intervention can achieve an improvement of 10% points (from 75% to 85%) in registration level within 12 months.

As a key secondary outcome, we will focus on the registration of CKD. The registration of CKD will be expressed as the percentage of registered CKD diagnoses in the EHR, with the numerator the number of registered diagnoses of CKD and the denominator the number of patients with a decreased (≤ 45 mL/min/1.73 m2) estimated glomerular filtration rate (eGFR). For the registration of CKD, we hypothesize the intervention can achieve an improvement of 20% points (from 50% to 70%) in registration level because preliminary analyses show the quality of registration of CKD is lower than for the primary outcome.

As exploratory secondary outcomes, we will evaluate the registration of heart failure, lifestyle factors (such as smoking and alcohol use), and biometric data (such as weight and length), and explore two EHR-extractable quality indicators for CKD and two for DM 2. The registration of heart failure will be expressed as the percentage of heart failure patients, with the numerator the number of registered heart failure diagnoses and the denominator the total active patient population older than 40. Lifestyle factors and biometric data will be expressed as the percentage of patients who have registered lifestyle factors or biometric data in their EHR.

The EHR-extractable quality indicators we will explore are as follows [27, 28]:

  • Percentage of patients with CKD, in whom the GFR, albuminuria, and total protein is determined at least once a year

  • Percentage of patients with CKD who are vaccinated with a pneumococcal vaccine

  • Percentage of patients with DM 2 and an eGFR of < 30 mL/min/1.73 m2 who no longer receive metformin.

  • Percentage of patients with DM 2 whose HbA1c level is measured at least once every 6 months

Ethical approval

The Intego procedures were approved by the KU Leuven Ethics Committee (nr. ML1723) and by the National Privacy Commission’s Sectoral Committee (decision nr. 13.026 of March 19, 2013). The procedures to collect data by Healthdata.be were approved by the Belgian Privacy Commission on April 17, 2018.

The protocol for this CRCT was approved by the Ethical Commission Research UZ/KU Leuven with number S62753 (May 13, 2019) and number S62753/0001 (March 20, 2020).

Trial registration

Registered on May 14, 2020, on ClinicalTrials.gov (number NCT04388228) https://clinicaltrials.gov/ct2/show/NCT04388228.

Informed consent

A researcher of the Intego network will be obtaining signed and written informed consent from each collaborating GP practice, and we will ensure that each GP practice is given a full explanation of the protocol in an information letter.

Data collection

The data collection takes 12 months (see Fig. 1).

The data in the EHR of the Intego practices will be collected by HealthData through eHealth. This is part of the basic Intego project. This collection will be a snapshot of the EHR, which is used daily by GPs during consultation with patients. The data will be hosted on the Healthdata.be platform, which is described in more detail elsewhere [22].

Based on this data collection, the GPs will receive an A&F intervention with feedback reports on 4 different time points (every 3 months). At these time points, we will remind the GPs to work on their registration level of specific parameters or coded diagnoses in the EHR.

Every week, we will collect the data of the same practices, which is part of the basic Intego project. Then, for this A&F project, we will evaluate every 3 months whether the extended feedback led to a significant improvement in the registration level of the intervention group compared with the control group. We will analyze only the data of the practices that signed an informed consent form.

Power calculation

To measure an improvement in our primary and key secondary outcome, these are the percentage of registered DM 2 and CKD diagnoses in the EHR, a power calculation was performed.

To measure an improvement of 10% points (from 75% to 85%) in our primary outcome with 80% power, an alpha of 0.05, a variance inflation factor (VIF) of 21.12 (based on an intra-class correlation (ICC) of 0.103), we need 56 practices to be included in this CRCT, or 28 practices in each group (Table 2). To measure an improvement of 20% points (from 50% to 70%) in our key secondary outcome with 80% power, an alpha of 0.05, a VIF of 9.1, and an ICC of 0.32, we need 68 practices to be included in this CRCT, or 34 practices in each group (Table 2). To have enough statistical power to measure an improvement in both outcomes, 68 practices will be recruited. The VIF and ICC were calculated based on historical data from the Intego database and were used to correct for clustering.

Table 2 Power calculation

A power calculation for our exploratory secondary outcomes was not performed.

Analysis

SAS software (SAS Institute Inc., Cary, NC, USA) and R (R Foundation for Statistical Computing, Vienna, Austria) will be used for analysis and for the graphs of the feedback. For this study, data will only be used on an aggregated level (per GP practice). Analysis will be performed using the numerator and denominator of each outcome per GP practice. The results will be reported according to the Consolidated Standards of Reporting Trials (CONSORT) [29]. The statistical model will not be adjusted by for any covariates.

To evaluate and explore the effect of the intervention on the primary and secondary outcome measures, a logistic generalized estimating equations (GEE) model will be used. This model is chosen because it can investigate the average response of an intervention on a population level. The effect of the intervention will be presented as the difference in proportions together with its 95% confidence interval. The GEE model will also be used to estimate the proportion of correctly registered diagnoses/parameters in the 2 allocated groups with their 95% confidence intervals.

Discussion

Overview

This cluster randomized controlled trial investigates the effect of an automated A&F intervention to primarily improve the level of registration of DM 2 and to secondarily improve the level of registration of CKD in the EHR of the GP. Furthermore, the registration of heart failure, lifestyle parameters, and biometric data in the EHR of the GP are explored together with 4 EHR-extractable quality indicators. In this way, this study aims to improve the data completeness of the EHR, paving the way for future quality assessments.

By implementing an automated A&F intervention with the help of the Intego database, this CRCT can form a basis for investigating future A&F interventions, which is particularly important for chronic diseases. Chronic diseases such as DM 2 and CKD have a high prevalence, and the quality of primary care for these conditions needs improvement [9, 10]. We already developed a set of EHR-extractable quality indicators for DM 2, CKD, and heart failure, which can be used in future A&F interventions [27, 28, 30]. In addition, large data repositories in primary care, which collect routine primary healthcare data anonymized at the source, can be used to address many research questions of interest [31]. Examples of these are the Intego database we are using, the Dutch NIVEL, and the British Royal College of General Practitioners’ Research and Surveillance Centre [21, 32, 33]. Using a large database consisting of all EHR files of the Intego practices creates an opportunity to power future studies that can measure small differences in effect between groups. Because of this, it will be possible to compare different A&F features on, for example, the content and delivery of the feedback. After all, recent research indicates the need to compare different A&F interventions with each other instead of comparing A&F interventions with control conditions [34].

The feedback features that are implemented in this CRCT are the use of benchmarks, the frequency, the evidence-based aspect, and a low cognitive load of the feedback. These features are only a few of the 313 theory-informed hypotheses that were suggested as potentially important for improving the effectiveness of A&F interventions [16]. However, in the case of primary care, there is evidence that GPs prefer brief feedback interventions and reports with comparisons and best practice guidelines [35]. With this in mind, the aforementioned features concerning the content and delivery of the feedback are chosen for this trial and combined into an action plan. Furthermore, the use of benchmarks is, according to the recent Clinical Performance Feedback Intervention Theory, useful for improving feedback because it will motivate and compare feedback recipients [36]. Because the Intego database gathers data from approximately 400 GPs, it will be possible to benchmark their results with each other and more specifically with the mean of the top 10% performers according to the Achievable Benchmarks of Care method [23]. Finally, the feedback will be given on four different occasions, since the frequency of the feedback delivery (more than once) is already known to be an important factor [14].

Strengths and limitations

One of the strengths of this study is the underlying generic approach we are implementing for improving the registration level of two chronic diseases (DM 2 and CKD) in the EHR of the GP. This generic approach is an automated A&F intervention that can be adapted for other diseases and with other features of feedback, so that facilitating factors of future interventions can be evaluated and compared. This thus opens up the possibility to step away from two-arm trials (control versus intervention) in favor of head-to-head trials, as suggested by other authors [34]. Implementing future interventions in this manner and using the Intego database as a large sample size, together with our generic approach, offers the opportunity to power studies that can measure small differences in the effectiveness of certain facilitating factors or features of A&F for improving the quality of primary care [34].

Our study also has some limitations. Since we are implementing an automated and EHR-based intervention, potential sources of bias when (re) using data gathered in the EHR need to be considered [31]. However, the generic aspect of our intervention can potentially assist in resolving some of those problems, such as the quality of the primary care EHR data. Another limitation is that if we want to apply automated quality assessment in the form of A&F for specific diseases, quality indicators that are extractable from the EHR of the GP for these diseases need to be available.

Trial status

The recruitment for the trial will start in January 2021 and is expected to be completed in June 2021. This protocol is version 4.0, dated June 7, 2020.

Trial funding

The Intego project is funded by the Flemish Agency “Zorg en Gezondheid.” This trial did not receive individual funding. There was no funding for the design of the study and for the collection, analysis, and interpretation of the data. There was also no funding for writing the manuscript.

Conclusion

This CRCT investigates the effectiveness of an automated A&F intervention in primary care to improve the data completeness of the EHR. Different features concerning the content and delivery of the feedback will be implemented and evaluated.

Availability of data and materials

Data will be available on reasonable request from the corresponding author.

Abbreviations

EHR:

Electronic health record

DM 2:

Diabetes mellitus type 2

CKD:

Chronic kidney disease

A&F:

Audit and feedback

GP:

General physician

SPIRIT:

Standard Protocol Items: Recommendations for Interventional Trials

CRCT:

Cluster randomized controlled trial

ABC:

Achievable Benchmarks of Care

eGFR:

Estimated glomerular filtration rate

ATC:

Anatomical therapeutic chemical code

VIF:

Variance inflation factor

ICC:

Intraclass correlation

CONSORT:

Consolidated Standards of Reporting Trials

GEE model (logistic):

Generalized estimating equations model

References

  1. 1.

    Campanella P, Lovato E, Marone C, Fallacara L, Mancuso A, Ricciardi W, Specchia ML. The impact of electronic health records on healthcare quality: a systematic review and meta-analysis. Eur J Pub Health. 2016;26(1):60–4. https://doi.org/10.1093/eurpub/ckv122.

    Article  Google Scholar 

  2. 2.

    Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev. 2010;67(5):503–27. https://doi.org/10.1177/1077558709359007.

    Article  PubMed  Google Scholar 

  3. 3.

    Roth CP, Lim YW, Pevnick JM, Asch SM, McGlynn EA. The challenge of measuring quality of care from the electronic health record. Am J Med Qual. 2009;24(5):385–94. https://doi.org/10.1177/1062860609336627.

    Article  PubMed  Google Scholar 

  4. 4.

    Madden JM, Lakoma MD, Rusinak D, Lu CY, Soumerai SB. Missing clinical and behavioral health data in a large electronic health record (EHR) system. J Am Med Inform Assoc. 2016;23(6):1143–9. https://doi.org/10.1093/jamia/ocw021.

    Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform. 2013;46(5):830–6. https://doi.org/10.1016/j.jbi.2013.06.010.

    Article  PubMed  Google Scholar 

  6. 6.

    Hjerpe P, Merlo J, Ohlsson H, Bengtsson Bostrom K, Lindblad U. Validity of registration of ICD codes and prescriptions in a research database in Swedish primary care: a cross-sectional study in Skaraborg primary care database. BMC Med Inform Decis Mak. 2010;10(1):23. https://doi.org/10.1186/1472-6947-10-23.

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Reimer AP, Milinovich A, Madigan EA. Data quality assessment framework to assess electronic medical record data for use in research. Int J Med Inform. 2016;90:40–7. https://doi.org/10.1016/j.ijmedinf.2016.03.006.

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    van der Bij S, Khan N, Ten Veen P, de Bakker DH, Verheij RA. Improving the quality of EHR recording in primary care: a data quality feedback tool. J Am Med Inform Assoc. 2017;24(1):81–7. https://doi.org/10.1093/jamia/ocw054.

    Article  PubMed  Google Scholar 

  9. 9.

    Van Gelder VA, Scherpbier-De Haan ND, De Grauw WJC, Vervoort GMM, Van Weel C, Biermans MCJ, Braspenning JCC, Wetzels JFM. Quality of chronic kidney disease management in primary care: a retrospective study. Scand J Prim Health Care. 2016;34(1):73–80. https://doi.org/10.3109/02813432.2015.1132885.

    Article  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Hermans MP, Brotons C, Elisaf M, Michel G, Muls E, Nobels F. Optimal type 2 diabetes mellitus management: the randomised controlled OPTIMISE benchmarking study: baseline results from six European countries. Eur J Prev Cardiol. 2013;20(6):1095–105. https://doi.org/10.1177/2047487312449414.

    Article  PubMed  Google Scholar 

  11. 11.

    Norton JM, Ali K, Jurkovitz CT, Kiryluk K, Park M, Kawamoto K, Shang N, Navaneethan SD, Narva AS, Drawz P. Development and validation of a pragmatic electronic phenotype for CKD. Clin J Am Soc Nephrol. 2019;14(9):1306–14. https://doi.org/10.2215/CJN.00360119.

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Liu C, Zowghi D, Talaei-Khoei A, Daniel J: Achieving data completeness in electronic medical records: a conceptual model and hypotheses development. University of Hawaii; 2018.

  13. 13.

    Tummalapalli SL, Peralta CA. An electronic CKD phenotype: a step forward in improving kidney care. Clin J Am Soc Nephrol. 2019;14(9):1277–9. https://doi.org/10.2215/CJN.08180719.

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:Cd000259.

    Google Scholar 

  15. 15.

    Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, Grimshaw JM. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9(1):14. https://doi.org/10.1186/1748-5908-9-14.

    Article  PubMed  PubMed Central  Google Scholar 

  16. 16.

    Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, Sales A, Brehaut JC. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017;12(1):117. https://doi.org/10.1186/s13012-017-0646-0.

    Article  PubMed  PubMed Central  Google Scholar 

  17. 17.

    Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, Ivers N, Grimshaw JM. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164(6):435–41. https://doi.org/10.7326/M15-2248.

    Article  PubMed  Google Scholar 

  18. 18.

    Colquhoun H, Michie S, Sales A, Ivers N, Grimshaw JM, Carroll K, Chalifoux M, Eva K, Brehaut J. Reporting and design elements of audit and feedback interventions: a secondary review. BMJ Qual Saf. 2017;26(1):54–60. https://doi.org/10.1136/bmjqs-2015-005004.

  19. 19.

    Van den Bulck S, Spitaels D, Vaes B, Goderis G, Hermens R, Vankrunkelsven P. The effect of electronic audits and feedback in primary care and factors that contribute to their effectiveness: a systematic review. Int J Qual Health Care. 2020;32(10):708–20. https://doi.org/10.1093/intqhc/mzaa128.

  20. 20.

    Chan A-W, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, Hróbjartsson A, Mann H, Dickersin K, Berlin JA, et al. SPIRIT 2013 Statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158:200–7.

    Article  Google Scholar 

  21. 21.

    Truyers C, Goderis G, Dewitte H, Akker M, Buntinx F. The Intego database: background, methods and basic results of a Flemish general practice-based continuous morbidity registration project. BMC Med Inform Decis Mak. 2014;14(1):48. https://doi.org/10.1186/1472-6947-14-48.

    Article  PubMed  PubMed Central  Google Scholar 

  22. 22.

    Delvaux N, Aertgeerts B, van Bussel JCH, Goderis G, Vaes B, Vermandere M. Health data for research through a Nationwide privacy-proof system in Belgium: design and implementation. JMIR Med Inform. 2018;6(4):e11428. https://doi.org/10.2196/11428.

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23.

    Weissman NW, Allison JJ, Kiefe CI, Farmer RM, Weaver MT, Williams OD, Child IG, Pemberton JH, Brown KC, Baker CS. Achievable benchmarks of care: the ABCs of benchmarking. J Eval Clin Pract. 1999;5(3):269–81. https://doi.org/10.1046/j.1365-2753.1999.00203.x.

    CAS  Article  PubMed  Google Scholar 

  24. 24.

    Moulton LH. Covariate-based constrained randomization of group-randomized trials. Clin Trials. 2004;1(3):297–305. https://doi.org/10.1191/1740774504cn024oa.

    Article  PubMed  Google Scholar 

  25. 25.

    Greene EJ. A SAS macro for covariate-constrained randomization of general cluster-randomized and unstratified designs. J Stat Softw. 2017;77(Code Snippet 1). https://doi.org/10.18637/jss.v077.c01.

  26. 26.

    Altman DG, Bland JM. Treatment allocation by minimisation. Bmj. 2005;330(7495):843. https://doi.org/10.1136/bmj.330.7495.843.

    Article  PubMed  PubMed Central  Google Scholar 

  27. 27.

    Van den Bulck SA, Vankrunkelsven P, Goderis G, Broekx L, Dreesen K, Ruijten L, Mpoukouvalas D, Hermens R. Development of quality indicators for type 2 diabetes, extractable from the electronic health record of the general physician. A rand-modified Delphi method. Primary Care Diabetes. 2019.

  28. 28.

    Van den Bulck SA, Vankrunkelsven P, Goderis G, Van Pottelbergh G, Swerts J, Panis K, Hermens R. Developing quality indicators for chronic kidney disease in primary care, extractable from the electronic medical record. A Rand-modified Delphi method. BMC Nephrol. 2020;21:161.

    Article  Google Scholar 

  29. 29.

    Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340(mar23 1):c332. https://doi.org/10.1136/bmj.c332.

    Article  PubMed  PubMed Central  Google Scholar 

  30. 30.

    Smets M, Smeets M, Van den Bulck S, Janssens S, Aertgeerts B, Vaes B. Defining quality indicators for heart failure in general practice. Acta Cardiol. 2019;74(4):291–8. https://doi.org/10.1080/00015385.2018.1492659.

  31. 31.

    Verheij AR, Curcin V, Delaney CB, McGilchrist MM. Possible sources of Bias in primary care electronic health record data use and reuse. J Med Internet Res. 2018;20(5):e185. https://doi.org/10.2196/jmir.9134.

    Article  PubMed  PubMed Central  Google Scholar 

  32. 32.

    Schweikardt C, Verheij RA, Donker GA, Coppieters Y. The historical development of the Dutch Sentinel General Practice Network from a paper-based into a digital primary care monitoring system. J Public Health. 2016;24(6):545–62. https://doi.org/10.1007/s10389-016-0753-4.

    Article  Google Scholar 

  33. 33.

    de Lusignan S, Correa A, Smith GE, Yonova I, Pebody R, Ferreira F, Elliot AJ, Fleming D. RCGP Research and Surveillance Centre: 50 years’ surveillance of influenza, infections, and respiratory conditions. Br J Gen Pract. 2017;67(663):440–1. https://doi.org/10.3399/bjgp17X692645.

    Article  PubMed  PubMed Central  Google Scholar 

  34. 34.

    Grimshaw JM, Ivers N, Linklater S, Foy R, Francis JJ, Gude WT, Hysong SJ. Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback. BMJ Qual Saf. 2019;28:416.

    CAS  Article  Google Scholar 

  35. 35.

    Sebo P, Maisonneuve H, Fournier JP, Senn N, Haller DM. General practitioners' views and preferences about quality improvement feedback in preventive care: a cross-sectional study in Switzerland and France. Implement Sci. 2017;12(1):95. https://doi.org/10.1186/s13012-017-0623-7.

    Article  PubMed  PubMed Central  Google Scholar 

  36. 36.

    Brown B, Gude WT, Blakeman T, van der Veer SN, Ivers N, Francis JJ, Lorencatto F, Presseau J, Peek N, Daker-White G. Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019;14(1):40. https://doi.org/10.1186/s13012-019-0883-5.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Dr. Steffen Fieuws for his help with the power calculation.

Funding

The Intego project is funded by the Flemish Agency “Zorg en Gezondheid.” This trial did not receive individual funding. There was no funding for the design of the study and for the collection, analysis, and interpretation of the data. There was also no funding for writing the manuscript.

Author information

Affiliations

Authors

Contributions

SVDB, TDB, and BV designed and conceptualized the study. SVDB and TDB drafted the manuscript. PM provided statistical advice. SVDB, TDB, BV, RH, PV, GG, PC, WR, and GVP reviewed and revised the manuscript. SVDB, TDB, and BV are the guarantors of this work. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Steve Van den Bulck.

Ethics declarations

Ethics approval and consent to participate

The Intego procedures were approved by the KU Leuven Ethics Committee (nr. ML1723) and by the National Privacy Commission’s Sectoral Committee (decision nr. 13.026 of March 19, 2013). The procedures to collect data by Healthdata.be were approved by the Belgian Privacy Commission on April 17, 2018.

The protocol for this CRCT was approved by the Ethical Commission Research UZ/KU Leuven with number S62753 (May 13, 2019) and number S62753/0001 (March 20, 2020).

A researcher of the Intego network will be obtaining signed and written informed consent from each collaborating GP practice, and we will ensure that each GP practice is given a full explanation of the protocol in an information letter.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Steve Van den Bulck and Tine De Burghgraeve shared first authorship.

Supplementary Information

Additional file 1.

SPIRIT checklist.

Additional file 2.

Ethical approval document.

Additional file 3.

Informed consent (Dutch).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Van den Bulck, S., De Burghgraeve, T., Raat, W. et al. The effect of automated audit and feedback on data completeness in the electronic health record of the general physician: protocol for a cluster randomized controlled trial. Trials 22, 325 (2021). https://doi.org/10.1186/s13063-021-05259-9

Download citation