- Open Access
- Open Peer Review
Understanding the impact of a multispecialty electronic consultation service on family physician referral rates to specialists: a randomized controlled trial using health administrative data
Trials volume 20, Article number: 348 (2019)
Electronic consultation (eConsult) services are secure online applications facilitating provider-to-provider communication. They have been found to improve access to specialist care. However, little is known about eConsult’s impact on family physicians’ referral rates to specialty care. The objective of this study was to assess the impact of a multispecialty eConsult service on referral rates from primary care.
In this parallel-arm, randomized controlled trial, we recruited primary care providers across Ontario not previously enrolled with eConsult. We randomly assigned participants to intervention and control arms. Participants in the intervention arm received access to eConsult for a period of 1 year while those in the control arm received no access to eConsult. The main outcome was specialist referral rate, expressed as the total number of referrals to (1) specialties available through eConsult, and (2) all medical specialties, per 100 patients seen. Multivariable negative binomial regression analysis was used to evaluate the effect of the intervention before and after adjusting for provider characteristics, using health administrative data.
One hundred and thirteen participants were randomized (56 to control and 57 to intervention). For the primary outcome (referrals to eConsult specialties), the results show a statistically significant reduction in the number of referrals in both arms (control-arm Rate Ratio (RR), 0.85, 95% CI 0.79 to 0.91; intervention-arm RR, 0.80, 95% CI 0.74 to 0.85; unadjusted and adjusted RR values almost identical), as compared to the baseline data collected during the 12-month period before randomization, with a non-statistically significant 6% greater reduction in referrals in the intervention arm, compared to the control arm (unadjusted RR 0.94, 95% CI 0.85 to 1.03; adjusted RR 0.93, 95% CI 0.85 to 1.03).
Our randomized controlled trial of a multispecialty eConsult service demonstrated inconclusive results in terms of the impact of eConsult on physician referral rates. Findings are discussed in light of important limitations associated with conducting randomized controlled trials (RCTs) of complex interventions in the primary care context with intent to inform the design and analysis of future trials.
Clinicaltrials.gov, ID: NCT02053467. Registered prospectively on 3 February 2014.
Excessive wait times for specialty care are a problem facing many countries, including Canada. The 2016 Commonwealth Fund Survey reported that Canada continues to perform below the international average for timely access to patient care, with Canadians in all provinces reporting the longest wait times for specialists among the 11 countries studied . More than half of Canadians (56%) waited longer than 4 weeks to see a specialist compared with the international average of 36% . Excessive wait times can cause patients anxiety and stress, delay diagnosis of their condition, necessitate repeating the tests, and potentially cause further deterioration of their conditions , leading to higher costs and reduced satisfaction for patients and providers [2,3,4,5].
In response, innovative solutions for delivering care that are not based on face-to-face visits, such as eConsult, are gaining momentum. In 2009 our team began developing, implementing, and evaluating an innovative eHealth solution called the Champlain BASE™ (Building Access to Specialists through eConsultation) eConsult service . eConsult is a form of asynchronous communication whereby primary care providers (PCP) and specialists can communicate directly about a patient through a secure web-based application. PCPs can submit a patient question (usually for a patient who would otherwise have been referred) to one of over 100 specialty services via a web-based portal. They can attach any additional information (e.g., photos, test results, Electronic Medical Record-generated letter). The case is assigned to a specialist, who receives an email notification prompting them to access the case via the secure site. Specialists are expected to provide an answer within 1 week. They can reply to the question, request additional information, or recommend a referral, and advise the PCP on other matters such as medication changes, additional tests, or other critical actions to be completed before the face-to-face specialty care appointment. PCPs ultimately decide how to apply the specialists’ suggestion and when the case can be closed. Specialists are compensated on a pro-rated hourly basis [6, 7].
The eConsult service’s effectiveness at improving access, high levels of patient and provider satisfaction, and ability to lower costs for care have been described in previous studies [6,7,8,9]. However, several knowledge gaps remain. Our recent systematic review summarizes 36 studies of eConsult services worldwide, of which half were set in the United States and 78% focused on a single specialty (commonly, dermatology) . Study quality was average, with no randomized controlled trials (RCTs) conducted on multispecialty services or in a Canadian setting where access to physician services is universally funded. A systematic review conducted by Vimalananda et al. found similar results, with two thirds of all included studies emerging from the same three eConsult services . Both reviews cite quick response times, avoided referrals for face-to-face visits, and high levels of provider satisfaction while noting an absence of data on clinical outcomes, cost-effectiveness and impact on referral rates for face-to-face visits [10, 11]. These findings, while promising, underscored the need for a RCT to examine the impact of a multispecialty eConsult service.
The objective of this study was to assess the impact of the Champlain BASE™ eConsult service on referral rates using an RCT of eConsult versus usual referral practices. We hypothesized that access to eConsult would lead to a decrease in referral requests for face-to-face specialist visits from PCPs in the intervention group.
This was a parallel-arm RCT that recruited family physicians in Ontario to use the Champlain BASE™ eConsult service. (Fig. 1) Physicians were randomized 1:1 to either the intervention or control arm between 31 January 2014 and 26 September 2014. Although originally intended as a stepped-wedge RCT, the design was changed to a parallel-arm before and after RCT due to the fact that randomization could not be performed at a discrete time point for all participants.
The full details of the eConsult service have been discussed elsewhere [6,7,8]. In brief, after undergoing registration, which includes orientation and brief training on the use of the service, users are able to submit patient-specific clinical questions to specialists through a secure, web-based application. Specialists are asked to respond within 7 days and, for each eConsult, are able to: (1) provide a recommendation, (2) request more information, or (3) recommend a face-to-face referral. The communication between PCPs and specialists is iterative and the discussion can occur back and forth until, ultimately, the PCP closes the case. Physicians randomized to the intervention group received access to eConsult right away (pending completion of an orientation session), while those randomized to the control group used standard referral practices for 1 year after randomization and then were given the option to use eConsult in the second year. The intervention period for the treatment group lasted 1 year following the enrollment/randomization period which began on 31 January 2014 and ended on 26 September (see “Randomization” section for details). As it took an average of 3 months from the randomization date for the intervention group physicians to complete the mandatory eConsult orientation and brief training, the post-randomization outcome was assessed for the following 9 months rather than for the full 12 months post randomization. The pre-randomization period was 12 months in duration for both groups. These timelines are depicted in Fig. 2.
All Ontario family physicians who were not already registered for the eConsult service were eligible to participate.
This study was conducted in Ontario, Canada, the country’s most populous province, with publicly funded, universal access to physician services to its over 13 million residents .
Every eligible family physician in the province of Ontario (not already using eConsult service) was sent a two-page information brochure outlining the eConsult service, along with a letter describing the research study between January 2014 and April 2014. Recruitment was conducted by a third party (Ontario Medical Association); the eConsult research team did not have access to the mail-out list. Family physicians interested in participating in the study were asked to contact the study team for more information. Those expressing interest were sent recruitment packages which included a study information sheet, a consent form, and a practice survey. Three weeks later, a follow-up letter and recruitment package was sent again by fax/email to the non-responders. Recruitment continued until the target sample (minimum n = 50 physicians/group) was reached.
Individual physicians were the units of randomization. A computer-generated random list of numbers was created using an online website (https://www.randomizer.org/) to allocate participants to either the control or the intervention arm. The allocation sequence was implemented by a research staff member not involved in this research project and concealed from the researchers and staff responsible for enrolling and assessing participants in sequentially numbered, opaque, sealed, and stapled envelopes. Due to the nature of the intervention, participating physicians were not blinded to their allocation status. No stratification was used since we recruited individual physicians (rather than practices) into the trial.
Our primary outcome was specialist referral rate defined as the total number of referrals for face-to-face visits to all medical specialties available through eConsult service during the study period (see Appendix) per 100 unique patients (not encounters) seen. The denominator “patients seen” included patients who were seen at least once during the assessment period (baseline or post intervention). The secondary outcome was referral rate to all medical specialties. Only referrals for face-to-face specialist visits initiated by the physicians in the trial were counted. If a patient was referred to the same specialist multiple times during our study period, this was counted as a single referral. If the patient was referred to two physicians of the same specialty in our time period, this was counted as two referrals. Referrals to other family physicians were excluded.
Sample size calculation
A total sample size of 80,000 patients seen (50 providers in each arm with an average of 800 patients per provider), was calculated to achieve 80% power to detect an absolute reduction in referrals of 6 per 100 patients from a control-arm referral rate of 31 per 100 patients, estimated from a previous study , using a two-sided significance level of 5% and assuming a between-provider coefficient of variation of 0.38 .
We used the following databases from the Institute for Clinical Evaluative Sciences (ICES, ices.on.ca) to obtain physician and patient characteristics: Registered Persons Database, containing demographic data for all residents eligible for provincial health care; Ontario Health Insurance Program (OHIP) billing claims system capturing approximately 95% of physician services in Ontario; the Client Agency Program Enrollment Registry and Corporate Provider Database, for patient enrollment with individual primary care physicians; and the ICES Physician Database, containing physician demographic information, training, and practice setting. 2006 Statistics Canada Census data was used to assign income quintile to patients based on their postal code. These data sets were linked using unique encoded identifiers and analyzed at ICES.
We also used the Champlain BASE™ eConsult utilization data, which was routinely collected throughout the study period. The data was used to identify the physicians in each group (treatment and control) as well as the date of randomization which was used to identify the relevant 12-month pre-intervention period and the 9-month post-intervention period, as per Fig. 2. Randomization took place between 31 January 2014 and 26 September 2014. The data was transferred to ICES and linked to the administrative data using the encrypted College of Physicians and Surgeons of Ontario license number for each physician. All analyses were carried out at ICES. The study was registered at Clinicaltrials.gov (NCT02053467).
Creation of patient rosters
We used the following process to construct patient rosters for each physician in the study. Patients officially rostered to each physician in the trial were identified using the Client Agency Program Enrollment Database. For patients who were not rostered to a physician, they were “virtually” rostered through the standard approach based on the physician who had billed the largest dollar amount over a 2-year period. Both physician and practice characteristics were obtained from the ICES Physician Database. Patient characteristics, including age, sex, and residential postal code, were obtained from the Registered Persons Database. Additionally, census data was used to assign patients to both income quintile (with lowest = 1, highest = 5) and rurality category based on the Rurality Index of Ontario scores, and categorized as urban (0–10), non-urban (10–39), and rural (40+). We assigned comorbidity using the Johns Hopkins Adjusted Clinical Groups Case-Mix Assignment software (The Johns Hopkins ACG® System Version 10) by assigning from 0 up to 32 distinct Aggregated Diagnosis Groups (ADGs), with a higher number of ADGs reflecting a higher level of diagnosed comorbidity .
All analyses estimate intention to treat (ITT) effects. Physicians were excluded from the analysis only if (1) they had no OHIP billings during the study period, and/or (2) they had less than 100 patients rostered to them on the randomization date. Both cases preclude examination of their referral rates due to lack of data.
Descriptive statistics were generated to describe patient and physician characteristics at the time of randomization for the two groups: eConsult and control. Provider characteristics were described using frequency and percent for categorical variables, or mean and standard deviation for continuous variables. Patient characteristics were aggregated to the level of the provider and described using mean and standard deviation.
The primary outcome, referral rate per 100 patients seen, was analyzed using a repeated measures multivariable random effects negative binomial regression model. The unit of analysis was the provider. The log link function was used and the natural log of the number of patients seen in each study period was included as an offset term. Intervention, study period, and the interaction between intervention and study period were specified as fixed covariates. The correlation in repeated measures on the same physician was accounted for by specifying a compound symmetric covariance matrix. The nesting of multiple providers in the same practice was additionally accounted for by specifying a random intercept for the practice. To account for underlying secular/time trends in the rate of referrals over the study period, the effect of the intervention was expressed as the between-arm difference in the change in referral rates from pre to post intervention. By exponentiating the regression coefficient for the eConsult by study year interaction term, we obtained this estimate as a Rate Ratio (RR) together with 95% confidence interval (CI). Least square mean estimates of the model-based referral rates in each arm in the pre- and post-intervention periods were obtained from the models. The models were adjusted for physician characteristics defined after inspecting baseline differences between the groups, but before implementing the primary outcome analysis: years since graduation, location of medical training, and practice model (Capitation: Interdisciplinary (Family Health Team (FHT))/Capitation: Non-interdisciplinary (not FHT)/Reformed Fee for service, Traditional Fee for service, other).
Because it is known that an analysis of change from baseline can be overly sensitive to baseline differences between the arms , we also conducted a secondary sensitivity analysis of covariance (ANCOVA) of the primary outcome post intervention by specifying the log of the provider’s baseline referral rate as a covariate. Results of these analyses were expressed as Rate Ratios (intervention versus control-arm referral rates), adjusted for the baseline referral rate.
SAS Version 9.4 (SAS Institute, Cary, NC, USA) was used for all analyses.
All eligible family physicians (12,723) in Ontario were invited, of whom 113 consented to participate in the eConsult trial and were randomized to the intervention (n = 57) and control groups (n = 56). However, gaps in administrative data caused some physician to be partially or fully excluded, and consequently among intervention group physicians only 49 physicians were analyzed pre intervention and 51 post intervention; in the control group, 48 physicians were analyzed pre intervention and 49 post intervention. The flow of participants through the trial is presented in Fig. 1.
Table 1 presents provider and patient characteristics for the intervention and control groups at the time of randomization. Patient panels were comparable between the groups. At the provider level, it can be seen that a higher proportion of intervention (eConsult) group physicians were practicing in the Interdisciplinary (FHT) model and in group practices with two to five physicians. eConsult physicians had a median of 20 years since graduation compared to control with 23 years.
Participants completed 44,066 referrals across all medical specialties during the study period: 22,079 by the eConsult physicians and 21,987 by the control physicians. The eConsult physicians and control physicians referred most frequently to the same five specialties before and after intervention: dermatology, general surgery, obstetrics and gynecology, gastroenterology, otolaryngology (see Table 2 for detailed breakdown).
Table 3 presents the observed mean and standard deviation of numbers of patients seen, as well as numbers of referrals to eConsult specialties and to all specialties during the baseline and post-randomization assessment periods. It also presents the observed referral rates per 100 patients seen for eConsult and for all specialties including those not available via eConsult (see “Appendix”). There were differences between the groups pre intervention: the eConsult group had slightly lower mean number of total referrals than the control group for the specialties available via eConsult (244.00, standard deviation (SD) = 142.88 versus 246.10, SD = 161.97) and all specialties (261.86, SD = 151.88 versus 264.48, SD = 172.95 respectively), but higher mean number of referrals per 100 patients seen for specialties available via eConsult (32.83, SD = 12.40 versus 28.15, SD = 10.89) and all specialties (35.24, SD = 13.20 versus 30.20, SD = 11.60). Post intervention, both groups showed decreases in their mean number of referrals and mean referral rates per 100 patients.
Table 4 shows the multivariable negative binomial regression analyses of the primary and secondary outcomes. Between-arm differences in the pre- and post-intervention periods, as well as differences in change from the pre-intervention arm, are presented as unadjusted and adjusted RRs together with 95% confidence intervals (CI). For the primary outcome (referrals to eConsult specialties), the results show a non-statistically significant 6% greater reduction in referrals in the intervention arm, relative to the control arm (unadjusted RR 0.94, 95% CI 0.85 to 1.03) and a non-statistically significant 7% relative reduction after adjusting for covariates (adjusted RR 0.93, 95% CI 0.85 to 1.03). The differences between the arms were statistically significant only at baseline (p = 0.03). While there was a statistically significant decrease in referral rates from pre to post intervention in the control arm (unadjusted and adjusted RRs 0.85, 95% CIs 0.79 to 0.91) there was also a decrease in referral rates in the intervention arm (unadjusted RR 0.80, adjusted RR 0.79, 95% CIs 0.79 to 0.91). As the change in referral behavior over time in the control arm cannot be the result of eConsult, this suggests an underlying decreasing secular trend in referrals.
Results were similar for the secondary outcome (referrals to all specialties). Overall, adjusting for provider characteristics did not make much difference in the rate ratios, and 95% confidence intervals barely crossed the value of 1.
Model-based unadjusted and adjusted referral rates, together with 95% CIs, are presented in Table 5. Referrals to specialties available via eConsult decreased in both arms: from 0.31 (95% CI 0.28–0.35) to 0.25 (95% CI 0.22–0.27) in the intervention arm versus 0.27 (95% CI 0.25–0.30) to 0.23 (95% CI 0.21–0.26) in the control arm, unadjusted. Adjusting for provider characteristics did not affect this finding, and similar referral rates patterns were obtained for referrals to all specialties (not restricted to those available via eConsult).
The results from the secondary ANCOVA analysis for the primary (referrals to eConsult specialties) and secondary (referrals to all specialties) outcomes are presented in Table 6 as Rate Ratios (RR) together with 95% CIs and show no statistically significant difference between the groups. In the “unadjusted,” ANCOVA analysis, only the provider’s pre-intervention referral rates were used as a covariate, while in the “adjusted,” version, all the remaining baseline covariates were included as well.
This RCT examined the effect of using a multispecialty eConsult service on referrals from primary to specialty care. We hypothesized that eConsult would lead to a decrease in the overall number of referrals in the intervention arm. Our analysis showed a statistically significant reduction in the number of referrals in both arms, with a non-significant 6% greater reduction in the eConsult intervention arm compared to the control arm after accounting for provider characteristics when analyzed using a difference in difference methodology. Given the significant imbalance in referral rates between the groups observed at baseline, a secondary sensitivity analysis using ANCOVA was conducted and revealed that the post-intervention difference between the groups after accounting for pre-intervention values was not statistically significant.
As we reflect on the implementation of the trial, there are several limitations and challenges associated with conducting an RCT of a complex intervention such as eConsult in a primary care context that have to be acknowledged. RCTs may minimize rather than eliminate bias . The participants in our study had to manifest their interest to be considered for enrollment and our final sample represented less than 1% of the total population invited to participate in the trial. Our groups were significantly imbalanced at baseline despite randomization, introducing a chance bias into the study. This is significant given the tremendous documented variability in physician referral rates and the methodological challenges of studying referral patterns in primary care, discussed in our previous paper . As discussed in depth by Fives et al. (2013), random allocation does not ensure baseline equality . Its role is to protect against selection bias, as allocation is not based on any systematically biased method. However, as was the case in our study, a statistically significant baseline differences can still occur. One way to deal with baseline imbalance would be to attempt to reduce the likelihood of it occurring by modifying the random allocation process by using “stratified” or “blocked” randomization. We recommend doing so in the future studies, though it has been argued that allocation of participants in a stratified study is still random, hence does not guarantee equivalence on relevant variables. RCTs often suffer from two major complications: non-compliance and missing outcomes, to which ITT analysis, which we employed, offers a potential solution . With respect to non-compliance, even though in an ideal scenario every participant enrolled in RCT would complete their allocated treatment, nine out of 57 physicians (16%) in the eConsult intervention arm did not complete the eConsult orientation and hence were not exposed to the intervention. Furthermore, the eConsult group completed a total of 413 eConsults (or 8.6 per each “exposed” physician) by the end of the study period (data not shown). This begs two questions. First, did our intervention group receive enough of a “dose” of the eConsult to impact their referral rates? Second, was our study time frame too short to detect the impact of the eConsult intervention? The answers to both questions are unknown as these ideas have not yet been a topic of empirical study. With respect to missing outcomes, due to limitations related to the use of administrative health databases (e.g., unavailability of health administrative data for certain physicians in certain study years), we did not end up with 50 providers per arm resulting in an underpowered trial. This may have been affected by the fact that administrative data: (1) are not complete since only 95% of outpatient physician encounters are recorded ; (2) contain no referral data for salaried primary care physicians working in community health centres; and (3) rely on imperfect algorithms to create patient rosters.
While the results of our study did not identify a statistically significant impact on PCP referral patterns, a growing body of international literature on telemedicine systems from various countries including United States (eConsult and eReferral), Brazil (Telehealth Network of Minas Gerais), and the international non-governmental organization Médecins Sans Frontières show that 19 to 68% of eConsult requests are resolved without the need for an in-person specialty visit [10, 11, 22]. Given these findings, it is reasonable to assume that frequent use of eConsult has the potential to lead to a lower overall number of referrals. Afterall, even though non-statistically significant, a 6% greater reduction in referrals in the intervention arm, relative to the control arm, can be clinically significant, especially on a population level, and is worth mentioning in the context of health services delivery improvement efforts. This potential effect on reducing face-to-face visits can be attributed, in part, to the fact that the rapid access to specialist expertise afforded through eConsult supports the patient and the PCP in not only continuing management of their current patient (thus supporting a patient medical home approach and improved continuity of care), but also by offering educational value through case-based learning, thus increasing the provider’s repertoire of expertise potentially impacting the need to refer for future patients [23,24,25]. Unfortunately, the effects of eConsult on primary care physician referral behavior have not been previously studied utilizing rigorous methodology. There have been only a few randomized trials of eConsult limited to a single specialty and none assessed impact on overall referral rates [26,27,28]. To our knowledge, the present trial is unique in that it is the first RCT examining the effect of a multispecialty eConsult model on physician behavior, with the main outcome of referral rates. Unfortunately, our results were inconclusive and given a small sample size, a possibility of type 2 error (or a false negative finding) must be acknowledged. Future trials examining this impact of eConsult on referral rate should be conducted and include a larger sample size to protect against selection bias, stratification at the level of the organization (e.g., primary care clinic), employing specific strategies aimed to optimize implementation of the intervention (e.g., a run-in period where those in the intervention group commit to submitting at least one eConsult to maximize compliance) , and finally a longer trial period. In addition, it would be interesting to examine if the use of eConsult services shortens the time to see specialists when a face-to-face referral is recommended from the eConsult communications.
eConsult remains an innovative solution that has the potential to transform the primary–specialty care interface by enabling a move away from reliance on face-to-face specialty visit-based care, addressing the documented fragmentation and care-coordination challenges, reducing the number of specialty care visits for conditions that can be managed by PCPs, increasing the effectiveness of face-to-face specialty visits when they occur, and producing cost savings [10, 11]. It is a complex intervention to study in a rigorous manner, and the evidence shows that many complex interventions that show promising effects in smaller studies (usually observational and uncontrolled) are not replicated in RCTs . While our results were not statistically significant, our trial provides important methodological insight to guide others seeking to implement and evaluate electronic consultation services.
This is the first RCT of a multispecialty eConsult service which aimed to demonstrate that the use of eConsult may be associated with fewer referrals from primary to specialist care. Due to various challenges associated with conducting a randomized controlled trial of a complex intervention in the primary care context, our conclusions are limited. Nonetheless, the results and reflections on the limitations encountered should prove invaluable in informing the planning of future trials to examine the effects of eConsult on physician referral behavior and the resulting population-level impacts. Further research is prudent to demonstrate that any reductions in the rate of referral are appropriate and of comparable quality to in-person consultations as well as to examine cost effectiveness to our single-payer system.
Availability of data and materials
The data set from this study is held securely in coded form at the Institute for Clinical Evaluative Sciences (ICES). While data-sharing agreements prohibit ICES from making the data set publicly available, access may be granted to those who meet pre-specified criteria for confidential access, available at www.ices.on.ca/DAS.
Aggregated Diagnosis Group
Building – Access to Specialists through eConsultation
Family Health Team
Institute of Clinical Evaluative Sciences
Intention to treat
Ontario Health Insurance Program
Primary care provider
Randomized Controlled Trial
Canadian Institute for Health Information. How Canada compares: results from the Commonwealth Fund’s 2016 international health policy survey of adults in 11 countries. Ottawa: CIHI; 2017. https://www.cihi.ca/sites/default/files/document/text-alternative-version-2016-cmwf-en-web.pdf. Accessed 30 Apr 2018
Canadian Intitute for Health Information. Health care in Canada, 2012: a focus on wait times. Ottawa: CIHI; 2012.
Barua B, Esmail N. Waiting your turn: wait times for health care in Canada. Vancouver: Fraser Institute; 2013. https://www.fraserinstitute.org/sites/default/files/waiting-your-turn-2013.pdf. Accessed 16 Mar 2019
Chen AH, Yee HF. Improving the primary care to specialty care interface: getting from here to there. Arch Intern Med. 2009;169(11):1024–6.
Barua B, Esmail N, Jackson T. The effect of wait times on mortality in Canada. Vancouver: Fraser Institute; 2014. https://www.fraserinstitute.org/sites/default/files/effect-of-wait-times-on-mortality-in-canada.pdf. Accessed 16 Mar 2019
Liddy C, Rowan MS, Afkham A, Maranger J, Keely E. Building access to specialist care through e-consultation. Open Med. 2013;7(1):e1–8.
Liddy C, Maranger J, Afkham A, Keely E. Ten steps to establishing an e-consultation service to improve access to specialist care. Telemed J E Health. 2013;19(12):982–90.
Keely E, Liddy C, Afkham A. Utilization, benefits, and impact of an e-consultation service across diverse specialties and primary care providers. Telemed J E Health. 2013;19(10):733–8.
Liddy C, Drosinis P, Deri Armstrong C, McKellips F, Afkham A, Keely E. What are the cost savings associated with providing access to specialist care through the Champlain BASE eConsult service? A costing evaluation. BMJ Open. 2016;6(6):e010920.
Liddy C, Drosinis P, Keely E. Electronic consultation systems: worldwide prevalence and their impact on patient care—a systematic review. Fam Pract. 2016;33(3):274–85.
Vimalananda VG, Gupte G, Seraj SM, Orlander J, Berlowitz D, Fincke BG, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015;21(6):323–30.
Statistics Canada. Population by year, by province and territory, CANSIM, table 051–001. http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/demo02a-eng.htm. Accessed 30 Apr 2018.
Alexander G. eReferral strategy white paper: clearing the communications fog. Ottawa: Champlain LHIN; 2011.
Franks P, Williams GC, Zwanziger J, Mooney C, Sorbero M. Why do physicians vary so widely in their referral rates? J Gen Intern Med. 2000;15(3):163–8.
Johns Hopkins University. The Johns Hopkins ACG System. https://www.johnshopkinssolutions.com/solution/acgsystem/. Accessed 30 Apr 2018.
Hooper R, Forbes A, Hemming K, Takeda A, Beresford L. Analysis of cluster randomised trials with an assessment of outcome at baseline. BMJ. 2018;360:k1121.
Kaptchuk TJ. The double-blind, randomized, placebo-controlled trial: gold standard or golden calf? J Clin Epidemiol. 2001;54(6):541–9.
Liddy C, Moroz I, Keely E, Taljaard M, Mark Fraser A, Deri Armstrong C, et al. The use of eConsult is associated with lower specialist referral rates: a cross sectional study using population-based health administrative data. Fam Pract. 2017;35(6):698–705.
Fives A. The role of random allocation in randomized controlled trials: distinguishing selection bias from baseline imbalance. J MultiDiscip Eval. 2013;9(20):33–42.
Gupta SK. Intention-to-treat concept: a review. Perspect Clin Res. 2011;2(3):109.
Chan BT, Schultz SE. Supply and utilization of general practitioner and family physician services in Ontario. Toronto: Institute for Clinical Evaluative Sciences; 2005.
Soriano Marcolino M, Minelli Figueira R, Pereira Afonso dos Santos J, Silva Cardoso C, Luiz Ribeiro A, Alkmim MB. The experience of a sustainable large scale Brazilian telehealth network. Telemed J E Health. 2016;22(11):899–908.
Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact and satisfaction with a new eConsult service: a mixed methods study of primary care providers. J Am Board Fam Med. 2015;28(3):394–403.
Barnett ML, Yee HF, Mehrotra A, Giboney P. Los Angeles Safety-Net Program eConsult system was rapidly adopted and decreased wait times to see specialists. Health Aff. 2017;36(3):492–9.
Kirsh SR, Ho PM, Aron DC. Providing specialty consultant expertise to primary care: an expanding spectrum of modalities. Mayo Clin Proc. 2014;89(10):1416–26.
Whited JD, Hall RP, Foy ME, Marbrey LE, Grambow SC, Dudley TK, et al. Patient and clinician satisfaction with a store-and-forward teledermatology consult system. Telemed J E Health. 2004;10(4):422–31.
Olayiwola JN, Anderson D, Jepeal N, Aseltine R, Pickett C, Yan J, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Ann Fam Med. 2016;14(2):133–40.
Golberstein E, Kolvenbach S, Carruthers H, Druss B, Goering P. Effects of electronic psychiatric consultations on primary care provider perceptions of mental health care: survey results from a randomized evaluation. Healthcare. 2017; [In press].
Levati S, Campbell P, Frost R, Dougall N, Wells M, Donaldson C, et al. Optimisation of complex health interventions prior to a randomised controlled trial: a scoping review of strategies used. Pilot Feasibility Stud. 2016;2(1):17.
Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to improve health care. BMJ. 2007;334(7591):455.
The authors wish to thank Glenys Smith and Susan Shurgold for their contributions to data analysis, and Justin Joschko for his assistance in preparing the manuscript for publication.
This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC), and also received funding from the Health Services Research Fund, Ministry Grant #06547, Province of Ontario, Primary Health Care Program (INSPRE-PHC). The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.
Ethics approval and consent to participate
Ethical approval was obtained from the Ottawa Hospital Research Ethics Board (Protocol #20130674-01H) and the Institutional Review Board at Sunnybrook Health Sciences Centre Toronto, Canada. Informed consent was obtained from all participants.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
List of eConsult specialties available at the beginning of the study period
Paediatric Clinical Immunology
Paediatric Infectious Diseases
Obstetrics and Gynaecology