We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.
Establishing core outcome sets for phenylketonuria (PKU) and medium-chain Acyl-CoA dehydrogenase (MCAD) deficiency in children: study protocol for systematic reviews and Delphi surveys
Trialsvolume 18, Article number: 603 (2017)
Inherited metabolic diseases (IMD) are a large group of rare single-gene disorders that are typically diagnosed early in life. There are important evidence gaps related to the comparative effectiveness of therapies for IMD, which are in part due to challenges in conducting randomized controlled trials (RCTs) for rare diseases. Registry-based RCTs present a unique opportunity to address these challenges provided the registries implement standardized collection of outcomes that are important to patients and their caregivers and to clinical providers and healthcare systems. Currently there is no core outcome set (COS) for studies evaluating interventions for paediatric IMD. This protocol outlines a study that will establish COS for each of two relatively common IMD in children, phenylketonuria (PKU) and medium-chain acyl-CoA dehydrogenase (MCAD) deficiency.
This two-part study is registered with the Core Outcome Measures in Effectiveness Trials (COMET) initiative. Part 1 includes a rapid review and development of an evidence map to identify a comprehensive listing of outcomes reported in past studies of PKU and MCAD deficiency. The review follows established methods for knowledge synthesis, including a comprehensive search strategy, two stages of screening citations against inclusion/exclusion criteria by two reviewers working independently, and extraction of important data elements from eligible studies, including details of the outcomes collected and outcome measurement instruments. The review findings will inform part 2 of our study, a set of Delphi surveys to establish consensus on the highest priority outcomes for each condition. Healthcare providers, families of children with PKU or MCAD deficiency, and health system decision-makers will be invited to participate in two to three rounds of Delphi surveys. The design of the surveys will involve parents of children with IMD who are part of a family advisory forum.
This protocol is a crucial step in developing the capacity to launch RCTs with meaningful outcomes that address comparative effectiveness questions in the field of paediatric IMD. Such trials will contribute high-quality evidence to inform decision-making by patients and their family members, clinicians, and policy-makers.
Inherited metabolic diseases (IMD) are a large group of single-gene diseases, most often diagnosed early in life, that are individually rare but collectively have an estimated birth prevalence of at least 1 in 2500 [1, 2]. These conditions exemplify the challenges of delivering high-quality care for children with rare chronic diseases . Healthcare needs are disproportionately high and often complex [4, 5], caregiver burden is known to be substantial , and evidence on the effectiveness of care is sparse and challenging to generate [7, 8]. With promising new interventions for IMD (including drug, dietary, stem cell, and surgical therapies) rapidly emerging , it is vital that their efficacy be evaluated in explanatory studies within controlled settings and that their effectiveness comparative to current standard treatment be assessed in pragmatic studies within real-world settings [8, 10, 11]. Such evaluations should be based upon outcomes of greatest clinical importance and of importance to patients and their family members and should capture measures of cost and health system impact [11, 12]. The careful consideration of these different perspectives is particularly critical when studying treatments for rare diseases, including IMD, since the relative benefits and harms of alternative therapies may differ depending on which outcomes are studied from across Berwick and colleagues’ ‘triple aim’ (i.e., medically-defined outcomes, patient experiences and quality of life, and health system impacts) .
While randomized controlled trials (RCTs) represent the evaluative study design with the greatest ability to minimize sources of bias and maximize internal validity, several challenges have been documented previously with the implementation of this design when assessing interventions for rare conditions. These include poor feasibility to attain adequate sample size to achieve planned statistical power [13,14,15], reliance on surrogate endpoints of limited importance to patients and their caregivers [14, 16, 17], and, associated with inadequate sample size, the challenge of sufficiently evaluating heterogeneity of benefits across relevant subgroups of patients [11, 13, 18]. To address these challenges, experimental and analytic methods tailored to rare disease settings have been developed [19,20,21,22]. One such method that has been described as an important innovation is the registry-based randomized trial . This approach involves the embedding of intervention trials within observational cohort studies or patient registries and adopting a pragmatic rather than explanatory focus to support real-world decision-making [23,24,25,26,27,28,29]. In the context of evaluating interventions for rare paediatric conditions, registry-based trials may offer numerous advantages including efficiencies in patient recruitment and data collection (given reliance on existing cohorts and routinely collected data), better access to follow-up data to understand long-term outcomes, suitability for addressing comparative effectiveness questions (given that registries recruit patients from routine clinical settings), and a high degree of external validity (given that registries are frequently population-based).
An important design element in the development of registry-based RCTs and other robust evaluative studies is the high-quality standardized collection of data on important outcomes, i.e., a core outcome set (COS) . The Core Outcome Measures in Effectiveness Trials (COMET) initiative (www.comet-initiative.org) promotes the performance of literature reviews and multi-stakeholder consensus approaches for the establishment of standardized COS to be used in subsequent evaluative studies [30,31,32,33]. These and similar methods have been used to develop outcome sets for paediatric conditions (e.g., traumatic brain injury, acute and chronic pain, fever and neutropenia in cancer, otitis media, inflammatory bowel disease) [34,35,36,37,38,39,40] and rare conditions [41,42,43]. Currently, there is no standardized COS for studies evaluating interventions for paediatric IMD. The Canadian Inherited Metabolic Diseases Research Network (CIMDRN) is ideally suited to address this research gap. CIMDRN is a pan-Canadian multidisciplinary network of investigators funded by the Canadian Institutes of Health Research, with an established cohort of children born during the years 2006–2015, diagnosed with one of 31 targeted IMD, and treated at one of 13 participating Canadian centres . CIMDRN collects observational data for participating children from a range of sources, including retrospective review of patient chart data. Developing and implementing standardized COS and data collection tools will enable the transition of the CIMDRN data platform to a registry format that can support registry-based randomized comparative effectiveness trials.
This protocol outlines a two-part study that will establish a COS for each of two of the most common IMD in children, phenylketonuria (PKU) and medium-chain acyl-CoA dehydrogenase (MCAD) deficiency (our study is registered in the COMET database, http://comet-initiative.org/studies/details/995). The first part of the study is a knowledge synthesis project that includes a rapid review and development of an evidence map to identify a comprehensive listing of outcomes reported or suggested in past studies of PKU and MCAD deficiency. The review findings will be used to inform the second part of the study, a Delphi consensus process involving patients or their family members, healthcare providers, and health system decision-makers, to establish priority outcomes. The final COS will be used in future studies to evaluate therapies for children diagnosed with PKU or MCAD deficiency.
Rationale for target conditions of interest
PKU is an inborn error of the phenylalanine hydroxylase enzyme resulting in elevated phenylalanine (phe) and reduced tyrosine concentrations in the blood and tissues, which if untreated is associated with both behavioural and intellectual disabilities. Current treatment involves dietary restriction of phe beginning in the newborn period . While this strategy is largely successful in preventing the most negative outcomes, its implementation is challenging. Currently available phe-restricted formulae and foods are unpleasant with regard to taste and require daily calculation and meal planning of phe/protein intake, resulting in challenges to diet adherence for patients and their caregivers [45, 46]. There remains a relatively high prevalence of neuropsychiatric (features of attention deficit hyperactivity disorder (ADHD), anxiety, depression) and executive functioning problems in this population [47, 48]. Recent therapeutic developments include pharmacological treatments such as sapropterin dihydrochloride, and supplements such as large neutral amino acids; both are add-ons to standard diet therapy and have the potential to further reduce blood phe levels. Sapropterin dihydrochloride therapy has been found to improve blood phe levels, dietary phe tolerance, ADHD symptoms and executive function [49, 50] in a subset of patients who respond to this therapy, but has not been evaluated with respect to other key patient-oriented outcomes. Sapropterin has a high annual cost (range C$12 K to C$170 K in Canada from infancy to adulthood and dependent on body weight ) with variable reimbursement coverage within Canada and internationally. Large neutral amino acids have not to date been evaluated using robust trial methodology with patient-oriented outcomes [52,53,54].
MCAD deficiency is an inborn error of the MCAD enzyme leading to impaired oxidation of medium-chain fatty acids. Affected patients are at risk of metabolic decompensation precipitated by catabolic stress during periods of prolonged fasting and fever. Clinical manifestations include hypoglycaemia, encephalopathy, cardiomyopathy, and cardiac arrest. Potentially lethal decompensation can be avoided by providing sufficient rapidly accessible caloric intake (e.g., carbohydrates) in at-risk situations. However, in addition to avoidance of prolonged fasting, it is unclear which, if any, interventions (e.g., routine dietary fat restriction, preventive use of cornstarch) have true preventive effectiveness. While daily oral L-carnitine supplementation is used for preventive management in some patients with MCAD deficiency, there is considerable practice variation  and concerns about potential adverse effects [56, 57]. To our knowledge there are no prospective trials (or other robust sources of evidence) on the use of L-carnitine for MCAD deficiency, nor studies that have focused on meaningful patient-oriented outcomes [58,59,60,61,62,63,64,65].
PKU and MCAD deficiency represent two of the most common IMD and are associated with important health effects for both children and caregivers. There are sizable gaps in the knowledge related to the evaluation of new and existing therapies for both diseases. Given these important limitations, further research into both conditions is of great importance. Thus, we have prioritized these two IMD for the initial establishment of COS for implementation in future registry-based trials.
Rapid review and evidence map
The rapid review and evidence map will address the following primary research question: “What outcomes are reported in key publications related to children diagnosed with phenylketonuria and medium-chain acyl-CoA dehydrogenase deficiency?” A secondary research question is “Among key publications as described above, what outcome measurement instruments are described or used to collect data for these outcomes?” The review will build on CIMDRN’s previous work, including a scoping review of patient/parent-reported outcomes in chronic paediatric illness  and qualitative studies of patient/family priorities [6, 66].
The rapid review protocol was prepared in consultation with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-P checklist (Additional file 1.1)  and registered with PROSPERO (CRD42017073524). The review will be performed according to the methods outlined below, and any post-hoc protocol modifications will be described in the final report.
Literature search to identify relevant studies
PKU and MCAD deficiency
To capture key publications specifically related to PKU and MCAD deficiency, we have developed separate search strategies for each of the two conditions. These will be implemented by a trained information specialist with extensive experience in systematic reviews (B Skidmore). Using a combination of controlled vocabulary (e.g., “Phenylketonurias”, “Acyl-CoA Dehydrogenase/df [Deficiency]”) and keywords (e.g., (“phenylketonuria”, “PKU”, “MCAD”), we will search MEDLINE, Embase, and the Cochrane Library. Study design filters will be applied to the PKU search (see inclusion criteria below), while no filters will be applied to the MCAD search due to the anticipated minimal volume of material associated with that topic based upon preliminary inspection of the literature.
Other key publications of interest focus on initiatives for the long-term follow-up evaluation of newborn screening programmes; such initiatives encompass but are not limited to outcomes of newborn screening for PKU and/or MCAD deficiency. These publications will be identified by conducting an additional search. The search strategy for this additional search has been developed for MEDLINE and Embase only, and includes controlled vocabulary (e.g., “Metabolism, Inborn Errors”, “Neonatal Screening”, “Outcome Assessment (Health Care”)) and keywords (e.g., “newborn screening”, “inborn metabolic disease”, “long-term follow-up”).
Due to resource constraints and in the context of this rapid review, all results will be limited to the English language and to the publication years 1990 to the present. When possible, animal-only records will be removed from the results. All strategies will be independently peer reviewed prior to execution by a second information specialist using the Peer Review of Electronic Search Strategies (PRESS) checklist . The proposed search strategies are provided in Additional file 1.2.
Given the challenge of searching this topic (e.g., unrefined and diverse vocabulary), we will undertake supplementary searches for citations to key articles pertaining to long-term follow-up initiatives for newborn screening using Web of Science and Google Scholar. We will also use the “related citation” feature in PubMed and other databases to find articles that have been similarly indexed. Additionally, we will perform a grey literature search of targeted sites (e.g., Newborn Screening Translational Research Network) to identify existing documents related to COS and will supplement this by searching additional relevant sites as identified in the Canadian Agency for Drugs and Technologies in Health (CADTH) tool, Grey Matters (https://www.cadth.ca/resources/finding-evidence/grey-matters). Given our resource constraints, the grey literature search will be limited to what can be accomplished within 15 hours by one reviewer.
Other COMET paediatric projects
In order to synthesize and acknowledge what other groups have been incorporating into their COS for other paediatric conditions, we will also conduct a grey literature search of the COMET database. This will not be specific to PKU or MCAD deficiency, rather it will focus on chronic paediatric conditions.
Study eligibility criteria
Eligibility criteria that will be used to identify relevant studies for each of the planned reviews have been pre-defined in terms of the patient, population or problem, intervention, comparison, outcomes, and study design/setting (PICOS) criteria (Table 1).
Process of study screening and selection
We will identify and remove duplicates from the bibliographic and grey literature searches. The remaining articles will be uploaded into an online systematic review software package, Distiller SR (Evidence Partners, Inc., Ottawa, Canada) for level 1 (title/abstract) and level 2 (full text) screening (draft screening forms are provided in Additional file 1.3). Level 1 screening will be performed by two independent reviewers (among M Pugliese, A Rahman, K Tingley). A liberal accelerated approach will be used for level 1, where a single reviewer will need to classify the title/abstract as potentially eligible for it to advance forward to level 2. At this first stage, in order for an article to be excluded, two reviewers need to agree on its exclusion. Level 2 screening will also be conducted by two independent reviewers (among M Pugliese, A Rahman, K Tingley), who will need to agree fully on the article’s inclusion/exclusion.
Conflicts at full-text screening will be resolved by consensus discussion or discussion with a third team member. A pilot screen will be implemented for level 1 (n = 20) and level 2 (n = 10) prior to the commencement of the screening process to achieve agreement between reviewers. A second pilot screen may be implemented if changes are made on the basis of the first pilot screen. Reports that are co-publications or multiple reports of the same study will be identified as such. A PRISMA flow diagram will be used to summarize the number of studies included and excluded with reasons provided for exclusion at level 2 . Articles presented in abstract form only (including conference abstracts) will not be included.
The data extraction form for this review will be implemented in Microsoft Excel and pilot tested on a sample of studies (n = 10). One reviewer will extract data and a second reviewer will verify the information collected. For all identified studies, we will extract the following: publication characteristics, study design, population descriptors (e.g., patient age distribution, diagnoses), intervention and comparator details (for studies with the primary purpose of evaluating one or more interventions), and outcomes (general description, outcome measurement instruments, validation status). Details of the data elements we plan to extract are provided in Additional file 1.4. As the purpose of the current study is the development of a map of the outcomes assessed and reported in studies of interest (without formal data syntheses), no risk of bias evaluations will be performed. Authors of the included studies will not be contacted for missing or unclear data, due to time restraints.
Analysis and presentation of summarized findings
We will summarize the results separately for each disease. In addition to summarizing study characteristics, evidence tables will be used to list outcomes and outcome measurement instruments, by study. We will summarize outcomes by date of publication and age of the children studied. In order to manage an expected large number of outcomes based on the rapid review findings [33, 35, 40], we will combine those outcomes that may have been given different names by study authors but that reflect the same underlying construct. We will also organize the outcomes into core areas and domains within these core areas. Following other paediatric COS projects, we will rely on the Outcome Measures in Rheumatology (OMERACT) core areas of death, life impact, pathophysiological manifestations, and resource use, and we will add the paediatric-specific core area of growth and development [35, 70]. The research team will establish a definition for each outcome and we will place outcomes within domains or categories established based on discussion among members of our multidisciplinary team. Each outcome will be mapped to a domain and each domain placed within a core area.
Delphi consensus process and final COS development
To establish consensus on COS for PKU and MCAD deficiency, we will use separate (disease-specific) Delphi surveys with patients/family members, healthcare providers, and health system decision-makers as survey participants, following methods recommended for COS development  and consensus approaches used by members of our research team [71, 72].
Delphi participants and recruitment strategy
Involving patients or their family members as well as providers and health system decision-makers in Delphi surveys to establish a COS ensures broad input with respect to the outcomes selected. Relative to other data collection strategies, surveys also offer respondent confidentiality, which may reduce the influence of power differentials among stakeholder groups . Thus, potential participants in the Delphi process will include Canadian metabolic physicians and dietitians who are members of the Garrod Association, a Canadian professional association of care providers and care centres in the field of IMD (approximately 80 eligible providers); family members of children with PKU (approximately 130 eligible families) or MCAD deficiency (approximately 70 eligible families) who are participating in the CIMDRN; and health system decision-makers involved in orphan drug regulation and reimbursement of drugs and supplements at the Canadian provincial or local (hospital) level (we will aim to identify approximately 30 eligible decision-makers). We will contact potential participants by email (healthcare providers, system decision-makers) or mail (CIMDRN families) up to three times to invite their participation in mailed and/or internet-based Delphi surveys, with consent provided by mail, email, or telephone.
CIMDRN families who are invited to participate will receive information about COS and Delphi surveys that is written in lay language. This will be supported by the work of the Family Advisory Forum, composed of approximately six to eight individuals who are family members of children with IMD (including but not limited to families of children with PKU and MCAD deficiency). Family Advisory Forum members have agreed to work in partnership with the investigator team, and particularly with the patient/family member-investigators on our research team (N Pallone, M Smith), to ensure that this project incorporates their perspectives and that the final outcomes that are part of the COS are meaningful to patients and their caregivers.
Delphi questionnaire development, data collection, and analysis
The research team and Family Advisory Forum will collaborate to develop disease-specific questionnaires for the first round of the Delphi survey, which will incorporate lists of candidate outcomes based on the results of the rapid review and evidence map. The outcomes will be organized within domains and core areas on the questionnaire in order to manage the long list and so that survey participants may think about their responses within these strata. We will ensure that the questionnaires include clear and meaningful content and language for family members of children with MCAD deficiency and PKU. Participants will be able to complete the surveys online or by mail.
For each disease, we will conduct at least two rounds of survey data collection; a possible third round will depend on the degree to which consensus is achieved . In the first-round survey, participants will be presented with a list of potentially relevant outcomes derived from the rapid review results. Respondents will be asked to score each outcome on a scale from 1 to 9 with respect to its perceived importance, whereby 1–3 represents an outcome “of limited importance”, 4–6 “important but not critical”, and 7–9 “critical” . Participants will also have the option of adding outcomes not already included on the list . The survey results will be analyzed separately for healthcare providers, family members, and health system decision-makers. For each outcome, we will tabulate the frequency and percentage of respondents within each stakeholder group who scored the outcome as 1–3, 4–6, or 7–9, together with the mean and standard deviation of the individual scores.
In the second round of each Delphi survey, the first round results will be summarized and presented to survey participants. The second survey will provide an opportunity for participants to re-score the original outcomes and to score any new suggested outcomes. To promote consensus across (rather than only within) stakeholder groups, the summary of the first-round results provided to each participant for each outcome in round two will include the participant’s own prior score and the summarized scores from every participant group, presented separately . The results will again be analyzed separately for each participant group. If at least 70% of respondents among any participant group (providers, family members, decision-makers) score an outcome as “critical” (score of 7–9) and fewer than 15% score it as “of limited importance” (score of 1–3), we will conclude that that participant group has reached consensus that the outcome is critical . Correspondingly, if at least 70% of respondents among any participant group (providers, family members, decision-makers) view an outcome as “of limited importance” and fewer than 15% score it as “critical”, we will conclude that that participant group has reached consensus that the outcome is of limited importance. The research team will decide whether a third Delphi round is necessary, based on consensus having been reached on a reasonable proportion of outcomes from the second-round findings. This is difficult to determine a priori, given uncertainty about how many outcomes will be included in the survey pending the rapid review findings.
Final COS development
The research team and Family Advisory Forum members will meet in person to review the Delphi results and to establish an initial COS for each of PKU and MCAD deficiency. Unfortunately it will not be feasible to invite all of the Delphi participants to attend the final consensus meeting in person, due to limited resources and the Canada-wide nature of the study. Some of the Delphi participants may be at the meeting by virtue of their inclusion on our research team (as patient/parent partners or clinician investigators) or on our Family Advisory Forum (some Forum members will be eligible to participate in the Delphi survey as parents of children with PKU or MCAD deficiency). To give other Delphi participants the opportunity to contribute to the final consensus, we propose to use web-casting or a similar approach to allow off-site (virtual) participation in the meeting. All Delphi respondents from all stakeholder groups will be invited to contribute in this manner and will also be invited to comment on each COS before they are finalized (see below).
At the final consensus meeting, all candidate outcomes considered critical by at least one stakeholder group based on the Delphi results will be considered for the final COS. Based on published outcome sets in other areas of paediatrics, we anticipate that each final COS will include approximately six to nine outcomes [38, 40]. If the number of outcomes deemed critical is viewed as too large to feasibly incorporate into a COS (e.g., more than ~10 outcomes), we will use the results of the Delphi to inform a discussion at the final consensus meeting in order to reduce the size of the COS. This may involve implementing stricter criteria for interpreting the Delphi findings (for example, ranking outcomes based on the mean numerical ratings). As the Delphi survey results are emerging, the study team will use literature searching, including the results of the rapid review described above, our previous scoping review , and initiatives such as PROMIS (paediatric item bank) , to identify potential outcome measurement instruments for the outcomes included in each COS, i.e., to operationalize the outcomes with data collection tools. Candidate outcome measurement instruments will be reviewed within the study team against the following criteria:
Face validity: do team members, particularly patients/family members, clinicians, and policy-makers, find the outcome measurement instrument meaningful as an indicator of the core outcome?
Measurement properties (guided by the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) initiative) : based on published evidence, is the outcome measurement instrument reliable? Does it have construct validity? Has it been demonstrated to be responsive to change? Have previous validation studies involved populations similar to those likely to be included in comparative effectiveness studies for paediatric PKU or MCAD deficiency?
Relevance: is the outcome measurement instrument likely to be broadly relevant across comparative effectiveness trials of different interventions? If appropriate, can it be used to evaluate cost-effectiveness?
Feasibility: as judged by the research team and Family Advisory Forum, can the outcome measurement instrument be implemented through online patient surveys or integrated into patients’ paper or electronic charts as part of routine clinical care?
We may revise the initial COS for each disease based on this process. For example, if there are outcomes deemed critical but for which there are no valid, reliable, relevant, and feasible outcome measurement instruments, these outcomes may be excluded from the COS but published as a separate list requiring urgent development of measurement instruments. Before being finalized, each COS will be made available on a study-specific web page to all Delphi participants for comment. The research team will hold a final teleconference to discuss any comments received and make any final revisions deemed important before publishing each COS. This study will yield a finalized COS for each of PKU and MCAD deficiency, and a set of candidate outcome measurement instruments for each COS.
Strong randomized trial-based evidence is seldom available to inform clinical practice guidelines for IMD such as PKU and MCAD deficiency [44, 76,77,78]. At a policy level, regulatory and reimbursement decisions about treatments for IMD are inconsistent across jurisdictions and frequently contentious, in part due to the scarcity of evidence to support decision-making [79, 80]. This protocol is a crucial step in developing the capacity to rapidly launch randomized trials with meaningful outcomes in response to emerging comparative effectiveness questions in the field of paediatric IMD. Such trials will support patients or their family members and clinicians in making decisions in routine practice and will help to resolve policy debates by contributing high-quality evidence for health system decision-making.
We describe a systematic approach to developing a COS, including the integration (through the Delphi process and/or within the research team) of the perspectives of patients/family members, clinical providers, health system decision-makers, and methodologists, suitable for incorporation into a paediatric IMD registry. This study will help to ensure that the methods of the trials catalyzed by our work meet all stakeholder expectations with respect to inclusion of meaningful and rigorous outcomes.
At the time of submission of this manuscript, the search strategy for the rapid review and evidence map has been peer-reviewed and implemented; and screening against inclusion/exclusion criteria is underway.
Applegarth DDA, Toone JJR, Lowry RB, Applegarth DDA, Toone JJR. Incidence of inborn errors of metabolism in British Columbia, 1969–1996. Pediatrics. 2000;105:e10.
Sanderson S, Green A, Preece MA, Burton H. The incidence of inherited metabolic disorders in the West Midlands, UK. Arch Dis Child. 2006;91:896–9.
Schieppati A, Henter J-II, Daina E, Aperia A. Why rare diseases are an important medical and social issue. Lancet. 2008;371:2039–41.
Jimenez-Sanchez G, Childs B, Valle D. The effect of mendelian disease on human health. In: Valle D, Beaudet AL, Vogelstein B, Kinzler KW, Antonarakis SE, Ballabio A, et al., editors. On-line Metab. Mol. Bases Inherit. Dis. McGraw-Hill. Available from: www.ommbid.com. Accessed 13 Dec 2017.
Cohen E, Berry JG, Camacho X, Anderson G, Wodchis W, Guttmann A. Patterns and costs of health care use of children with medical complexity. Pediatrics. 2012;130:e1463–70.
Siddiq S, Wilson BJ, Graham ID, Lamoureux MF, Khangura SD, Tingley K, on behalf of the Canadian Inherited Metabolic Diseases Research Network, et al. Experiences of caregivers of children with inherited metabolic diseases: a qualitative study. Orphanet J Rare Dis. 2016;11:168.
Vockley J, Vockley CMW. Clinical trials: Curing a critical deficiency in metabolic medicine. Mol Genet Metab. 2010;99:244–5.
Potter BK, Chakraborty P, Kronick JB, Wilson K, Coyle D, Feigenbaum A, et al. Achieving the “triple aim” for inborn errors of metabolism: a review of challenges to outcomes research and presentation of a new practice-based evidence framework. Genet Med. 2013;15:415–22.
Vernon HJ. Inborn errors of metabolism: advances in diagnosis and therapy. JAMA Pediatr. 2015;169:778–82.
Westfall JM, Mold J, Fagnan L. Practice-based research - “blue highways” on the NIH roadmap. JAMA. 2007;297:403–6.
Potter BK, Khangura SD, Tingley K, Chakraborty P, Little J. Translating rare-disease therapies into improved care for patients and families: what are the right outcomes, designs, and engagement approaches in health-systems research? Genet Med. 2016;18:117–23.
Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff. 2008;27:759–69.
Augustine EF, Adams HR, Mink JW. Clinical trials in rare disease: challenges and opportunities. J Child Neurol. 2013;28:1142–50.
Wilcken B. Rare diseases and the assessment of intervention: what sorts of clinical trials can we use? J Inherit Metab Dis. 2001;24:291–8.
Griggs RC, Batshaw M, Dunkle M, Gopal-Srivastava R, Kaye E, Krischer J, et al. Clinical research for rare disease: opportunities, challenges, and solutions. Mol Genet Metab. 2009;96:20–6.
Basch E, Bennett AV. Patient-reported outcomes in clinical trials of rare diseases. J Gen Intern Med. 2014;29:13–5.
Lachmann R, Schoser B. The clinical relevance of outcomes used in late-onset Pompe disease: can we do better? Orphanet J Rare Dis. 2013;8:160.
Facey K, Granados A, Guyatt G, Kent A, Shah N, van der Wilt GJ, et al. Generating health technology assessment evidence for rare diseases. Int J Technol Assess Health Care. 2014;30:416–22.
Gagne JJ, Thompson L, O’Keefe K, Kesselheim AS. Innovative research methods for studying treatments for rare diseases: methodological review. BMJ. 2014;349:g6802.
Cornu C, Kassai B, Fisch R, Chiron C, Alberti C, Guerrini R, et al. Experimental designs for small randomised clinical trials: an algorithm for choice. Orphanet J Rare Dis. 2013;8:48.
Gupta S, Faughnan ME, Tomlinson GA, Bayoumi AM. A framework for applying unfamiliar trial designs in studies of rare diseases. J Clin Epidemiol. 2011;64:1085–94.
Hampson LV, Whitehead J, Eleftheriou D, Brogan P. Bayesian methods for the design and interpretation of clinical trials in very rare diseases. Stat Med. 2014;33:4186–201.
Lauer MS, D’Agostino RB. The randomized registry trial - the next disruptive technology in clinical research? N Engl J Med. 2013;369:1579–81.
Vickers AJ, Scardino PT. The clinically-integrated randomized trial: proposed novel method for conducting large trials at low cost. Trials. 2009;10:14.
Relton C, Torgerson D, O’Cathain A, Nicholl J. Rethinking pragmatic randomised controlled trials: Introducing the “cohort multiple randomised controlled trial” design. BMJ. 2010;340:963–7.
Kwakkenbos L, Jewett LR, Baron M, Bartlett SJ, Furst D, Gottesman K, et al. The Scleroderma Patient-centered Intervention Network (SPIN) Cohort: protocol for a cohort multiple randomised controlled trial (cmRCT) design to support trials of psychosocial and rehabilitation interventions in a rare disease context. BMJ Open. 2013;3:e003563.
Newman AB, Avilés-Santa ML, Anderson G, Heiss G, Howard WJ, Krucoff M, et al. Embedding clinical interventions into observational studies. Contemp Clin Trials. 2016;46:100–5.
Li G, Sajobi TT, Menon BK, Korngut L, Lowerison M, James M, et al. Registry-based randomized controlled trials: advantages, challenges and areas for future research. J Clin Epidemiol. 2016;80:16–24.
TwiCs network. Trials within cohorts, the design. Available: https://www.twics.global/the-design-twics. Accessed 5 Nov 2017.
Williamson PR, Altman DG, Blazeby JM, Clarke M, Gargon E. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative. Trials. 2011;12:A70.
Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One. 2014;9(6):e991114.
Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:1.
Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET handbook: version 1.0. Trials. 2017;18:280.
Sinha IA, Altman DG, Beresford MW, Boers M, Clarke M, Craig J, et al. Standard 5: selection, measurement, and reporting of outcomes in clinical trials in children. Pediatrics. 2012;129:S146–52.
Kapadia MZ, Joachim KC, Balasingham C, Cohen E, Mahant S, Nelson K, et al. A core outcome set for children with feeding tubes and neurologic impairment: a systematic review. Pediatrics. 2016;138. https://doi.org/10.1542/peds.2015-3967
Mccauley SR, Wilde EA, Anderson VA, Bedell G, Beers SR, Campbell TF, et al. Recommendations for the use of common outcome measures in pediatric traumatic brain injury research. J Neurotrauma. 2012;29:678–705.
McGrath PJ, Walco GA, Turk DC, Dworkin RH, Brown MT, Davidson K, et al. Core outcome domains and measures for pediatric acute and chronic/recurrent pain clinical trials: PedIMMPACT recommendations. J Pain. 2008;9(9):771–83.
Haeusler GM, Phillips RS, Lehrnbecher T, Thursky KA, Sung L, Ammann RA. Core outcomes and definitions for pediatric fever and neutropenia research: a consensus statement from an international panel. Pediatr Blood Cancer. 2015;62(3):483–9.
Ruemmele FM, Hyams JS, Otley A, Griffiths A, Kolho K-L, Dias JA, et al. Outcome measures for clinical trials in paediatric IBD: an evidence-based, expert-driven practical statement paper of the paediatric ECCO committee. Gut. 2014;64(3):438–46.
Bruce I, Harman N, Williamson P, Tierney S, Callery P, Mohiuddin S, et al. The management of otitis media with effusion in children with cleft palate (mOMEnt): a feasibility study and economic evaluation. Health Technol Assess. 2015;19:1–374.
Choquet R, Maaroufi M, De Carrara A, Messiaen C, Luigi E, Landais P. A methodology for a minimum data set for rare diseases to support national centers of excellence for healthcare and research. J Am Med Inform Assoc. 2015;22(1):76–85.
Shapiro E, Bernstein J, Adams HR, Barbier AJ, Buracchio T, Como P, et al. Neurocognitive clinical outcome assessments for inborn errors of metabolism and other rare conditions. Mol Genet Metab. 2016;118(2):65–9.
Taruscio D, Mollo E, Gainotti S, Posada De La Paz M, Bianchi F, Vittozzi L. The EPIRARE proposal of a set of indicators and common data elements for the European platform for rare disease registration. Arch Public Health. 2014;72(1):35.
Vockley J, Andersson HC, Antshel KM, Braverman NE, Burton BK, Frazier DM, et al. Phenylalanine hydroxylase deficiency: diagnosis and management guideline. Genet Med. 2014;16:188–200.
MacDonald A, Gokmen-Ozel H, van Rijn M, Burgard P. The reality of dietary compliance in the management of phenylketonuria. J Inherit Metab Dis. 2010;33:665–70.
Hartnett C, Salvarinova-Zivkovic R, Yap-Todos E, Cheng B, Giezen A, Horvath G, et al. Long-term outcomes of blood phenylalanine concentrations in children with classical phenylketonuria. Mol Genet Metab. 2013;108:255–8.
Sullivan J, Chang P. Review: Emotional and behavioral functioning in phenylketonuria. J Pediatr Psychol. 1999;24:281–99.
Smith I, Knowles J. Behaviour in early treated phenylketonuria: a systematic review. Eur J Pediatr. 2000;159 Suppl 2:S89–93.
Trefz FK, Burton BK, Longo N, Casanova MM-P, Gruskin DJ, Dorenbaum A, et al. Efficacy of sapropterin dihydrochloride in increasing phenylalanine tolerance in children with phenylketonuria: a phase III, randomized, double-blind, placebo-controlled study. J Pediatr. 2009;154:700–7.
Burton B, Grant M, Feigenbaum A, Singh R, Hendren R, Siriwardena K, et al. A randomized, placebo-controlled, double-blind study of sapropterin to treat ADHD symptoms and executive function impairment in children and adults with sapropterin-responsive phenylketonuria. Mol Genet Metab. 2015;114:415–24.
Canadian Agency for Drugs and Technologies in Health. CADTH Canadian Drug Expert Committee final recommendation: sapropterin - resubmission. Available: https://www.cadth.ca/sites/default/files/cdr/complete/SR0472_complete_Kuvan-Oct-28-16.pdf. (Accessed 5 Nov 2017).
Matalon R, Michals-Matalon K, Bhatia G, Burlina ABP, Burlina ABP, Braga C, et al. Double blind placebo control trial of large neutral amino acids in treatment of PKU: effect on blood phenylalanine. J Inherit Metab Dis. 2007;30:153–8.
Matalon R, Michals-Matalon K, Bhatia G, Grechanina E, Novikov P, McDonald JD, et al. Large neutral amino acids in the treatment of phenylketonuria (PKU). J Inherit Metab Dis. 2006;29:732–8.
Schindeler S, Ghosh-Jerath S, Thompson S, Rocca A, Joy P, Kemp A, et al. The effects of large neutral amino acid supplements in PKU: an MRS and neuropsychological study. Mol Genet Metab. 2007;91:48–54.
Potter BK, Little J, Chakraborty P, Kronick JB, Evans J, Frei J, et al. Variability in the clinical management of fatty acid oxidation disorders: results of a survey of Canadian metabolic physicians. J Inherit Metab Dis. 2012;35:115–23.
Ufnal M, Zadlo A, Ostaszewski R. TMAO: A small molecule of great expectations. Nutrition. 2015;31(11-12):1317–23.
Miller MJ, Bostwick BL, Kennedy AD, Donti TR, Sun Q, Sutton VR, et al. Chronic oral L-Carnitine supplementation drives marked plasma TMAO elevations in patients with organic acidemias despite dietary meat restrictions. JIMD Rep. 2016;30:39–446.
Walter JH. L-Carnitine in inborn errors of metabolism: What is the evidence? J Inherit Metab Dis. 2003;26:181–8.
Rinaldo P, Schmidt-Sommerfeld E, Posca AP, Heales SJR, Woolf DA, Leonard JV. Effect of treatment with glycine and l-carnitine in medium-chain acyl-coenzyme A dehydrogenase deficiency. J Pediatr. 1993;122:580–4.
Schmidt-Sommerfeld E, Penn D, Kerner J, Bieber LL, Rossi TM, Lebenthal E. Quantitation of urinary carnitine esters in a patient with medium-chain acyl-coenzyme A dehydrogenase deficiency: Effect of metabolic state and l-carnitine therapy. J Pediatr. 1989;115:577–82.
Treem WR, Stanley CA, Goodman SI. Medium-chain acyl-CoA dehydrogenase deficiency: Metabolic effects and therapeutic efficacy of long-term l-carnitine supplementation. J Inherit Metab Dis. 1989;12:112–9.
Van Hove JL, Kahler SG, Millington DS, Roe DS, Chace DH, Heales SJ, et al. Intravenous L-carnitine and acetyl-L-carnitine in medium-chain acyl-coenzyme A dehydrogenase deficiency and isovaleric acidemia. Pediatr Res. 1994;35:96–101.
Wilson CJ, Champion MP, Collins JE, Clayton PT, Leonard JV. Outcome of medium chain acyl-CoA dehydrogenase deficiency after diagnosis. Arch Dis Child. 1999;80:459–62.
Huidekoper HH, Schneider J, Westphal T, Vaz FM, Duran M, Wijburg FA. Prolonged moderate-intensity exercise without and with L-carnitine supplementation in patients with MCAD deficiency. J Inherit Metab Dis. 2006;29:631–6.
Lee PJ, Harrison EL, Jones MG, Jones S, Leonard JV, Chalmers RA. L-Carnitine and exercise tolerance in medium-chain acyl-coenzyme A dehydrogenase (MCAD) deficiency: a pilot study. J Inherit Metab Dis. 2005;28:141–52.
Khangura SD, Karaceper MD, Trakadis Y, Mitchell JJ, Chakraborty P, Tingley K, et al. Scoping review of patient- and family-oriented outcomes and measures for chronic pediatric disease. BMC Pediatr. 2015;15:7.
Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. 2015;7647:1–25.
Sampson M, Mcgowan J, Cogo E, Grimshaw J. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62:944–52.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.
Kapadia M, Chan W, Jegathesan T, Offringa M. Development of core outcome set for pediatric health conditions: a systematic review. Trials. 2015;16:P25.
Payne K, Nicholls SG, McAllister M, MacLeod R, Ellis I, Donnai D, et al. Outcome measures for clinical genetics services: a comparison of genetics healthcare professionals and patients’ views. Health Policy. 2007;84:112–22.
Nicholls SG, Quach P, von Elm E, Guttmann A, Moher D, Petersen I, et al. The reporting of studies conducted using observational routinely-collected health data (RECORD) statement: methods for arriving at consensus and developing reporting guidelines. PLoS One. 2015;10:e0125620.
Young B, Bagley H. Including patients in core outcome set development: issues to consider based on three workshops with around 100 international delegates. Res Involv Engagem. 2016;2:25.
Irwin DE, Stucky BD, Thissen D, DeWitt EM, Lai JS, Yeatts K, et al. Sampling plan and patient characteristics of the PROMIS pediatrics large-scale survey. Qual Life Res. 2010;19:585–94.
Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63:737–45.
Arnold GL, Van Hove J, Freedenberg D, Strauss A, Longo N, Burton B, et al. A Delphi clinical practice protocol for the management of very long chain acyl-CoA dehydrogenase deficiency. Mol Genet Metab. 2009;96:85–90.
Kolker S, Christensen E, Leonard JV, Greenberg CR, Boneh A, Burlina AB, et al. Diagnosis and management of glutaric aciduria type I–revised recommendations. J Inherit Metab Dis. 2011;34:677–94.
Scarpa M, Almássy Z, Beck M, Bodamer O, Bruce IA, De Meirleir L, et al. Mucopolysaccharidosis type II: European recommendations for the diagnosis and multidisciplinary management of a rare disease. Orphanet J Rare Dis. 2011;6:72.
Onakpoya IJ, Spencer EA, Thompson MJ, Heneghan CJ. The effectiveness, safety and costs of orphan drugs: an evidence-based review. BMJ Open. 2015;37:e22.
Atkins D, Siegel J, Slutsky J. Making policy when the evidence is in dispute. Health Aff. 2005;24:102–13.
This study will be conducted using funds provided from a successful catalyst grant application for innovative clinical trials submitted to the Canadian Institutes of Health Research (CIHR grant reference number 151614). The funders will play no role in the design, conduct or final approval of the review protocol or final review manuscripts.
Availability of data and materials
Not applicable (no datasets have been generated as part of this protocol).
Not applicable (optional, not included).
Ethics approval and consent to participate
Approval of the Delphi protocol will be sought from the University of Ottawa Research Ethics Board for clinicians and health system decision-makers; and from participating centres for families (see Additional file 2) prior to inviting participants to the Delphi surveys.
Consent for publication
Not applicable (no individual person’s data are included in this manuscript).
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.