Use of comparative effectiveness research for similar Chinese patent medicine for angina pectoris of coronary heart disease: a new approach based on patient-important outcomes
© Cao et al.; licensee BioMed Central Ltd. 2014
Received: 8 November 2013
Accepted: 24 February 2014
Published: 19 March 2014
The practice of traditional Chinese medicine (TCM) has a profound history in many Asian countries. TCM syndrome is a set of characteristic physical signs and symptoms shared by a group of patients. Syndrome diagnosis and treatment assignment according to the identified TCM syndrome is a long-held practice of Chinese medicine. Owing to its distinctive way of interpreting illness and administering care, medical practitioners not well educated in TCM theories and practices are generally incapable of giving out prescriptions for Chinese patent drugs. Currently, the existence of a multitude of Chinese patent drugs marked with largely identical indications is further complicating this situation.
In this multicenter, randomized, controlled, double-blind, double-dummy clinical trial, in which we will use the comparative effectiveness research method, we will compare the efficacy of two commonly used Chinese patent medicines for angina patients diagnosed with qi deficiency and blood stasis syndrome. A total of 160 patients will be recruited and randomly assigned to receive either (1) QiShenYiQi dripping pills, Tongxinluo placebo and routine medication or (2) Tongxinluo capsules, QiShenYiQi placebo and routine medication. These treatment regimens will be carried out for 4 weeks, followed by a 10-day washout period and a 4-week crossover phase in which the treatments in the two patient groups will be exchanged. Patients will be allowed to choose symptoms that matter most to them and will be grouped accordingly. Patient-reported outcomes such as the Seattle Angina Questionnaire score and the 15-point Likert scale score will be measured and reported. The minimally clinical important difference will be calculated and used for efficacy assessment, and correspondence analysis will be performed to identify the best indications for each drug.
The goal of the study is to establish a methodology for the precise identification of the characteristic indications for which a Chinese patent drug is most effective. The findings of this study will inform the practicality of the proposed evaluation method.
Chinese clinical trials register Chi CTRTTRCC13003732
KeywordsAngina pectoris Chinese patent medicine Comparative effectiveness research Patient-important outcomes
Traditional Chinese medicine interpretation of angina and therapeutic rules
Syndrome differentiation, also termed pattern diagnosis (Bian Zheng in Chinese pinyin), is one of the kernel theories of traditional Chinese medicine (TCM). The TCM syndrome, or Zheng, is a characteristic profile of clinical signs and symptoms manifested by a group of patients. The clinical efficacy of Chinese medicine relies heavily on the correct diagnosis of a specific TCM syndrome. According to the 2002 Guidelines for Clinical Research of Chinese Medicine (new drug)  released by the Chinese Food and Drug Administration patients with angina pectoris can be diagnosed on the basis of any one of the following eight TCM patterns: (1) qi deficiency and blood stasis syndrome, (2) heart blood stasis obstruction syndrome, (3) qi stagnation and blood stasis syndrome, (4) syndrome of phlegm obstruction in the heart vessel, (5) syndrome of congealing yin cold, (6) dual-deficiency of qi and yin syndrome, (7) heart and kidney yin deficiency syndrome and (8) yang qi debilitation syndrome. Among these criteria, the qi deficiency and blood stasis syndrome is most typically observed in Chinese angina patients.
For the management of angina pectoris patients with qi deficiency and blood stasis syndrome, herbal remedies intended to invigorate qi, activate blood circulation and transform stasis will be administered. The main function of the herbal remedy is to improve clinical signs and symptoms of the patients.
Comparative effectiveness research and traditional Chinese medicine
The concept of comparative effectiveness research (CER) was initiated in the 1990s by Mark Boutin, the deputy executive president and chief operating officer of the US National Health Council . In the years that followed, CER has been used to aid public health policy-making and introduced into the field of clinical research in a number of countries . In May 2011, Claudia M Witt (Institute of Social Medicine, Epidemiology and Health Economics, Charité University Medical Center, Berlin, Germany) proposed introducing CER techniques into TCM research at the sixth annual meeting of the International Society of Complementary Medicine Research .
Efficacy evaluation based on patient-reported outcomes
A patient-reported outcome (PRO) is a report directly from the patient, without influences from doctors or others, about how they function or feel in relation to their health status and the treatment they are receiving . The PRO measurement tools are developed to provide insights from the patient’s perspective regarding the impact of therapeutic interventions on their health. Patient-important outcomes (PIOs) are a selection of outcome measures that are considered most relevant to the patients. Examples of PIOs include pain, fatigue, quality of life and death. Efforts to assess PIOs for a particular disease or condition and to promote the use of PIOs in clinical research settings facilitate effective interpretation of the results by emphasizing the evaluation of treatment effectiveness. The integration of PIOs and PROs into Comparative Effectiveness Research could be a useful tool for the assessment of outcomes requiring patients’ subjective perceptions and judgments, such as consumer satisfaction and health-related quality of life .
Minimal clinically important differences and patient-reported outcome measurements
Minimal clinically important differences (MCIDs) are self-perceived scores that signify clinically significant improvements in PROs that can be used to identify the minimal changes in a specific aspect of a patient’s health , and it is believed to reflect the smallest change in the PRO evaluations of a clinical intervention that is meaningful to the patient.
Distribution-based methods for the determination of MCIDs have been widely used and can be subdivided into the effect size (ES) and the standard error of measurement (SEM). The latter measurement is particularly recommended for its relative independence from sample source and stability across different studies [8, 9]. In our present study, changes in PROs will be calculated using the SEM, which will represent patients’ self-reported improvements in symptoms or health-related function.
Traditional Chinese medicine for angina pectoris due to coronary heart disease
Angina pectoris is one of the most common symptoms of coronary heart disease (CHD) and afflicts a large portion of CHD patients. It has been found over the course of years of clinical practice that TCM is effective in treating angina. Because the qi deficiency and blood stasis syndrome together have been identified as a major TCM syndrome among patients with angina, in this CER study we will select two Chinese patent drugs most frequently used for treating angina patients diagnosed with this common syndrome. The patent drugs to be compared are QiShenYiQi (QSYQ) dripping pills (Tasly Pharmaceutical Co Ltd, Tianjin, China) and Tongxinluo (TXL) capsules (Yiling Pharmaceutical Co Ltd, Shijiazhuang City, China). QSYQ pills are composed mainly of Radix Astragali, Radix Salvia miltiorrhiza, Radix Notoginseng and Lignum Dalbergia Odorifera. TXL capsules are an herbal mixture of Radix Ginseng, Hirudo, Quan Xie Eupolyphaga seu Steleophaga, Scolopendra, Periostracum Cicadae, Radix Paeoniae rubra, Lignum Santali Albi, Lignum Dalbergia Odorifera, frankincense, Semen Ziziphi spinosae and borneol.
We are conducting a multicenter, randomized, controlled, double-blind, double-dummy clinical trial to compare the effectiveness of two Chinese patent medicines for angina patients diagnosed with qi deficiency and blood stasis syndrome. The ultimate goal is to test the proposed methodology for the individualized evaluation of the therapeutic effects of Chinese patent drugs and for distinguishing between drugs targeting a multitude of similar indications.
To fulfill these objectives in our present study, we highlight the introduction of trial participants’ therapeutic needs and willingness in terms of group allocation and the use of CER methodology. Specifically before being entered into the randomization process, all eligible participants with qi deficiency and blood stasis syndrome will first be allowed to choose from among four prespecified groups of symptoms which they would most like to be addressed. The outcomes for these symptoms are considered PIOs and will be assessed using PRO instruments. Furthermore, we will test the use of the CER method to compare the curative effects of QSYQ dripping pills and TXL capsules, two widely used Chinese patent medicines for angina patients with qi deficiency and blood stasis syndrome. By analyzing and comparing the effects of the two drugs for each symptom group, we aim to summarize the characteristics of the drugs’ curative effects and identify the best targeted indications for each drug.
This study has been approved by the medical ethics committee of Tianjin University of Traditional Chinese Medicine (registration number TJUTCM-EC20130004).
Participants will be recruited simultaneously at the cardiology departments of four hospitals in Tianjin City in mainland China.
Potentially eligible angina patients will be provided with general information regarding the trial, the objectives of the research and the rights and obligations of the participants at the first visit. Those who are willing to participate will be asked to give their informed consent to participate before being entered into the screening process. Participants will then be included or excluded according to the predefined inclusion and exclusion criteria.
Diagnostic criteria for traditional Chinese medicine patterns
According to the 2002 Guidelines for Clinical Research of Chinese Medicine (new drug), any patient presenting with the primary symptoms mentioned below and at least one of the secondary symptoms, with typical tongue and pulse presentations, will be diagnosed with qi deficiency and blood stasis syndrome. Primary symptoms include chest pain and chest tightness. Secondary symptoms include shortness of breath, fatigue, palpitations and/or spontaneous perspiration. .
Patients ages 40 to 75 years old
Provision of signed informed consent
Diagnosis of angina pectoris due to CHD (stable angina, New York Heart Association (NYHA) classes I to III)
TCM pattern diagnosis of qi deficiency and blood stasis
Patients younger than 40 or older than 75 years old
Diagnosis of NYHA class IV CHD
Uncontrolled NYHA class III hypertension (systolic blood pressure ≥180 mmHg and/or diastolic blood pressure ≥110 mmHg)
Presence of severe arrhythmia
Patients with serious primary hepatic, renal, hematologic or mental illness and those with malignant tumors
Presence of active peptic ulcers and other hemorrhagic diseases
Pregnancy or lactation
Patients with other complications who should not be included in the trial as adjudged by the recruiting personnel
Patients with an allergic constitution or a history of allergy to the investigational drug
Participation in another clinical trial, either currently or within the past 3 months
Incidence of serious adverse events.
Major design flaws or serious deviation from research protocol
Cancellation of the study by administrative authorities
Patient grouping and treatment allocation
Using stratified and blocked randomization, we will first categorize eligible participants into four groups on the basis of their varied choice of secondary symptoms to be addressed, and then we will randomly assign them to one of the two treatment arms at a ratio of 1:1. To minimize imbalances across groups, gender will also be used as a stratification factor. The random number sequence will be generated by a third-party statistician using SAS version 9.1 software (SAS Institute, Cary, NC, USA).
Combination 1: chest pain + chest tightness + shortness of breath
Combination 2: chest pain + chest tightness + fatigue
Combination 3: chest pain + chest tightness + palpitations
Combination 4: chest pain + chest tightness + spontaneous perspiration
Sample size calculation
Sample size estimation is based on the results of a pilot study that showed the difference in changes in Seattle Angina Questionnaire (SAQ) scores of 2 ± 2.5 between the group receiving QSYQ dripping pills and the group taking TXL capsules. Assuming a two-sided P-value of 0.05, 90% power and calculating for a dropout rate of 20%, one would need to have approximately 160 patients with 80 in each treatment group. We used the PAST software program to calculate these statistics (http://folk.uio.no/ohammer/past/) . Therefore, we plan to recruit 40 patients for each of the four symptom combinations, from among whom 20 will be assigned to treatment arm A (QSYQ) and 20 to treatment arm B (TXL). In all, 160 patients will be recruited.
Randomization of the trial participants will be completed using an independent data center using an interactive voice response system.
Blinding and allocation concealment
The test drug will be coded and placed in indistinguishable containers by specially assigned personnel who will not participate in other procedures in the trial. Drug assignments will be enclosed in sealed, opaque envelopes and kept confidential by the trial management board. Thus, the patients, clinicians, participating nurses, trial coordinators, outcome assessors and statisticians will be blinded to treatment assignment.
General information about the patient’s gender, age and heart function will be collected and assessed after eligibility screening to ensure balanced baseline values.
Assignment of interventions
- 1.The first treatment period (4 weeks): A total of 160 participants will be randomly assigned to receive treatment A or treatment B for 4 weeks.
Treatment A: QSYQ dripping pills + TXL placebo + routine medication
Treatment B: TXL capsules + QSYQ placebo + routine medication
Washout phase (10 days): After the first round of treatment, patients in both groups will receive QSYQ placebo and TXL placebo plus routine medication for 10 days.
Crossover of treatments (4 weeks): In this stage, patients formerly assigned to treatment A will be switched to treatment B and those assigned to treatment B will be switched to treatment A.
Routine medications will include aspirin, angiotensin-converting enzyme inhibitors or angiotensin receptor blockers, β-receptor blockers, statins and nitrates.
Dihydropyridines and diuretics
Digoxin and diuretics
Measurement tools and time of data capture
The SAQ and 15-point Likert scale scores will be used to collect PROs immediately following the first treatment period and the crossover treatment period. The minimal clinically important differences (MCIDs) of the SAQ and the Likert scale scores at these two time points will be calculated for efficacy evaluation and comparison.
Efficacy assessment tools
MCID calculation : The MCID value will be calculated using the SEM method and following the steps outlined below:
- 1.a.Calculation of M (M0 and M1) (Figure 2):
Find patients receiving treatment A (or treatment B) whose self-reported Likert scores are within the range of +2 to +3.
Collect the corresponding SAQ score of these patients, use X1 to refer to it and calculate the mean value .
Find patients receiving Treatment A (or Treatment B) whose self-reported Likert scores are within the range of 0 to +1.
Collect the SAQ scores of these patients, use X0 to refer to it and calculate the mean value X0
Calculation of MCID We will calculate the MCIDs for treatment A (referred to as MCID1) and treatment B (referred to as MCID2) during the first treatment period and for treatment A (referred to as MCID3) and treatment B (referred to as MCID4) during the crossover period.
Individualized efficacy evaluation upon the first round of treatment: Treatment interventions will be defined as either effective or ineffective by comparison of the patient’s personal SAQ score with the calculated MCID value.
Treatment A will be defined as effective if a patient receiving it has a SAQ score greater than the MCID1 score.
Treatment B will be defined as effective if a patient receiving it has a SAQ score greater than the MCID2 score.
A treatment will be adjudged ineffective if a patient receiving it reports a SAQ score lower than the corresponding MCID value (MCID1 or MCID2).
Individualized efficacy evaluation upon crossover of treatment: Because of the complexity of the study design, all the patients will be categorized into four groups by using the following assessment tools:
Group 1: Both treatments A and B are effective for patients in this group:
For patients who first receive treatment A and then treatment B, the first SAQ score > MCID1 and the second SAQ score > MCID4
For patients who first receive treatment B and then treatment A, the first SAQ score > MCID2 and the second SAQ score > MCID3
Group 2: Treatment A is effective and treatment B is ineffective for patients in this group:
For patients who first receive treatment A and then treatment B, the first SAQ score > MCID1 and the second SAQ score < MCID4
For patients who first receive treatment B and then treatment A, the first SAQ score > MCID2 and the second SAQ score < MCID3.
Group 3: Treatment B is effective and treatment A is ineffective for patients in this group:
For patients who first receive treatment A and then treatment B, the first SAQ score < MCID1 and the second SAQ score > MCID4
For patients who first receive treatment B and then treatment A, the first SAQ score < MCID2 and the second SAQ score > MCID3
Group 4: Both treatments A and B are ineffective for patients in this group:
For patients who first receive treatment A and then treatment B, the first SAQ score < MCID1 and the second SAQ score < MCID4
For patients who first receive treatment B and then treatment A, the first SAQ score < MCID2 and the second SAQ score < MCID3
Test for baseline balance
Baseline data will be analyzed using an independent t-test, analysis of variance and the χ 2 test to check whether the randomization has resulted in equal distributions of known confounding factors, such as age, gender, living status and education level.
Comparison of the curative effects of treatment A and treatment B on symptoms related to angina
Efficacy assessment by category of single symptom combination
Type of symptom combination
Four groups of patients categorized using the efficacy assessment tool
AB1 to AB4 refer to the number of patients requiring treatment for symptom combinations 1 to 4 and for whom both QSYQ and TXL are effective. A1 to A4 refer to the number of patients requiring treatment for symptom combinations 1 to 4 and for whom QSYQ is effective and TXL is ineffective. B1 to B4 refer to the number of patients requiring treatment for symptom combinations 1 to 4 and for whom QSYQ is ineffective and TXL is effective. ab1 to ab4 refer to the number of patients requiring treatment for symptom combinations 1 to 4 and for whom both QSYQ and TXL are ineffective N refers to the total number of patients who fall within each symptom combination group, which is planned to be 40.
Efficacy assessment by category of multiple symptom combinations
Multiple symptom combinations
Four groups of patients categorized using the efficacy assessment tool
Combination (1 + 2)
(AB1 + AB2)/2 N
(A1 + A2)/2 N
(B1 + B2)/2 N
(ab1 + ab2)/2 N
Combination (1 + 3)
(AB1 + AB3)/2 N
(A1 + A3)/2 N
(B1 + B3)/2 N
(ab1 + ab3)/2 N
Combination (1 + 4)
(AB1 + AB4)/2 N
(A1 + A4)/2 N
(B1 + B4)/2 N
(ab1 + ab4)/2 N
Combination (2 + 3)
(AB2 + AB3)/2 N
(A2 + A3)/2 N
(B2 + B3)/2 N
(ab2 + ab3)/2 N
Combination (2 + 4)
(AB2 + AB4)/2 N
(A2 + A4)/2 N
(B2 + B4)/2 N
(ab2 + ab4)/2 N
Combination (3 + 4)
(AB3 + AB4)/2 N
(A3 + A4)/2 N
(B3 + B4)/2 N
(ab3 + ab4)/2 N
Combination (1 + 2 + 3)
(AB1 + AB2 + AB3)/3 N
(A1 + A2 + A3)/3 N
(B1 + B2 + B3)/3 N
(ab1 + ab2 + ab3)/3 N
Combination (1 + 2 + 4)
(AB1 + AB2 + AB4)/3 N
(A1 + A2 + A4)/3 N
(B1 + B2 + B4)/3 N
(ab1 + ab2 + ab4)/2 N
Combination (1 + 3 + 4)
(AB1 + AB3 + AB4)/3 N
(A1 + A3 + A4)/3 N
(B1 + B3 + B4)/3 N
(ab1 + ab3 + ab4)/3 N
Combination (2 + 3 + 4)
(AB2 + AB3 + AB4)/3 N
(A2 + A3 + A4)/3 N
(B2 + B3 + B4)/3 N
(ab2 + ab3 + ab4)/3 N
Combination (1 + 2 + 3 + 4)
(AB1 + AB2 + AB3 + AB4)/4 N
(A1 + A2 + A3 + A4)/4 N
(B1 + B2 + B3 + B4)/4 N
(ab1 + ab2 + ab3 + ab4)/4 N
Specify the value of dimensions.
For both tables, the number of dimensions will be set at two.
Production of a CA plot.
Calculation of “D (A–B)” refers to the relative distance between point A and point B in the biplot. “a1” and “b1” refer to the abscissa values of point A and point B. “a2” and “b2” refer to the ordinate values of point A and point B.
The relative distance between points displayed in the CA plot will be summarized in a table. As prespecified, the pair of symptom combinations and outcome types with the nearest relative distance will signify the most significant correlation. The characteristic indications for which the patent drug is most effective will be found upon careful screening.
- 6.The value of relative distance between two points in the biplot will be calculated using the following formula.
Adverse events (AEs) and adverse drug reactions (ADRs) will be monitored and reported throughout the study. The incidence of AEs and ADRs will be compared between groups using the χ 2 test with the level of significance set at P < 0.05.
The central drug administrator will be responsible for drug distribution to and reclamation from each center. Drug administrators at each center will hand out test drugs to the patients and reclaim unused drugs. A drug management log book will be kept.
Oracle Clinical Version 4.6 software (Oracle, Redwood Shores, CA, USA) will be used for clinical data management.
TCM syndrome differentiation is the key to understanding the condition or disease and also to choosing the best means of treatment according to traditional Chinese beliefs. It is also held that the efficacy of TCM is most obviously observed in alleviating or improving the characteristic symptoms of patients diagnosed with the corresponding TCM syndrome.
We will conduct a multicenter, randomized, double-blind, double-dummy trial to compare the effectiveness of two Chinese patent drugs, namely, QSYQ and TXL, for angina patients with qi deficiency and blood stasis syndrome.
The study features the grouping of patients with varied treatment needs, the integration of PIO and PRO in efficacy assessment and the use of CA in identifying the most significant correlations between different symptom combinations and treatment outcomes. The ultimate goal of this exploratory study is to establish a methodology for the precise identification of the characteristic indications for which a Chinese patent drug is most effective, and thus differentiation from other drugs used for similar clinical indications in clinical use.
Strengths and limitations
Evaluation and interpretation of the therapeutic effects of Chinese patent medicine from the patient’s own perspective
Highlighting of individualized evaluation of Chinese patent medicine.
Analysis of the curative effects of Chinese patent medicine on single- or multiple-symptom combinations
Visualized display of the correspondence between the drug’s efficacy and a specific indication or group of indications
Comparison of the therapeutic effects of Chinese patent drugs with other drugs used for similar clinical
The proposed methodology needs to be tested for practicality and feasibility in practice.
By comparing the effects of two kinds of Chinese patent medicine on angina symptom improvement, we aim to establish an innovative evaluation methodology that can facilitate the widespread and rational use of Chinese patent drugs by practitioners of TCM and Western medicine.
The funding support of the National Natural Science Foundation of China (grant 81202849) is gratefully acknowledged. We especially wish to acknowledge the cooperation of the research staff. The study is cofunded by the National Natural Science Foundation of China (grant 81202849) and Tianjin training plan for innovative team of university.
- ZHENG XY: Clinical guideline of new drugs for traditional Chinese medicine. 2002, Beijing: Medicine Science and Technology Press of ChinaGoogle Scholar
- [No authors listed]: Comparative effectiveness: its origin, evolution, and influence on health care. J Oncol Pract. 2009, 5: 80-82.View ArticleGoogle Scholar
- Congress of the United States, Congressional Budget Office: Research on the Comparative Effectiveness of Medical Treatments. 2007, Washington, DC: Congressional Budget Office: Issues and Options for an Expanded Federal Role (Publication No. 2975), Available at http://www.cbo.gov/sites/default/files/cbofiles/ftpdocs/88xx/doc8891/12-18-comparativeeffectiveness.pdf (accessed 7 March 2014)Google Scholar
- National Information Center on Health Services Research and Health Care Technology (NICHSR), National Library of Medicine, National Institutes of Health: Comparative Effectiveness Research (CER) [website]. 2014, Bethesda, MD: NICHSR, Available at http://www.nlm.nih.gov/hsrinfo/cer.html [accessed 7 March 2014]Google Scholar
- Patrick DL, Burke LB, Powers JH, Scott JA, Rock EP, Dawisha S, O’Neill R, Kennedy DL: Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value Health. 2007, 10 (Suppl 2): S125-S137.View ArticlePubMedGoogle Scholar
- Cao HB, Shang HC: [Connotations and extensions of comparative effectiveness research and feasibility analysis for clinical evaluation in traditional Chinese medicine] [in Chinese]. Zhong Xi Yi Jie He Xue Bao. 2012, 10: 369-374. 10.3736/jcim20120403.View ArticlePubMedGoogle Scholar
- Varni JW, Seid M, Kurtin PS: PedsQL 4.0: reliability and validity of the Pediatric Quality of Life Inventory version 4.0 generic core scales in healthy and patient populations. Med Care. 2001, 39: 800-812. 10.1097/00005650-200108000-00006.View ArticlePubMedGoogle Scholar
- de Vet HC, Terwee CB, Ostelo RW, Beckerman H, Knol DL, Bouter LM: Minimal changes in health status questionnaires: distinction between minimally detectable change and minimally important change. Health Qual Life Outcomes. 2006, 4: 54-10.1186/1477-7525-4-54.View ArticlePubMedPubMed CentralGoogle Scholar
- Crosby RD, Kolotkin RL, Williams GR: An integrated method to determine meaningful changes in health-related quality of life. J Clin Epidemiol. 2004, 57: 1153-1160. 10.1016/j.jclinepi.2004.04.004.View ArticlePubMedGoogle Scholar
- Hammer O, Harper DAT, Ryan PD: PAST: paleontological statistics software package for education and data analysis. Palaeontologia Electronica. 2001, 4 (1): Available at http://palaeo-electronica.org/2001_1/past/issue1_01.htm (accessed 11 March 2014)Google Scholar
- Landorf KB, Radford JA, Hudson S: Minimal important difference (MID) of two commonly used outcome measures for foot problems. J Foot Ankle Res. 2010, 3: 7-10.1186/1757-1146-3-7.View ArticlePubMedPubMed CentralGoogle Scholar
- Wyrwich KW, Tierney WM, Wolinsky FD: Further evidence supporting an SEM- based criterion for identifying meaningful intra-individual changes in health-related quality of life. J Clin Epidemiol. 1999, 52: 861-873. 10.1016/S0895-4356(99)00071-2.View ArticlePubMedGoogle Scholar
- Schmidt FL, Le H, Ilies R: Beyond α: an empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychol Methods. 2003, 8: 206-224.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.