Quality assessment of reporting of randomization, allocation concealment, and blinding in traditional chinese medicine RCTs: A review of 3159 RCTs identified from 260 systematic reviews

Background Randomized controlled trials (RCTs) which are of poor quality tend to exaggerate the effect estimate and lead to wrong or misleading conclusions. The aim of this study is to assess the quality of randomization methods, allocation concealment and blinding within traditional Chinese medicine (TCM) RCTs, discuss issues identified for current TCM RCTs, and provide suggestions for quality improvement. Methods We searched Chinese Biomedical Database (CBM, 1978 to July 31, 2009) and the Cochrane Library (Issue 2, 2009) to collect TCM systematic reviews and meta-analyses according to inclusion/exclusion criteria, from which RCTs could be identified. The quality assessment involved whether the randomization methods, allocation concealment and blinding were adequate or not based the study reported. Stratified analyses were conducted of different types of diseases published in different journals (both Chinese and foreign) using different interventions. SPSS 15.0 software was used for statistic analyses. Results A total of 3159 RCTs were included, of which 2580 were published in Chinese journals and 579 in foreign journals. There were 381 (12%) RCTs which used adequate randomization methods; 207 (7%) RCTs which used adequate allocation concealment and 601 (19%) which used adequate blinding; there were 130 (4%) RCTs which both used adequate randomization methods and allocation concealment; and there were only 100 (3%) RCTs which used adequate randomization methods, allocation concealment, as well as blinding. In the RCTs published in foreign journals, the adequate randomization methods, allocation concealment and blinding accounted for a relatively large proportion (25%, 26%, and 60%, respectively) and increased with years, while in the RCTs published in Chinese journals, only the adequate randomization methods improved over time. The quality of non-drug intervention (chiefly acupuncture) RCTs was higher than that of drug intervention RCTs. In drug intervention, the quality of listed drugs is higher than the others. The quality of all included RCTs of all types of diseases was generally poor and no studies that were large in size and of high quality were found. Conclusion The quality of the current TCM RCTs as judged by their publications is generally poor, especially those published in Chinese journals. In future, researchers of TCM RCTs should attach more importance to experimental design and methodological quality, receive relevant training, and improve reporting quality using the Consolidated Standards of Reporting Trials (CONSORT) statement, so as to improve the quality of TCM clinical research and ensure truth and reliability of conclusions.


Background
Traditional Chinese medicine (TCM) is centuries old and has developed a unique system of diagnosis and treatment of disease in the practice [1]. TCM is not only an important part of Chinese health but also is accepted in many parts of the world thus offering a complementary or alternative form of health care. International medicine has been increasingly interested in TCM [2][3][4]. Randomized controlled trials (RCTs) and systematic reviews are commonly regarded first-class evidence in judging the treatment effect of interventions [5,6]. Systematic reviews are based on RCTs, whose deficiency may yield unreliability of the conclusions [7,8], therefore, the quality of included RCTs is of obvious relevance to the interpretability and reliability of the conclusions. Research shows that the RCTs which are of poor quality tend to exaggerate the effect estimates and lead to misleading conclusions [9][10][11][12][13][14][15][16]. The aim of this study is to assess the quality of randomization methods, allocation concealment and blinding of TCM RCTs based on study reports, discuss issues identified for current TCM RCTs, and provide suggestions for quality improvement.

Literature selection
The following databases were searched: Chinese Biomedical Database (CBM, 1978 to July 31, 2009)  All identified RCTs, published in Chinese journals and The Cochrane Library, included in systematic reviews and meta-analyses of TCM (including Chinese herbal medicine, Chinese medicine, Chinese standardized remedies, Chinese medicine preparations and Chinese medicine extracts), integrated TCM and Biomedicine, as well as acupuncture and massage (including acupuncture and electro-acupuncture), were included. Repeatedly-published articles, translated articles, non-full text articles and articles without mentioning sources of RCTs were excluded.

Quality assessment
The quality assessment in this study was performed according to three major procedures in methodological part of the Cochrane Handbook: ① Whether the randomization method was adequate (such as referring to a randomized number table, using a computer random number generator, coin tossing and throwing dice, etc.) or inadequate (such as sequence generated by odd or even date of birth, sequence generated by some rule based on date of admission or hospital record number, etc.). ② Whether allocation concealment was adequate (such as central allocation, sequentially numbered drug containers of identical appearance, sequentially numbered, opaque, sealed envelopes, etc) or inadequate (such as using an open random allocation schedule, alternation or rotation, date of birth, case record number, etc.). ③ Whether blinding was adequate (such as no blinding, but the review authors judge that the outcome and the outcome measurement are not likely to be influenced by lack of blinding; blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken; either participants or some key study personnel were not blinded, but outcome assessment was blinded and the non-blinding of others unlikely to introduce bias) or inadequate (such as no blinding or incomplete blinding, and the outcome or outcome measurement is likely to be influenced by lack of blinding; blinding of key study participants and personnel attempted, but likely that the blinding could have been broken, either participants or some key study personnel were not blinded, and the non-blinding of others likely to introduce bias) [17]. Assessment was based on trial publication only. We did not contact trial authors for clarification.

Stratified analysis
Included RCTs were published in Chinese journals or foreign journals.
Medicine and non-medicine interventions were divided with the former including listed medicines (Chinese standardized remedies, Chinese medicine extracts and single herbs), individualized prescription, mixture (listed medicines and individualized prescription) and unclear (the intervention was not reported).
Types of Diseases: classified according to the 10 th version of the international disease classification (ICD-10) [18].

Data extraction
The researchers used EndNote software to manage titles and abstracts of citations obtained through database search. They(J He, and L Du) read titles and abstracts for primary screening based on the inclusion criteria, and then acquired the full text to read for further screening, before finally deciding whether they would be included or not. Four researchers extracted data, using a pro-forma, and then input the extracted materials into Epidata software. Two other researchers checked the data. Any differences were settled through discussion.
Data analysis SPSS 15.0 was used for descriptive analyses and c 2 test.

Results
A total of 2041 articles were retrieved (1790 from CBM and 251 from The Cochrane Library). After the primary screening, there were 1781 articles were excluded and 260 articles (173 from CBM and 87 from The Cochrane Library) were included. A total of 3159 RCTs were identified from the 260 systematic reviews/meta-analyses, of which 2580 were published in Chinese journals and 579 in foreign journals ( Figure 1).
RCTs published in foreign journals were better than those published in Chinese journals, and the difference was significant (P = 0.000) ( Table 1).

Changes of RCTs reported quality with time
All 3159 RCTs were published between 1965 and 2008 ( Figure 2). The proportion of adequate randomization methods of RCTs increased year by year, but there was no clear increase in allocation concealment and blinding, and the allocation concealment accounted for a very small proportion ( Figure 3). In terms of Chinese journals, the proportion of adequate randomization methods of RCTs between 1993 and 2007 increased sharply, but there was no clear increase in the allocation concealment and blinding, and the allocation concealment just occupied a very small proportion ( Figure 4). By comparison, there was a large rise in the proportion of adequate randomization methods, allocation concealment and blinding of the RCTs published in foreign journals from 1993 through 2007 ( Figure 5). In order to avoid bias caused by chance, RCTs published between 1965 and 1992, as well as in 2008, which number of the included RCTs was very small, were excluded from this analysis.

Reported quality of RCTs of different interventions
Among 3159 included RCTs, 2546 (81%) were drug interventions and 613 (19%) were non-drug interventions. In the non-drug group, the proportion of adequate randomization methods, allocation concealment and blinding were 21%, 19%, and 38%, respectively, which were larger than those in drug intervention, and the difference were significant (P = 0.000) ( Table 2). In different drug intervention, the proportion of adequate randomization of Chinese standardized remedies RCTs were the largest (13%), the proportion of adequate allocation concealment of Chinese medicine extracts RCTs were the largest (9%), and the proportion of adequate blinding of single herbs RCTs were the largest (40%).
In different non-drug intervention, the proportion of adequate randomization and blinding (22% and 41%, respectively) of acupuncture RCTs were larger, and the proportion of adequate allocation concealment (19%) of acupuncture RCTs was the same with massage RCTs.
In terms of drug intervention RCTs published in foreign journals, except that the proportion of adequate randomization methods of mixed RCTs (including listed medicine and individualized prescription) and unclear RCTs were smaller than those of RCTs published in Chinese journals, other proportion of adequate randomization methods, allocation concealment, and blinding were all larger than those of RCTs published in Chinese journals.
In terms of non-drug RCTs published in foreign journals, the proportion of adequate randomization methods, allocation concealment and blinding were all larger than those of RCTs published in Chinese journals.

Reported quality of RCTs of different types of diseases
The included 3159 RCTs were classified according to the 10th version of the international disease classification (ICD-10), involving 16 types of diseases and lacking 5 types (Table 3). Quantities and qualities of all types of diseases had imbalanced distribution. Circulatory system RCTs had the largest number but were of low quality.

Discussion
Many studies have showed that RCTs not using randomization, allocation concealment or blinding exaggerate estimates of effect to various extents. Compared with the RCTs using blinding, the RCTs not using blinding yield 17% larger estimates of treatment effects and in trials with subjective outcomes, effect estimates are exaggerated by 25%. Compared with the RCTs using adequate allocation concealment, RCTs using unclear or inadequate concealment of allocation exaggerate estimates of effect by 30%-41% [10][11][12][13][14]. These showed that compared with other "flaws", unclear or inadequate allocation concealment will cause a larger bias, which highlights the importance of allocation concealment. This study indicates that the adequate allocation concealment takes up the smallest proportion (7%) of the three assessed aspects. Although the adequate randomization methods accounted for a larger proportion (12%) than allocation concealment, there are also some investigations which showed that only 6.8% of the RCTs published in Chinese journals were deemed authentic randomized trials [19]. So the quality of the TCM RCTs in this study may be overstated.
Compared with those published in Chinese journals, the TCM RCTs published in foreign journals are of higher quality. In terms of changes over time, the proportion of reports of adequate randomization has increased year on year, suggesting that more attention has been paid to the randomization methods of Chinese TCM RCTs. However, the proportion of adequate allocation concealment and blinding are small and has remained at approximately the same level. By contrast, the proportion of adequate randomization methods, allocation concealment and blinding of the TCM RCTs published in foreign journals has increased overtime.       Diseases of the ear and mastoid process 4  9 11 0(0) 0(0) 0(0) 0(0) 0(0) 0(0) 0(0) 1(11) 1 (9) Factors influencing health status and contact with health services This suggests that Chinese journals are failing to improve the standards of published RCTs.
Acupuncture, thousands of years of history in China, is a traditional medicine recognized by the World Health Organisation and has become a component of health care in many countries [20]. Since late 1990s, numbers of acupuncture clinical trials have grown dramatically with the increase of research funds in western countries [21].
The results of this study shows that in the light of interventions of TCM RCTs, the quality of non-drug intervention is evidently higher than that of drug intervention, and acupuncture is the largest group. In drug intervention, the quality of RCTs of Chinese standardized remedies, Chinese medicine extracts and single herbs is comparatively high while that of individualized prescription, mixtures (including listed medicine and individualized prescription) and those where the intervention is unclear is low. Individualized prescription which is common in TCM is influenced by a doctor's experience and training. There are no agreed formulae for drug ingredients, no standard sources of drugs, or preparation methods. In addition, doses of Chinese herbs are not standard. Listed medicine and single herbs can overcome the above shortcomings of individualized prescription, so the quality of RCTs of these kinds is higher and clinical research develops in a more extensive way.
With the development of TCM, current TCM RCTs involve wide study fields and many types of diseases. This study was aimed at determining which medical specialties produce with high-quality RCTs, but these were not identified.
The quality assessment in this study was based on how the original study was reported, so it was affected by the quality of reporting. There also may be some other factors not in the power of the reseachers such as journal word length etc. Some studies have showed that use of the Consolidated Standards of Reporting Trials (CON-SORT) statement is associated with improvements in the reporting quality of RCTs, but few researchers and journals adopt it [22][23][24][25][26][27][28][29][30]. The CONSORT extensions for acupuncture (the Standards for Reporting Interventions in Clinical Trials of Acupuncuture, STRICTA) and herbal medicine were designed to improve the completeness and transparency of reporting of interventions in controlled trials of acupunture and herbal medicine [31,32].

Conclusion
The quality of the current TCM RCTs as judged by their publications is generally poor, especially those published in Chinese journals. In future, researchers should attach more importance to design and methodological quality, especially to allocation concealment. The quality of RCTs can also be improved through promoting participation of methodologists and statisticians, enhancing international cooperation, and adopting guidelines such as the CON-SORT statement and its extensions for acupuncture and herbal medicine. Meanwhile, medical journals should also be more discerning in what they accept for publication, so as to improve the quality of TCM clinical research and ensure truth and reliability of conclusions.