An assessment of the quality of randomised controlled trials conducted in China
© Zhang et al; licensee BioMed Central Ltd. 2008
Received: 13 February 2007
Accepted: 24 April 2008
Published: 24 April 2008
Despite the rapid increase in research in China, little is known about the quality of clinical trials conducted there.
A systematic review and critical appraisal of randomised controlled trials (RCTs) conducted in China and published in 2004 was undertaken to describe their characteristics, assess the quality of their reporting, and where possible, the quality of their conduct. Randomised controlled trials in all disease areas and types of interventions, which took place in China and included Chinese citizens were identified using PubMed and hand searching the Journal Series of the Chinese Medical Association. Quality was assessed against a subset of criteria adapted from the CONSORT statement.
Three hundred and seven RCTs were included. One hundred and ninety-nine (64.8%) failed to report methods of randomization and 254 (82.4%) did not mention blinding of either participants or investigators. Reporting of baseline characteristics, primary outcome and length of follow-up was inadequate in a substantial proportion of studies. Fewer than 11% of RCTs mentioned ethical approval and only 18.0% adequately discussed informed consent. However, dropout rates were very favourable with nearly 44% of trials reporting a zero dropout rate.
Reporting of RCTs in China requires substantial improvement to meet the targets of the CONSORT statement. The conduct of Chinese RCTs cannot be directly inferred from the standard of reporting; however without good reporting the methods of the trials cannot be clearly ascertained.
Randomised controlled trials (RCTs) are considered the 'gold standard' for assessing the effectiveness of pharmacological and other interventions in the field of medicine [1–4]. They are widely accepted as the best research design because they distribute both known and unknown prognostic factors between treatment groups by the play of chance [5–8] thereby minimizing the possibility that any treatment effect is due to bias or confounding, and providing the basis for valid statistical comparison .
However, RCTs vary in their methodological rigour, and it is well known that poor quality studies tend to produce systematically different results from larger, better quality studies, often erroneously showing larger treatment effects [9, 10]. The conduct of studies cannot be assessed without clear reporting. Many medical journals now expect authors to adhere to internationally agreed standards of reporting thus allowing the reader to assess the conduct of each trial ; this has assisted in raising the standards of trial reporting in developed countries.
China is a developing country with the biggest population in the world. Research in China has been rapidly gaining momentum, but as yet there is no systematic evaluation of the current standard of trials conducted there. Evaluations of the quality of Chinese RCTs have been restricted to selected journals or fields, and often a limited list of quality indicators [13–19]. In one example, a recently published systematic review of the effectiveness of hyperbaric oxygen using Chinese RCTs found that the published papers reported inadequate information and were generally of poor quality .
We present a critical evaluation of randomised controlled trials conducted in China and published in 2004. Our aim was to describe their general characteristics, evaluate the quality of their reporting, and evaluate their conduct where adequately reported.
The study was carried out according to a pre-defined protocol.
Using the PubMed database. PubMed includes MEDLINE and OLDMEDLINE  but papers published in many non-English Journals are not listed. We searched PubMed for Chinese randomised controlled trials published in 2004 using the textwords 'chin*' and the PubMed filter for randomised controlled trials.
Since many of the main medical journals in China are not indexed in PubMed, or in any electronic database, we also accessed the online versions of each journal in the Journal Series of the Chinese Medical Association. The Journal Series of the Chinese Medical Association includes 71 journals, which comprise the main core medical journals in Mainland China and additionally the Chinese version of the British Medical Journal.
For both sources, reference lists of included studies were checked. No language or other limitations were imposed. Chinese text was translated into English by two authors fluent in Chinese (DZ and PY). Titles were initially scanned for relevance and abstracts read if titles were unclear. The full text of papers with no abstract was viewed and checked for eligibility.
Inclusion and exclusion criteria
We included any papers reporting randomised controlled trials on all disease groups and all types of interventions, which were published in 2004, took place in China and included Chinese citizens. We excluded reports that did not include any participants from Mainland China. We excluded papers from Hong Kong and Taiwan where research and clinical practice are different from those in the Mainland.
Assessment of quality
Indicators used to describe and evaluate included randomised controlled trials
Chinese or English
Nationality of authors
Chinese, international or collaboration
Choice of comparator interventions
Placebo/alternative treatment/no treatment
Size of trial
Number of participants
Ethical committee approval
Informed consent from participants
Quality of reporting: CONSORT indicators
How was sample size determined?
Was the trial randomised?
What method was used to implement the random allocation sequence?
Whether or not patients and/or investigators were blinded to group assignment
Were the baseline demographic and clinical characteristics of each group reported
Did they report which outcome was designated as the primary outcome?
Length of follow-up
Loss to follow-up
Were confidence intervals or p values reported to indicate precision?
Data extraction and analysis
One reviewer extracted data from all included papers. A second reviewer independently checked a random sample of 26% of the papers. Discrepancies were resolved where possible by discussion, and the sample results compared with the full results using Kappa scores. Data on the quality of the included papers were presented in tabular format accompanied by a critical description.
Reasons for excluding papers
Reason for exclusion
Number of papers
Brief report given only
Case control study
Phase II trial (no control arm)
Randomisation not mentioned in full text
Allocated by patients' choice
Allocated by patients' economic status
Agreement between reviewers
Agreement between the reviewers was good with a kappa score of greater than 0.7 for the main indicators (funding source, disease area, choice of comparator interventions, ethical committee approval, informed consent from participants, sample size, randomisation, allocation concealment, blinding, baseline characteristics, primary outcomes, loss to follow-up, length of follow-up, statistical reporting).
Characteristics of the included trials
1. Publication language
Of the 307 included RCT papers, 259 (84%) were written in Chinese. The remaining 48 papers were published in English.
2. Nationality of authors
292 (95%) included papers were written by authors based in Chinese research institutes; the remaining papers were collaborations between Chinese and foreign researchers. There were no trials conducted or reported only by foreign researchers.
3. Funding source
Of the 307 papers, 232 (75.6%) did not report their sources of funding. Funding was from provincial/municipal and national sources in 38 trials (12.4%) and 23 trials (7.2%) respectively. Foreign pharmaceutical companies, universities, international research agencies and the military financed five or fewer trials each.
4. Disease area
Disease area of included trials
Number of papers
Digestive system diseases
Circulatory system diseases
Urogenital system diseases
Nervous system diseases
Motor system diseases
Respiratory system diseases
Endocrine system diseases
Immune system diseases
5. Choice of comparator interventions
Thirty-nine (12.7%) of the included trials compared active treatment with a placebo group. Three of these were randomised controlled crossover trials where participants were blinded to the order of drug taken. In 179 (58.3%) trials the new treatment being tested was compared with an alternative named treatment, and in 79 (25.7%) trials the new treatment was compared with a treatment described as the "standard", but with no specific details. Seven additional studies included a control group receiving no treatment: three of them were health education and promotion projects, two of them were health rehabilitation and two drug trials. A further three papers described trials with three different treatment arms: active treatment, standard treatment and no treatment.
6. Size of the trials
The number of participants in each included trial ranged from 3 to 19200, with a median of 78.
7&8. Ethical issues (Ethics Committee Approval and Informed Consent)
Only 33 (10.8%) of the included Chinese trials reported approval by an ethics committee. The majority of the study reports (249 or 81%) did not provide any information about informed consent although 54 (17.6%) of papers stated that the participants did give consent. The remaining four studies stated that participants were included in the trial of their own free will.
Quality of reporting
9. Sample size
Only nine (2.9%) of the 307 papers mentioned sample size calculation.
10&11. Methods of randomisation & allocation concealment
Methods of randomisation
Number of papers (%)
Random number sheet
Random allocation card
Toss of a coin
254 (82.7%) papers provided no information about blinding of either participants or investigators. In 39 (12.7%) trials, both the investigators and participants were blinded. In 9 (2.9%) trials the participants were not blind, and in 5 (1.6%) the investigators were not blinded to the participants' treatments.
13. Reporting of baseline characteristics
Eighty-nine (29%) of the included papers fully reported the baseline characteristics of the participants in a separate table. Two hundred and nine (67%) of the papers described baseline characteristics using either text or mixed tables, which also included results. In 9 papers (2.9%) only age was given in the baseline information, and in two papers no information was given other than a statement that the baseline characteristics matched in both arms.
14. Reporting of primary outcomes
Only 11 (3.6%) of the included trials indicated which measure was used as the primary outcome; the remainder merely reported an ordered list of results from which it was not possible to distinguish which outcome was the primary.
15. Length of follow-up
Length of follow up of included RCTs (days)
Days of follow-up
Number of papers (%)
Dropout rate of included trials
Dropout rate (%)
Number of papers
17. Statistical reporting
The majority of the papers (298 trials or 97.1%) conducted t-tests to examine the statistical significance of their results, and presented p values. In only 20 papers (6.5%) did the authors use confidence intervals to describe the uncertainty around their estimates.
Study of trial quality is rare in developing countries, and tends to focus on limited clinical areas [22, 23]. Although among Chinese publications there are a few paperswhich describe trial quality in specific journals or fields [13–19], this is the first systematic study to evaluate the quality of trial conduct and reporting in a sample which is likely to be more representative of Chinese RCTs in general.
Our review revealed that the standard of reporting of trials was generally poor, which concurs with the other published reports on Chinese trials [16–19]. For example, nearly two-thirds failed to report any information on their methods of randomisation, reinforcing previous work [18, 19]. In the remainder there were various methods of random allocation, of which about a quarter reported using a computer-generated method or a random number table, which are the usually acceptable ways of randomisation. None of the trials discussed allocation concealment. If the allocation of the patient is not adequate and fully independent of the enrolling investigator, then this may allow either conscious or unconscious selection of participants into the trial, or into particular arms of the trial, thus introducing selection bias and undermining the randomisation. The internal validity of a randomised controlled trial has been shown to be directly associated with a clear description of appropriate methods of random allocation of participants, and concealment of their allocation .
Over 80% of trials provided no information about blinding of either participants or investigators. This confirms the result observed in a review of RCTs of traditional Chinese medicine. Without blinding the groups may have been treated differently by the investigator and the outcomes not measured objectively, thus creating further assessment bias. Participants aware of their treatment may behave differently or have particular expectations , thus affecting the results.
Interestingly, among the included Chinese studies in this review, over half stated that none of their participants dropped out. This is unusual compared with trials in countries with more established research programmes, where a drop-out rate of below 5% is generally considered a very good result. Over 60% of the trials in this review reported a drop-out rate of less than 5%, and two-thirds less than 10%. The reasons behind these very low rates warrant further investigations.
The reporting of ethical issues was inadequate in the Chinese RCTs. Fewer than 11% of the trials reported having ethical committee approval, although the latter is a legal requirement in China . Also, only a minority of the Chinese studies (17.4%) gave adequate details about informed consent procedures; a few mentioned that participants attended of "their own free will" but the remainder made no mention of consent. However, this level appears better than in a recent review of traditional Chinese medicine trials .
Compared with many published trials in developed countries, the standard of reporting in China is lagging behind, although there are still many fields in Western countries which have inadequate standards of reporting . However, the application of the CONSORT statement has demonstrated benefits in improving reporting  and could be expected to do the same in China.
Although we undertook a thorough search for eligible studies using both PubMed and the Journal Series of the Chinese Medical Association, we may have missed relevant studies not included in the databases. The Journal Series of 71 Chinese journals comprises the core of the Chinese medical journals, but only approximately 20% of the total. RCTs which were not described as such in the abstract would have been excluded; however it is not clear how many such false negatives there would have been. Indeed failure to mention correctly the study design in the abstract is a mark of poor quality.
Reporting of RCTs in China requires substantial improvement to meet the targets of the CONSORT statement. The conduct of Chinese RCTs cannot be directly inferred from the standard of reporting; however without good reporting the methods of the trials cannot be clearly ascertained. Research bodies in China should ensure that the reporting of RCTs is improved to meet internationally agreed standards, thereby allowing the conduct of their studies to be monitored and encouraging high quality standards.
We thank Dr June Jones for her advice and comments on the manuscript.
- Freidman LM, Furberg C, DeMets DL: Fundamentals of clinical trials. 1981, Boston:Wright PSGGoogle Scholar
- Byar DP, Simon RM, Friedewald WT, Schlesselman JJ, DeMets DL, Ellenberg JH, Gail MH, Ware JH: Randomized clinical trials. Perspectives on some recent ideas. N Engl J Med. 1976, 295: 74-80.View ArticlePubMedGoogle Scholar
- Chalmers TC, Sacks HS, Smith H: Historical versus randomized controls for clinical trials. Control Clin Trials. 1980, 1: 177-10.1016/0197-2456(80)90057-4.View ArticleGoogle Scholar
- Tygstrup N, Lachin JM, Juhl E: The randomized clinical trial and therapeutic decisions. 1982, New York: Marcel DekkerGoogle Scholar
- Klimt CR: The conduct and principles of randomized clinical trials. Control Clin Trials. 1981, 1: 283-293. 10.1016/0197-2456(81)90031-3.View ArticlePubMedGoogle Scholar
- Ruthstein DD: The ethical design of human experiments. Daedalus. 1969, 523-541.Google Scholar
- Chalmers TC: The impact of controlled trials on the practice of medicine. Mt Sinai J Med. 1974, 41: 753-759.PubMedGoogle Scholar
- Prescott RJ, Counsell CE, Gillespie WJ, Grant AM, Russell IT, Kiauka S, Colthart IR, Ross S, Shepherd SM, Russell D: Factors that limit the quality, number and progress of randomised controlled trials. Health Technol Assess. 1999, 3: 1-143.PubMedGoogle Scholar
- Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.273.5.408.View ArticlePubMedGoogle Scholar
- Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352: 609-613. 10.1016/S0140-6736(98)01085-X.View ArticlePubMedGoogle Scholar
- Moher D, Schulz KF, Altman DG: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001, 357: 1191-1194. 10.1016/S0140-6736(00)04337-3.View ArticlePubMedGoogle Scholar
- Moher D, Jones A, Lepage L: Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation. JAMA. 2001, 285: 1992-1995. 10.1001/jama.285.15.1992.View ArticlePubMedGoogle Scholar
- He Q, Wang C, Lin M, Xue C: The analysis of recognition of respiratory physician on evidence based medicine. Chinese Journal of Tuberculosis and Respiratory Disease. 2004, 27: 460-462.Google Scholar
- Chen F, Zeng Y, Wei H: Analysis of the quality of clinical trials about therapeutic research using the standard of evidence based medicine. Chinese Journal of Ophthalmology. 2004, 40: 609-613.PubMedGoogle Scholar
- Wang J: Conducting high quality randomized controlled clinical trials. Chinese Journal of Hepatology. 2004, 12: 451-452.PubMedGoogle Scholar
- Liu Z, Xiong T, Meads C: Clinical effectiveness of treatment with hyperbaric oxygen for neonatal hypoxic-ischaemic encephalopathy: systematic review of Chinese literature. BMJ. 2006, 333: 334-376. 10.1136/bmj.333.7563.334.View ArticleGoogle Scholar
- Bian ZX, Moher D, Dagenais S, Li YP, Wu TX, Liu L, Miao JX, Song L, Zhang HM: Improving the quality of randomized controlled trials in Chinese herbal medicine, part IV: applying a revised CONSORT checklist to measure reporting quality. Journal of Chinese Integrative Medicine. 2006, 4 (3): 233-242. 10.3736/jcim20060303.View ArticlePubMedGoogle Scholar
- Peng X, Zhao Y, Liang X, Wu L, Cui S, Guo A, Wang W: Assessing the quality of RCTs on the effect of beta-elemene, one ingredient of a Chinese herb, against malignant tumours. Contemporary Clinical Trials. 2006, 27 (1): 70-82. 10.1016/j.cct.2005.07.002.View ArticlePubMedGoogle Scholar
- Wang G, Mao B, Xiong Z-Y, Fan T, Chen X-D, Wang L, Liu G-J, Liu J, Guo J, Chang J, Wu T-X, Li T-Q: The quality of reporting of randomized controlled trials of traditional Chinese medicine: A survey of 13 randomly selected journals from mainland China. Clin Ther. 2007, 29 (7): 1456-1467. 10.1016/j.clinthera.2007.07.023.View ArticlePubMedGoogle Scholar
- National Center for Biotechnology Information. http://www.ncbi.nlm.nih.gov
- Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282: 1054-1060. 10.1001/jama.282.11.1054.View ArticlePubMedGoogle Scholar
- Yousefi-Nooraie R, Shakiba B, Mortaz-Hejri S: Country development and manuscript selection bias: a review of published studies. BMC Med Res Methodol. 2006, 6: 37-10.1186/1471-2288-6-37.View ArticlePubMedPubMed CentralGoogle Scholar
- Gibson RS, Sazawal S, Peerson JM: Design and quality control issues related to dietary assessment, randomized clinical trials and meta-analysis of field-based studies in developing countries. J Nutr. 2003, 133 (5 Suppl 1): 1569S-73S.PubMedGoogle Scholar
- Jadad AR, Moore A, Carrol D: Assessing the quality of reports of randomized clinical trials: Is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.View ArticlePubMedGoogle Scholar
- Chinese State Food and Drug Administration. Regulation No.13. http://www.sda.gov.cn/cmsweb/webportal/W53384/A61246593.html?searchword=%28%C1%D9%B4%B2%CA%D4%D1%E9%29
- Chan A-W, Altman DG: Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005, 365: 1159-62. 10.1016/S0140-6736(05)71879-1.View ArticlePubMedGoogle Scholar
- Strippoli GFM, Craig JC, Francesco P: The number, quality, and coverage of randomized controlled trials in nephrology. J Am Soc Nephrol. 2004, 15 (2): 411-9. 10.1097/01.ASN.0000100125.21491.46.View ArticlePubMedGoogle Scholar
- Kane RL, Wang J, Garrard J: Reporting in randomized clinical trials improved after adoption of the CONSORT statement. J Clin Epidemiol. 2007, 60: 241-249. 10.1016/j.jclinepi.2006.06.016.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.