This article has Open Peer Review reports available.
Subversion of allocation concealment in a randomised controlled trial: a historical case study
© The Author(s). 2017
Received: 11 January 2017
Accepted: 12 April 2017
Published: 2 May 2017
If the randomisation process within a trial is subverted, this can lead to selection bias that may invalidate the trial’s result. To avoid this problem, it is recommended that some form of concealment should be put into place. Despite ongoing anecdotal concerns about their susceptibility to subversion, a surprising number of trials (over 10%) still use sealed opaque envelopes as the randomisation method of choice. This is likely due in part to the paucity of empirical data quantifying the potential effects of subversion. In this study we report a historical before and after study that compares the use of the sealed envelope method with a more secure centralised telephone allocation approach in order to provide such empirical evidence of the effects of subversion.
This was an opportunistic before and after study set within a multi-centre surgical trial, which involved 654 patients from 28 clinicians from 23 centres in the UK and Ireland. Two methods of randomly allocating subjects to alternative treatments were adopted: (a) a sealed envelope system administered locally, and (b) a centralised telephone system administered by the trial co-ordination centre. Key prognostic variables were compared between randomisation methods: (a) age at trial entry, a key prognostic factor in the study, and (b) the order in which ‘randomisation envelopes’ were matched to subjects.
The median age of patients allocated to the experimental group with the sealed envelope system, was significantly lower both overall (59 vs 63 years, p < 0.01) and in particular for three clinicians (57 vs 72, p < 0.01; 33 vs 69, p < 0.001; 47 vs 72, p = 0.03). No differences in median age were found between the allocation groups for the centralised system.
Due to inadequate allocation concealment with the sealed envelope system, the randomisation process was corrupted for patients recruited from three clinicians. Centralised randomisation ensures that treatment allocation is not only secure but seen to be secure. Where this proves to be impossible, allocation should at least be performed by an independent third party. Unless it is an absolute requirement, the use of sealed envelopes should be discontinued forthwith.
It is widely accepted that randomised controlled trials are the method of choice for the evaluation of new clinical treatments. They differ from other prospective designs principally in the way in which the groups for comparison are generated. True random allocation leads to groups at trial entry that differ only by chance in potentially confounding prognostic factors, both known and unrecognised [1, 2].
Failure to secure random allocation can lead to selection bias that invalidates a trial’s results [3–6]. Knowledge of the next allocation may bias the decision to recruit a person or lead to participants with a better prognosis being assigned to one group rather than another . Eliminating selection bias at trial entry requires two things: first, unpredictable sequencing of allocations, and second, secure allocation concealment, such that ‘irretrievable’ trial entry occurs before the assigned treatment is known .
If the allocation is deciphered this can lead to the possibility of subverting the randomisation, which introduces selection bias. Quantifying the frequency of subversion is difficult as it is a form of academic and clinical misconduct. Nevertheless, Berger  has collected over 30 examples where there was concern that the allocation schedule had been subverted and a survey of clinicians who have had experience of recruiting participants into clinical trials found that 16% of them kept a log of previous allocations in order to help them predict future ones . To avoid the problem of subverting the allocations it is recommended that some form of concealment should be put into place to reduce the possibility of allocation subversion.
The use of sealed opaque envelopes is still often used in randomised trials, with a recent review finding that 11% of trials published in 2016 in four major medical journals used sealed envelopes . This is despite Schulz  reporting, more than 20 years ago, instances where this approach can be manipulated by either opening envelopes in advance or through transillumination. A methodological study  that compared the significance levels of trials that mainly used sealed envelopes with those who used more secure, third-party, randomisation systems found that trials using less secure procedures had significantly smaller p values than those using a more secure allocation method. However, it was not possible in that review to identify individual instances of misallocation and the reporting of actual cases of misallocation is rare. In this present study we report a before and after study that compared the use of the sealed-envelope method with a more secure centralised telephone-allocation approach. Although this study took place in the mid-1990s we believe it still has relevance today, given the continuing prevalence of poorly concealed allocation in current clinical trials .
The trial was a multi-centre randomised controlled trial in surgery in which age was judged to be a key prognostic variable. Age distribution was first described in an initial cohort of 327 participants randomised using a sealed envelope system , and then compared with a similar-sized cohort recruited after the introduction of a central telephone randomisation system.
The sealed envelope system
The generation of the random sequence of envelope allocations was performed using a matched block method stratified by clinician. Allocation was assigned to each block using simple randomisation; this block sequence was then repeated swapping the order of the two treatments, giving an equal number of patients in the two treatment groups over the matched block. To ensure that the sequence could not be anticipated the block size was selected randomly to be 5, 10 or 15. The random sequence for these blocks was generated using a random number generator within the statistical analysis package SPSS, the seed calculated by multiplying the seconds and minutes portion of the computer’s internal clock. This generated a pseudo-random distribution in the range 0 to 1. Values <0.5 were allocated the control treatment, and those ≥0.5 to the experimental treatment.
These allocations were printed onto cards which were then sealed in sequentially numbered envelopes. This process was performed for each clinician on joining the collaborative trial group. At randomisation, the envelope number and patient identifying details were recorded on a form which was then returned to the trial administration centre to confirm recruitment.
The centralised telephone randomisation system
The centralised system used a toll-free, dedicated telephone line. To recruit a patient the clinician now telephoned the trial administration centre giving details of the prognostic factors. Details were entered directly into a customised database package to generate the allocation. In addition to stratification by clinician, the allocation was minimised on five prognostic factors (age, sex, and three clinical factors - site of hernia, type of hernia and presence of recurrent hernia). These include age, which was split into three groups; the other factors were dichotomised.
For each randomisation system, the age distribution in the trial groups was compared first for all patients, and then for those recruited by the five clinicians who had recruited more than 25 people in the first time period; all other clinicians’ patients were combined to give a sixth group. Median ages were compared using the Mann-Whitney U test. Median differences in age between the groups (with 95% confidence intervals) were then generated. Apparent imbalances in the age distributions were further investigated by plotting the recruitment sequence against the envelope number, for those cases where this information was available.
Age distribution in trial groups: (a) using the sealed envelope method and (b) using central telephone randomization
(a) Sealed envelope method
(b) Central telephone randomisation
This study provides one of the very few empirical examples of subversion of randomisation and the consequences of such subversion for trial characteristics. In this study we detected the subversion problem by testing for age equivalence between the randomised groups. Although baseline testing is generally not promoted (e.g. Altman and Dore 1990 ) because in a properly randomised trial any differences in baseline are most likely due to chance, Berger  argues strongly that baseline testing still has a role in order to detect possible allocation subversion. He argues that because subversion is likely to be hidden from the investigators baseline testing is legitimate to reassure the investigators and readers that subversion is unlikely. The level of subversion is, however, required to be relatively substantial as in our example for it to effectively identify a subversion problem. A complementary approach, if trials use block randomisation, has been suggested to detect suspicious imbalances .
Although there is indirect and anecdotal evidence of important bias introduced by faulty randomisation procedures, investigators are understandably reluctant to describe such problems. The new surgical procedure that we were helping to evaluate was likely to need a longer period of general anaesthesia and so likely to be considered less suitable for older, frailer patients. We were therefore concerned at the apparent imbalance in age amongst the cohort recruited during the start-up phase of this trial; further investigations indicated that the system had been corrupted amongst patients recruited by three clinicians (their results were not used in the final trial analysis). The problem was restricted to the period when a sealed envelope method was used and was not seen after central telephone randomisation had been instituted.
The problem was one of inadequate concealment rather than with the generation of the sequence. There are a number of ways in which the allocation can be prematurely revealed before formal trial entry when sealed envelopes are used. More than one envelope can be opened until a desired allocation is found, the order in which patients are recruited may be manipulated such that a particular patient is allocated a chosen treatment, or the allocation may be discerned without opening the envelope (such as by transillumination) with the (biased) decision about trial entry dependent on the allocation. The plots of recruitment sequence against envelope number were used to explore these possibilities; these should show a straight line if the process is carried out correctly, although even this does not rule out biased entry. We did find examples of skipping envelopes and order changes amongst the participants recruited by the three clinicians. Order changes are unlikely to be an important factor; however, because in only two of the five times it occurred was the skipped allocation different from the card used.
Sequentially numbered sealed envelopes remain a widely used method for allocation in intended randomised trials, and occasionally are the only practical method of undertaking randomisation. For example, in a sample of ‘open’ trials published in major medical journals in 2016, about 11% were using sealed envelopes to conceal allocation  Clark et al. Our experience, however, which supports the previous anecdotal evidence reported by Berger  and Brown et al.  illustrates why the use of envelopes is more susceptible to corruption - often through well-intentioned, human ingenuity rather than other approaches - and why it is a less than ideal method of concealment. It also underlines the recommendation in the Consolidated Standards of Reporting Trials (CONSORT) statement on the reporting of randomised controlled trials that a full description be given of the actual methods used to generate and conceal trial allocations .
The mistake not to generate the allocation sequence from the executor of the treatment assigned is clearly illustrated. If a sealed envelope or other locally based method of random allocation has to be used a clear message from our study is that this should be the responsibility of an independent third party who has no direct involvement in the trial or in patient care. A centralised system such as the one in place for the main recruitment period of this trial is preferable.
In this trial the method of allocation was changed from blocked randomisation to minimisation, and allocation was minimised by centre. Using any form of restriction to the randomisation can, however, increase the risk of predictability and possible subversion through scheduling . Indeed, a case study has been reported where subversion occurred despite using a central telephone randomisation service . This was probably caused due to use of a predictable sequence of blocks in the randomisation.
In summary, this study provides further empirical evidence that using sealed opaque sequentially numbered envelopes is an inadequate method of concealing treatment allocation and we recommend that, unless it is an absolute requirement, their use should be discontinued forthwith. Trialists should use other means to prevent allocation subversion.
The Health Services Research Unit is core funded by the Chief Scientist Office of the Scottish Health and Social Care Directorates. The views expressed are those of the authors alone.
Availability of data and materials
The data are available from the corresponding author on reasonable request.
ADMK and AMG wrote the primary drafts of this manuscript; DJT and MKC commented on, added to, and revised subsequent drafts. All authors read and approved the final manuscript.
This paper is dedicated to the memory of our friend and colleague, Adrian Grant, who died on 16 August 2015.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Not applicable - this is a secondary analysis of anonymised data from an original study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised studies. BMJ. 1998;317:1185–90.View ArticlePubMedPubMed CentralGoogle Scholar
- Martyn C. Not quite as random as I pretended. Lancet. 1996;347:70.View ArticlePubMedGoogle Scholar
- Chalmers TC, et al. Bias in treatment assignment in controlled clinical trials. NEJM. 1983;309:1358–61.View ArticlePubMedGoogle Scholar
- Altman, DG, Dore CJ. Randomisation and baseline comparisons in clinical trials. Lancet. 1990;335:149–53.Google Scholar
- Altman DG. Randomisation: essential for reducing bias. BMJ. 1991;302:1482.View ArticleGoogle Scholar
- Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408–12.View ArticlePubMedGoogle Scholar
- Schulz KF. Subverting randomisation in controlled trials. JAMA. 1995;274:1456–8.View ArticlePubMedGoogle Scholar
- Berger VW. Selection bias and covariate imbalances in randomized clinical trials. Chichester: John Wiley & Sons Ltd; 2005.View ArticleGoogle Scholar
- Brown S, Thorpe H, Hawkins K, Brown J. Minimization – reducing predictability for multi-centre trials whilst retaining balance with centre. Stat Med. 2005;24:3715–27.View ArticlePubMedGoogle Scholar
- Clark L, Fairhurst C, Torgerson DJ. Allocation concealment in randomised controlled trials: could do better. BMJ. 2016;315:i5663.View ArticleGoogle Scholar
- Hewitt C, Hahn S, Torgerson DJ, Watson J, Bland JM. Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ. 2005;330:1057–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Pocock S. Clinical trials: A practical approach. Chichester: Wiley; 1983.Google Scholar
- Berger VW, Exner DV. Detecting selection bias in randomized clinical trials. Control Clin Trials. 1999;20:319–27.View ArticlePubMedGoogle Scholar
- Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, et al. CONSORT 2010 Explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;240:c869.View ArticleGoogle Scholar
- Turner J, Russell D, Russell I, Gemmell LW, France B, Bride M, Edwards AE. Baseline imbalance in a randomised trial: a cautionary tale. Care Critically Ill. 2006;22(2):1–4.Google Scholar