Lancet 1996; 348: 596-9

Division of Sexually Transmitted Disease Prevention, National Center for HIV, STD, and TB Prevention, MS-E02, Centers for Disease Control and Prevention, Atlanta, GA 30333, USA (K F Schulz PhD)

Randomisation has two parts
Randomisation and the human spirit
Double-blinding and exclusions after randomisation
Reporting
Conclusions
References

A flurry of activity on the reporting of randomised controlled trials (RCTs) has culminated in the CONSORT guidelines .¹ These recommendations followed the SORT,² Asilomar Working Group,³ and the British Journal of Obstetrics and Gynaecology⁴ guidelines and a widely circulated editorial⁵ on the reporting of RCTs. Why this guideline avalanche? RCTs, the methodological paragon for assessing evidence, have historically been poorly reported in general medical and specialist journals.^6-13 Little seems to have changed over the years. The scientific world worries, of course, that sloppy reporting reflects sloppy methods, and that with sloppy methods come biased results.

With that long history of poor reporting, why rush to institute stricter guidelines now? First, methodologically astute journal editors now constitute a substantial part of the whole--the CONSORT guidelines¹ are a symbol of this trend. Second, we now have empiric evidence that many improperly implemented RCTs are biased: poor reporting was associated with exaggerated estimates of treatment effects.¹⁴ Readers and editors can no longer feel comfortably insulated from bias. RCTs provide the gold standard, but they are also anathema to the human spirit. Human beings, given the opportunity, sometimes subvert the aims of random allocation.  

Randomisation has two parts

The prevention of selection and of confounding biases in trials hinges largely on the adequacy of allocation to the intervention. Randomisation depends on two processes: generation of an unpredictable assignment sequence and concealment of that sequence until allocation occurs. Many medical researchers mistakenly regard the sequence-generating process as randomisation, and overlook the concealment.^15,16 But without adequate concealment, even random, unpredictable assignment sequences can be subverted.^14,17 Knowledge of the next assignment could lead to the exclusion of certain patients based on prognosis because they would have been allocated to the "wrong" group. Moreover, knowledge of the next assignment could lead to some participants being directed to "desired" groups, which can easily be accomplished by delaying a participant's entry into the trial until the next desired allocation appears. Crucially, allocation concealment shields those who admit participants to a trial from knowing the upcoming assignments.^15,16

My colleagues and I assessed the quality of randomisation for 250 controlled trials from 33 meta-analyses, and analysed the associations between those assessments and estimated treatment effects.¹⁴ Trials in which the allocation sequence had been inadequately concealed yielded larger estimates of treatment effects than trials in which authors reported adequate concealment (odds ratio exaggerated, on average, by 30-40%). However, trials without adequate sequence generation led to estimates of treatment effects that were similar to those from trials with adequate generation. Thus the approach to sequence generation has a smaller role overall in the prevention of bias than does the approach to concealment. That observation makes sense--having a randomised (unpredictable) sequence should make little difference without adequate concealment.

However, one of our findings hinted at the importance of a randomised sequence. In limiting an analysis to trials with adequate concealment, those with inadequate sequence generation yielded larger estimates of effects than trials with adequate generation (p=0·07). That seems plausible in that proper randomisation yields unpredictability. Without unpredictability, even given adequate concealment, some deciphering of allocation sequences can occur. Unblinded trials are particularly susceptible because group assignments become known after allocation. If those assignments are predictable (eg, alternation or blocking with a short, fixed block length), a sequence can be discovered from the order of the past assignments. Trialists could then foresee all or some of the upcoming assignments. Thus researchers should ensure adequate generation¹⁸ with adequate concealment in allocation mechanisms.  

Randomisation and the human spirit

Do investigators actually confess the delicate details of subverting the intended purpose of randomisation? Some do¹⁹ but with the sensitivities involved, documentation is rare. However, when doctors responded anonymously to queries during epidemiological workshops, more than half reported deciphering an allocation concealment scheme.15 This should not be interpreted as representing more than half of all the trials, however, because many participants had been involved in multiple trials. Nevertheless, although most published RCTs probably estimate treatment effects reliably, allocation breaches seem to be more than rare. Whilst the stories frequently revealed ingenious efforts, they also frequently reflected naive, questionable judgment. Assignment manipulations by investigators manifest larger underlying conflicts. The scientific community's need for unbiased research clashes with the inherently biased sources of that information--human beings. Trialists theoretically understand the need for unbiased research, but remaining dispassionate may be too difficult once the trial starts. They perhaps "know" what treatment works better so they may want certain patients to benefit or may want the results to show what they believe to be valid.¹⁶

Those who fail to comply with an assignment scheme do not necessarily have sinister motives. Many subversions reflect human curiosity rather than scientific malevolence. For those involved in implementing a trial that has not incorporated proper procedures for sequence generation and allocation concealment, the challenge of deciphering the allocation scheme may become irresistible. Whatever the motivation, however, the effects are the same if the introduction of bias invalidates the trial comparisons.

We must all acknowledge the human elements of this important scientific process and realise that, given the opportunity, researchers sometimes subvert randomisation. Fortunately, subversions of randomisation can generally be prevented with assiduous attention to design and implementation.^15,16 Trial designers and trialists need to embrace the rationale and ethos of randomisation. The proper construction of methodological barriers inoculates trials against selection and confounding biases.  

Double-blinding and exclusions after randomisation

Double-blinding and avoidance of exclusions after trial entry are the most important other methods for reducing bias.^2,20-22 Blinding should not be confused with allocation concealment. Concealment seeks to prevent selection bias, protects the assignment sequence before and until allocation, and can always be successfully implemented.⁷ By contrast, blinding seeks to prevent ascertainment bias, protects the sequence after allocation, and cannot always be done.⁷ We found that trials that were not double-blinded yielded larger estimates of treatment effects than double-blind trials (odds ratios exaggerated, on average, by 17%).¹⁴ Although the strength of this effect falls short of that for allocation concealment, double-blinding appears to prevent bias.

Trials that reported exclusion of participants after randomisation did not yield exaggerated estimates of treatment effects compared with trials in which the reports gave the impression of no exclusions.14 We had expected exaggerated estimates, because absence of bias persists throughout a RCT only if the analysis includes all randomised participants in the originally assigned groups. Our unexpected finding could have resulted from some authors inappropriately reporting that they had randomised the same number of participants as they had analysed, even though some randomised participants had actually been excluded.^14,22  

Reporting

Breaches of randomisation are probably more frequent than suspected. Only 32% of reports published in some specialist journals⁷ and 49% in some premier general journals6 specified an adequate method for generating random assignments. In both groups of journals, only about a quarter of trials reported adequate allocation concealment.^6,7 Only 9% in the specialist journals and 15% in the general journals reported both an adequate method of generating random sequences and an adequate method of concealment.²³

Up to twice as many trials could have been double-blinded as were reported to be so.²² Of those reported as double-blind, only 45% described similarity of the treatment and control regimens and only 26% provided information on the protection of the allocation schedule. Moreover, only 16% provided statements that blinding had been successful. Investigators reported testing effectiveness in only two of 31 trials, and both trials had substantial unblinding of assignments. These results reinforce earlier findings.^8,9,24

The bleakest reporting quality may be on exclusions after randomisation. Trials with no apparent reported exclusions may have made exclusions during the trial but ignored them in the report.^14,22 Moreover, two trials with documented exclusions have published reports indicating no exclusions.²⁵ Paradoxically, trials that reported exclusions generally reflected a higher methodological standard than those reporting no apparent exclusions.^14,22 Thus, reporting of exclusions may be flawed in a way that frequently provides a misleading impression of methodological quality.  

Conclusions

Authors of RCT reports often write inadequate methodological details. Anecdotal reports indicate that aspects of properly conducted RCTs annoy human beings and that researchers sometimes subvert the intended aims of randomisation. Empiric evidence suggests that inadequate methodological reporting correlates with bias in estimation of treatment effects. Faulty reporting appears to portray faulty methods.^14,26

Obviously, to assess accurately the scientific merit of RCTs, readers deserve more detailed methodological information. The CONSORT guidelines address that need. More importantly, we hope that requirements for reporting will stimulate increased methodological rigour in RCTs. Whilst requirements for adequate reporting will not guarantee proper design and implementation, CONSORT constitutes a major step forward in the scientific community's quest for unbiased research.  

References

1 CONSORT Group. Improving the quality of reporting of randomised controlled trials: the CONSORT statement. JAMA 1996; 276: 637-39.

2 The Standards of Reporting Trials Group. A proposal for structured reporting of randomised controlled trials. JAMA 1994; 272: 1926-31.

3 Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature. Call for comments on a proposal to improve reporting of clinical trials in the biomedical literature. Ann Intern Med 1994; 121: 894-95.

4 Grant JM. Randomised trials and the British Journal of Obstetrics and Gynaecology: minimum requirements for publication. Br J Obstet Gynaecol 1995; 102: 849-50.

5 Rennie D. Reporting randomised controlled trials: an experiment and a call for responses from readers. JAMA 1995; 273: 1054-55.

6 Altman DG, Doré CJ. Randomisation and baseline comparisons in clinical trials. Lancet 1990; 335: 149-53.

7 Schulz KF, Chalmers I, Grimes DA, Altman DG. Assessing the quality of randomisation from reports of controlled trials published in obstetrics and gynecology journals. JAMA 1994; 272: 125-28.

8 Mosteller F, Gilbert JP, McPeek B. Reporting standards and research strategies for controlled trials: agenda for the editor. Controlled Clin Trials 1980; 1: 37-58.

9 DerSimonian R, Charette LJ, McPeek B, Mosteller F. Reporting on methods in clinical trials. N Engl J Med 1982; 306: 1332-37.

10 Tyson JE, Furzan JA, Reisch JS, Mize SG. An evaluation of the quality of therapeutic studies in perinatal medicine. J Pediatr 1983; 102: 10-13.

11 Moher D, Fortin P, Jadad AR, et al. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 1996; 347: 363-66.

12 Williams DH, Davis CE. Reporting of assignment methods in clinical trials. Controlled Clin Trials 1994; 15: 294-98.

13 Sonis J, Joines J. The quality of clinical trials published in The Journal of Family Practice. J Fam Pract 1994; 39: 225-35.

14 Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273: 408-12.

15 Schulz KF. Subverting randomisation in controlled trials. JAMA 1995; 274: 1456-58.

16 Schulz KF. Unbiased research and the human spirit: the challenges of randomised controlled trials. Can Med Assoc J 1995; 153: 783-86.

17 Pocock SJ. Statistical aspects of clinical trial design. Statistician 1982; 31: 1-18.

18 Various. Properties of randomisation in clinical trials. Controlled Clin Trials 1988; 9: 287-382.

19 Carleton RA, Sander CA, Burack WR. Heparin administration after acute myocardial infarction. N Engl J Med 1960; 263: 1002-04.

20 Chalmers TC, Levin H, Sacks HS, Reitman D, Berrier J, Nagalingam R. Meta-analysis of clinical trials as a scientific discipline I: control of bias and comparison with large cooperative trials. Stat Med 1987; 6: 315-25.

21 Chalmers I, Hetherinton J, Elbourne D, Keirse MJNC, Enkin M. Materials and methods used in synthesising evidence to evaluate the effects of care during pregnancy and childbirth. In: Chalmers I, Enkin M, Keirse MJNC, eds. Effective care in pregnancy and childbirth: volume I. Pregnancy. Oxford: Oxford University Press, 1989: 39-65.

22 Schulz KF, Grimes DA, Altman DG, Hayes RJ. Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology. BMJ 1996; 312: 742-44.

23 Schulz KF, Chalmers I, Altman DG, Grimes DA, Doré CJ. The methodologic quality of randomisation as assessed from reports of trials in specialist and general medical journals. Online J Curr Clin Trials 1995 (doc no 197).

24 Gøtzsche PC. Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. Controlled Clin Trials 1989; 10: 31-56.

25 Gøtzsche PC. Multiple publication of reports of drug trials. Eur J Clin Pharmacol 1989; 36: 429-32.

26 Liberati A, Himel HN, Chalmers TC. A quality assessment of randomised control trials of primary treatment of breast cancer. J Clin Oncol 1986; 4: 942-51.