October 17, 2021

Understanding science

Ten errors in randomized experiments

A recent review discusses errors in the implementation, analysis, and reporting of randomization within obesity and nutrition research

Read Time 6 minutes

Anyone who’s read my stuff with any regularity is acutely aware of my disdain for the way many observational studies are conducted and interpreted in health and nutrition research, as well as my admiration for randomized-controlled trials (RCTs). Randomization, a method by which study participants are assigned to treatment groups based on chance alone, is a critical component in distinguishing cause and effect. Randomization helps to prevent investigators from introducing systematic (and often hidden) biases between experimental groups.

But there are also many ways in which randomized experiments can fall short. Recently, David Allison and his colleagues published an excellent review discussing ten errors in the implementation, analysis, and reporting of randomized experiments — and outlined best practices to avoid them. David is the Dean of Public Health at Indiana University, where he conducts research on obesity and practices psychology. He is also one of the best statisticians in the world, and will be joining me soon as a guest on the podcast. I’ve provided a brief summary of his review below, but to anyone interested in improving their ability to read and understand research, I suggest reading the original text in its entirety. Here, I focus on the general point that while RCTs may be considered the gold standard for establishing reliable knowledge, they are also prone to error and bias.

§

A) Errors in implementing group allocation

1 | Representing nonrandom allocation methods as random

Occasionally, in studies styled as “randomized,” participants are allocated into treatment groups by use of methods that are not, in fact, random.

The review authors provide the example of a vitamin D supplementation trial in which the control group came from a nonrandomized cohort from another hospital.
Lack of appropriate randomization can introduce selection bias: the selection of subjects into a study that is not representative of the target population.

A 2017 analysis by John Carlisle suggested that nonrandom allocation may be a concern in many studies labeled as “randomized.” One of the trials flagged was the well-known PREDIMED trial. Study participants at high cardiovascular risk were randomly assigned to a Mediterranean diet supplemented with mixed nuts or olive oil, or to a low-fat diet. In some cases, whole households were collectively assigned to the same diet. Even more problematic, one of the sites in the trial assigned entire clinics to the same diet. However, the investigators did not initially report this, and they analyzed their data at the level of individual participants rather than at the level of household or clinic. After discovering these problems in a post-publication audit, PREDIMED investigators retracted and reanalyzed the study, leading to various changes in findings.

2 | Failing to adequately conceal allocation

Allocation concealment hides the sorting of trial participants into treatment groups, preventing researchers from knowing the allocation of the next participant, and participants from knowing their assignment ahead of time. Allocation concealment is different from blinding. Allocation concealment ensures the treatment to be allocated is not known before that participant is entered into, while blinding ensures either the participant or investigator (or both, in the case of double-blinding) remains unaware of treatment allocation after the participant is enrolled in the study. Studies with poor allocation concealment are prone to selection bias.

Poor allocation concealment from participants can lead to bias when, for example, certain study participants prefer one possible treatment over another. Those participants may drop out of the study if they become aware that they will not receive their preferred treatment, potentially skewing the group populations.

Poor allocation concealment from investigators can also lead to bias. Researchers may — consciously or unconsciously — place participants expected to have the best outcomes in the treatment group and those expected to have poorer outcomes in the control group.

3 | Not accounting for changes in allocation ratios

When designing an RCT, one step in the process is determining the ratio of subjects to each group. It’s not always 1:1 – that is, one subject assigned to treatment for every subject assigned to placebo. Sometimes it’s necessary from a statistical standpoint to assign twice (2:1) or three times (3:1) as many individuals to the treatment group as the placebo group. Further, investigators may choose to change the ratios in the middle of a study for various reasons. However, changing the allocation ratio partway through a study requires corresponding changes to statistical analyses, which doesn’t always happen.

Dr. Allison gives the example of a study investigating body weight changes associated with daily intake of sucrose or one of four low-calorie sweeteners. Participants were initially randomly allocated evenly among the five treatment groups (1:1:1:1:1). Because one group had a high attrition rate, the investigators changed to a 2:1:1:1:1 ratio halfway through the study, but they did not account for these different study phases in their statistical analyses.

4 | Replacements are not randomly selected

In virtually all RCTs, some participants will inevitably drop out. One way that investigators try to mitigate this problem is by using intention-to-treat (ITT) analysis, which we discussed in more depth in this article on the efficacy vs. effectiveness of a time-restricted eating trial. In ITT analyses, every participant that is assigned to a treatment group must be included in outcome analyses, regardless of whether those participants followed the protocol or dropped out of the study.

In some cases, investigators replace dropouts with more participants to ensure the study remains adequately powered. These replacements must be randomized to avoid another form of Error #3: changing allocation ratios. (For more information on statistical power, which represents the probability that a study will correctly identify a genuine effect, read Part V of our Studying Studies series.)

B) Errors in the analysis of randomized experiments

5 | Failing to account for non-independence

Sometimes groups of subjects are randomly assigned to a treatment together, but are analyzed as if they were randomized individually. For instance, an entire classroom might be randomized to one group while a separate classroom is assigned to another. These types of studies are referred to as cluster RCTs and are subject to error when they are powered and analyzed at the individual level instead of the group level. The PREDIMED study exemplifies this error, as groups of individuals within certain households or clinics were assigned to a treatment together, but the authors did not initially adjust their statistical analysis to account for clustering.

6 | Basing conclusions on within-group statistical tests instead of between-groups tests

The strength of an RCT lies in its ability to compare the results between two or more groups. For example, I recently wrote about a study that randomized men to morning exercise, evening exercise, or no exercise. The investigators reported that nocturnal glucose profiles improved only in men who exercised in the evening. The improvement, however, was “in-group,” meaning that nocturnal glucose levels had improved relative to baseline values, not compared to the other groups in the study. The authors’ conclusion that evening exercise conferred greater benefit for glycemic control than morning or no exercise is thus an example of the Difference in Nominal Significance (DINS) error. This error occurs when differences in “in-group” effects are used to draw conclusions about differences in “between-group” effects, rather than directly comparing groups to each other.

7 | Improper pooling of data

Pooling data under the umbrella of one study without accounting for it in statistical analyses can introduce bias. Dr. Allison cites an example of a trial on the effects of weight loss on telomere length in women with breast cancer. Data were pooled from two different phases of an RCT with different allocation ratios (see Error #3), which wasn’t taken into account in the analysis.

The different sites, subgroups, or phases of a study need to be taken into account during analysis. Otherwise, any differences in the subsets of data being pooled together can bias the estimation of an effect in the trial.

8 | Failing to account for missing data

Missing data — whether due to dropouts, errors in measurement, or other reasons — may not occur completely at random, breaking the randomization component of the study and introducing bias.

The review authors provide the example of a trial of intermittent energy restriction vs. continuous energy restriction on body composition and resting metabolic rate. The study had a 50% dropout rate, yet only data from participants who completed the protocol were analyzed. (This is an example of “per protocol” analysis, in which data from noncompliant subjects is removed from analyses.) Reanalysis of the study including all participants halved the magnitude of effect estimates compared with original reported results.

Investigators may mitigate this problem by reporting both per protocol and ITT results: efficacy and effectiveness, respectively. However, Dr. Allison suggests that this isn’t a perfect fix: “ITT can estimate the effect of assignment, not treatment per se, in an unbiased manner, whereas the per protocol analysis can only estimate in a way that allows the possibility for bias.”

(As noted earlier, this article details efficacy vs. effectiveness of time-restricted eating.)

C) Errors in the reporting of randomization

9 | Failing to fully describe randomization

Investigators must provide sufficient information so that readers can fully comprehend and evaluate the methods used for randomization. The review authors themselves admit to having a history of inadequate reporting of randomization methods.

10 | Failing to properly communicate inferences from randomized studies

When following the ITT principle, an RCT tests the effect of assigning participants to a treatment on the outcome of interest, but investigators often communicate results as the effect of the treatment itself (meaning, how well the treatment works if followed exactly as it’s prescribed).  Avoidance of this error depends on conscientious framing of the precise causality question addressed by the study.

For example, in the article I wrote reviewing a time-restricted eating trial, I highlighted the investigators’ statement that, “Time-restricted eating, in the absence of other interventions, is not more effective in weight loss than eating throughout the day.” In actuality, the investigators found that being assigned to time-restricted eating, in the absence of other interventions, is not more effective in weight loss than being assigned to eating throughout the day.

§

The review from David Allison and his colleagues highlights that while randomized controlled trials are powerful tools for examining cause-and-effect relationships, they are not immune to errors and bias. The paper is a great reminder of the high level of rigor involved in designing, conducting, and reporting randomized experiments, as well as a useful guide for investigators and readers alike for avoiding many pitfalls associated with this study design.

Disclaimer: This blog is for general informational purposes only and does not constitute the practice of medicine, nursing or other professional health care services, including the giving of medical advice, and no doctor/patient relationship is formed. The use of information on this blog or materials linked from this blog is at the user's own risk. The content of this blog is not intended to be a substitute for professional medical advice, diagnosis, or treatment. Users should not disregard, or delay in obtaining, medical advice for any medical condition they may have, and should seek the assistance of their health care professionals for any such conditions.
  1. Greetings Peter,

    Incredible and well detailed article. I am curious, however, that you haven’t seemed to comment at all on the data coming out of Israel which refutes the vaccine for cov 19 (as this seems to be the biggest health care issue of our lifetimes).
    Wouldn’t it feel good for you to call a spade a spade and call out this shill
    narrative peddled by the CDC.

    thanks,

    James

  2. Peter,

    Excellent article, and I especially appreciate your emphasis on allocation concealment (one of the most important aspects of the process of randomization that most investigators with whom I’ve worked don’t fully understand). And, I’m very excited that you’ll have David Allison on your podcast, he truly is one of the best!

    I do, however, have one correction to offer. While you correctly note that lack of randomization can induce selection bias, for randomized trials that really has nothing to do with a target population. Instead, in RCTs, selection bias has to do with who gets what treatment. And, as you correctly noted, the whole goal is to take selection away from either the clinician or the patient and leave it entirely up to chance (which can only be accomplished with adequate allocation concealment!).

    Cheers,
    Mark

  3. The biggest issue in RCTs might be the improper control of comorbidities.

    Take age for example (as a variable). Two arms might have the same avg age, but one arm might have a larger variance, having both a higher number of older people and younger people. This can easily explain a couple of % difference in trial outcomes!

    Fish Oil: One RCT finds reduced morbidity, another doesn’t.
    Aspirin: One RCT finds reduced morbidity, another doesn’t.

    How do you explain all this? Probably no one can. My guess is two arms would be imbalanced in terms of baseline death probability. This leads to conflicting outputs in trials.

    Nobody, remember nobody controls for probability of dying. Studies control for avg age, cholesterol, # of people with diabetes etc. But this isn’t enough. You literally need a statistical model for mortality rate, then equalize two arms in distribution by mortality rate. Then do the test.

    Most RCTs with mortality-endpoint are suspect. They have way too many confounders and/or underpowered to detect small deltas. Yet, the authors present as if they do.

    Best.

  4. Kudos for a well laid out article.

    Gorkan’s insight that “nobody controls for probability of dying,” was very good too. Most researchers would probably gloss over unequal end-of-study populations.

    Is there any chance you and Dr. Allison might chew a bit on how these systemic errors ( and others ) can tilt outcomes when using Bayesian methodologies?

    Best regards, Ted

[gravityform id="5" title="false" description="false"]
<div class='gf_browser_unknown gform_wrapper form--inline_wrapper gtm--newsletter-form-popup_wrapper gform_legacy_markup_wrapper' id='gform_wrapper_5' ><form method='post' enctype='multipart/form-data' id='gform_5' class='form--inline gtm--newsletter-form-popup gform_legacy_markup' action='/ten-errors-in-randomized-experiments/' novalidate> <div class='gform_body gform-body'><ul id='gform_fields_5' class='gform_fields top_label form_sublabel_below description_below'><li id="field_5_1" class="gfield gfield_contains_required field_sublabel_below field_description_below gfield_visibility_visible" ><label class='gfield_label' for='input_5_1' >Email<span class="gfield_required"><span class="gfield_required gfield_required_asterisk">*</span></span></label><div class='ginput_container ginput_container_email'> <input name='input_1' id='input_5_1' type='email' value='' class='large' placeholder='Your email address' aria-required="true" aria-invalid="false" /> </div></li><li id="field_5_10" class="gfield gfield--width-full field_sublabel_below field_description_below gfield_visibility_visible" ><label class='gfield_label' for='input_5_10' >CAPTCHA</label><div id='input_5_10' class='ginput_container ginput_recaptcha' data-sitekey='6LeuYQ0dAAAAAHQqB_Z3xNQAwCRlN5ykarrLsrZw' data-theme='light' data-tabindex='-1' data-size='invisible' data-badge='bottomright'></div></li><li id="field_5_2" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_2' id='input_5_2' type='hidden' class='gform_hidden' aria-invalid="false" value='Ten errors in randomized experiments' /></div></li><li id="field_5_3" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_3' id='input_5_3' type='hidden' class='gform_hidden' aria-invalid="false" value='https://peterattiamd.com/ten-errors-in-randomized-experiments/' /></div></li><li id="field_5_4" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_4' id='input_5_4' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_5_5" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_5' id='input_5_5' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_5_9" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_9' id='input_5_9' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_5_8" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_8' id='input_5_8' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_5_7" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_7' id='input_5_7' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_5_6" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_6' id='input_5_6' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li></ul></div> <div class='gform_footer top_label'> <input type='submit' id='gform_submit_button_5' class='gform_button button' value='Sign up' onclick='if(window["gf_submitting_5"]){return false;} if( !jQuery("#gform_5")[0].checkValidity || jQuery("#gform_5")[0].checkValidity()){window["gf_submitting_5"]=true;} ' onkeypress='if( event.keyCode == 13 ){ if(window["gf_submitting_5"]){return false;} if( !jQuery("#gform_5")[0].checkValidity || jQuery("#gform_5")[0].checkValidity()){window["gf_submitting_5"]=true;} jQuery("#gform_5").trigger("submit",[true]); }' /> <input type='hidden' class='gform_hidden' name='is_submit_5' value='1' /> <input type='hidden' class='gform_hidden' name='gform_submit' value='5' /> <input type='hidden' class='gform_hidden' name='gform_unique_id' value='' /> <input type='hidden' class='gform_hidden' name='state_5' value='WyJbXSIsIjU1MGM3MDVmNDUwOTYxNmQyYTU0NTkwMjUzZjU2NGI1Il0=' /> <input type='hidden' class='gform_hidden' name='gform_target_page_number_5' id='gform_target_page_number_5' value='0' /> <input type='hidden' class='gform_hidden' name='gform_source_page_number_5' id='gform_source_page_number_5' value='1' /> <input type='hidden' name='gform_field_values' value='' /> </div> <p style="display: none !important;"><label>&#916;<textarea name="ak_hp_textarea" cols="45" rows="8" maxlength="100"></textarea></label><input type="hidden" id="ak_js" name="ak_js" value="40"/><script>document.getElementById( "ak_js" ).setAttribute( "value", ( new Date() ).getTime() );</script></p></form> </div>
[gravityform id="1" title="false" description="false"]
<div class='gf_browser_unknown gform_wrapper form--inline_wrapper gtm--newsletter-form_wrapper gform_legacy_markup_wrapper' id='gform_wrapper_1' ><form method='post' enctype='multipart/form-data' id='gform_1' class='form--inline gtm--newsletter-form gform_legacy_markup' action='/ten-errors-in-randomized-experiments/' novalidate> <div class='gform_body gform-body'><ul id='gform_fields_1' class='gform_fields top_label form_sublabel_below description_below'><li id="field_1_1" class="gfield gfield_contains_required field_sublabel_below field_description_below gfield_visibility_visible" ><label class='gfield_label' for='input_1_1' >Email<span class="gfield_required"><span class="gfield_required gfield_required_asterisk">*</span></span></label><div class='ginput_container ginput_container_email'> <input name='input_1' id='input_1_1' type='email' value='' class='large' placeholder='Your email address' aria-required="true" aria-invalid="false" /> </div></li><li id="field_1_11" class="gfield gfield--width-full ohnohoney field_sublabel_below field_description_below gfield_visibility_hidden" ><div class='admin-hidden-markup'><i class='gform-icon gform-icon--hidden'></i><span>Hidden</span></div><label class='gfield_label' for='input_1_11' >H-o-n-e-y-p-o-t</label><div class='ginput_container ginput_container_text'><input name='input_11' id='input_1_11' type='text' value='' class='large' aria-invalid="false" /> </div></li><li id="field_1_10" class="gfield gfield--width-full field_sublabel_below field_description_below gfield_visibility_visible" ><label class='gfield_label' for='input_1_10' >CAPTCHA</label><div id='input_1_10' class='ginput_container ginput_recaptcha' data-sitekey='6LeuYQ0dAAAAAHQqB_Z3xNQAwCRlN5ykarrLsrZw' data-theme='dark' data-tabindex='-1' data-size='invisible' data-badge='bottomleft'></div></li><li id="field_1_2" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_2' id='input_1_2' type='hidden' class='gform_hidden' aria-invalid="false" value='Ten errors in randomized experiments' /></div></li><li id="field_1_3" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_3' id='input_1_3' type='hidden' class='gform_hidden' aria-invalid="false" value='https://peterattiamd.com/ten-errors-in-randomized-experiments/' /></div></li><li id="field_1_4" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_4' id='input_1_4' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_1_5" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_5' id='input_1_5' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_1_9" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_9' id='input_1_9' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_1_8" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_8' id='input_1_8' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_1_7" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_7' id='input_1_7' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_1_6" class="gfield gform_hidden field_sublabel_below field_description_below gfield_visibility_visible" ><div class='ginput_container ginput_container_text'><input name='input_6' id='input_1_6' type='hidden' class='gform_hidden' aria-invalid="false" value='' /></div></li><li id="field_1_12" class="gfield gform_validation_container field_sublabel_below field_description_below gfield_visibility_visible" ><label class='gfield_label' for='input_1_12' >Name</label><div class='ginput_container'><input name='input_12' id='input_1_12' type='text' value='' autocomplete='new-password'/></div><div class='gfield_description' id='gfield_description_1_12'>This field is for validation purposes and should be left unchanged.</div></li></ul></div> <div class='gform_footer top_label'> <input type='submit' id='gform_submit_button_1' class='gform_button button' value='Sign up' onclick='if(window["gf_submitting_1"]){return false;} if( !jQuery("#gform_1")[0].checkValidity || jQuery("#gform_1")[0].checkValidity()){window["gf_submitting_1"]=true;} ' onkeypress='if( event.keyCode == 13 ){ if(window["gf_submitting_1"]){return false;} if( !jQuery("#gform_1")[0].checkValidity || jQuery("#gform_1")[0].checkValidity()){window["gf_submitting_1"]=true;} jQuery("#gform_1").trigger("submit",[true]); }' /> <input type='hidden' class='gform_hidden' name='is_submit_1' value='1' /> <input type='hidden' class='gform_hidden' name='gform_submit' value='1' /> <input type='hidden' class='gform_hidden' name='gform_unique_id' value='' /> <input type='hidden' class='gform_hidden' name='state_1' value='WyJbXSIsIjU1MGM3MDVmNDUwOTYxNmQyYTU0NTkwMjUzZjU2NGI1Il0=' /> <input type='hidden' class='gform_hidden' name='gform_target_page_number_1' id='gform_target_page_number_1' value='0' /> <input type='hidden' class='gform_hidden' name='gform_source_page_number_1' id='gform_source_page_number_1' value='1' /> <input type='hidden' name='gform_field_values' value='' /> </div> <p style="display: none !important;"><label>&#916;<textarea name="ak_hp_textarea" cols="45" rows="8" maxlength="100"></textarea></label><input type="hidden" id="ak_js" name="ak_js" value="40"/><script>document.getElementById( "ak_js" ).setAttribute( "value", ( new Date() ).getTime() );</script></p></form> </div>
Facebook icon Twitter icon Instagram icon Pinterest icon Google+ icon YouTube icon LinkedIn icon Contact icon