Learn more about Peter Attia’s digital program
Recently, I came across a study with a title that caught my eye: “Metformin Monotherapy Alters the Human Plasma Lipidome Independent of Clinical Markers of Glycemic Control and Cardiovascular Disease Risk in a Type 2 Diabetes Clinical Cohort.” Metformin, lipid biology, cardiovascular disease, and type 2 diabetes? This study must be a goldmine of information relevant to health and longevity.
But if there was gold here, it wasn’t in the form that I expected. Though I found nothing in the way of fresh insights on disease or treatments, I did come away with a ready-made lesson in flaws that often plague scientific research – using results to retroactively create hypotheses, and inappropriately drawing conclusions based on isolated pieces of evidence.
About the study
Metformin is an effective and widely used medication for maintaining glycemic control among individuals with type 2 diabetes (T2D). Despite the ubiquitous use of this drug, the cellular and molecular mechanisms underlying metformin’s impact on glucose regulation remain poorly understood. Furthermore, some evidence has suggested that metformin may have benefits for cardiovascular (CV) health beyond its impact on glycemic control, prompting investigators Wancewicz et al. to conduct the present study in order to shed new light on potential mechanisms by which metformin might exert these positive effects.
To do so, the authors utilized mass spectrometry to analyze metabolites present in plasma samples from three groups of participants: nondiabetic subjects (“ND,” n=14), subjects with T2D who were being treated exclusively with diet and lifestyle modifications (“T2D-DL,” n=7), and subjects with T2D who were being treated with metformin monotherapy (“T2D-M,” n=8). After cataloging and quantifying an astonishing total of 2,925 different compounds and determining which varied significantly between groups, the researchers applied pathway analysis to identify the biochemical pathways that appeared to be impacted by T2D, lifestyle, and metformin therapy.
Of the compounds identified through mass spectrometry, 251 were found to be exclusive to the ND group, 11 exclusive to the T2D-DL group, and 10 exclusive to the T2D-M group. Pathway analysis revealed that both T2D groups had significant elevations relative to the ND group in phospholipids and in metabolites related to sphingolipid, purine, and linoleic acid metabolism, as well as steroidogenesis. (For the purpose of this newsletter, there’s no need to concern yourself with the details of these pathways or their functions, as we’ll see below.) Similar results were observed when comparing the ND group to the T2D-DL group and to the T2D-M group.
Of course, to determine the metabolic effects of metformin – as opposed to T2D itself – we must also examine the differences between the T2D-DL group and the T2D-M group. In this comparison, Wancewicz et al. report that although the T2D-DL group demonstrated significant elevations relative to the ND group in certain saturated and unsaturated fatty acids (including palmitic, stearic, oleic, linoleic, and arachidonic acid), the T2D-M group did not differ significantly from the ND group in these lipids. Palmitoyl carnitine (an acylcarnitine) was also elevated in the T2D-DL group – but not the T2D-M group – relative to ND controls, as were oxidized phosphatidylcholines. Citing previous studies linking circulating levels of fatty acids, palmitoyl carnitine, and phosphatidylcholines to CV risk, the authors argue that these results are indicative of cardioprotective benefits of metformin independent of the drug’s effects on glucose control.
Where it went wrong
Putting aside the authors’ interpretation of their results for a moment, let’s first examine the validity of the basic result that metformin leads to changes in circulating levels of various lipids and other metabolites.
Importantly, these results were obtained from samples collected at a single time point in non-randomized groups. We cannot conclude whether metformin (or lifestyle) are associated with a change in plasma lipids without collecting data from multiple timepoints, and we cannot conclude whether metformin (or lifestyle) are causing the alleged change if the T2D groups were not randomly assigned. Those who are managing their diabetes effectively with diet and exercise alone may differ in critical ways from those on metformin – for instance, they may be in the earlier stages of the disease – and these differences may be the underlying cause of the variation in circulating metabolites. (Indeed, data on HbA1c, fasting glucose, and fasting insulin all suggest that the T2D-M had poorer diabetes management than the T2D-DL group, despite these discrepancies not achieving significance.)
Additionally, this study consisted of only 29 total subjects across all three groups, with T2D-DL and T2D-M groups consisting of only seven and eight participants, respectively. As sample size decreases, the likelihood that the trends observed in the data represent real, universal effects also decreases. Instead, results might (a) be driven by one or a handful of people with exceptionally high or exceptionally low values in any given study group, (b) represent the effect of any number of confounding variables, which have the potential for large impacts in a small group, and (c) apply only to the narrow population tested (for instance, of the 29 participants, only two identified as non-white).
Finally, the most significant findings in this study were relative to the ND group, not between the T2D-DL group and the T2D-M group. A statistically significant difference between the T2D-DL and ND group and the absence of that difference in the T2D-M group relative to the ND group does not imply that the T2D-DL and T2D-M groups are significantly different from each other, and indeed, some of the reported findings were insignificant when T2D-DL and T2D-M groups were compared directly.
In short, even at face value, I have little faith that these results reveal any reliable insights as to the metabolic changes caused by – or even associated with – metformin use.
Painting bullseyes around bullet holes
Let’s go out on a (very shaky) limb and assume these results reflect genuine, consistent effects of metformin on metabolism. How compelling is the authors’ conclusion that these metabolic findings indicate that metformin is cardioprotective beyond effects on glycemic control? The answer, as it turns out, is “not very.”
Wancewicz et al. reported elevations in palmitoyl carnitine, certain oxidized phospholipids, and certain fatty acids in the T2D-DL group relative to the T2D-M group. It appears that, based on these results, the authors then pored over existing literature in search of evidence correlating these specific metabolites to cardiovascular disease (CVD) – in other words, a search that was heavily biased – and ultimately combined this evidence with their own data in order to conclude that metformin impacts CVD risk. Spot the circular logic? In essence, the investigators generated theories and hypotheses based on their own results, then concluded that their results validate those theories and hypotheses. It’s another example of a “Texas sharpshooter fallacy” – the equivalent of firing a gun at a wall, drawing a target around the bullet hole, and calling the shot a bullseye. (I did a post on a similar but slightly different statistical sin a while back, which you can see here.)
Leapfrogging from metformin to CVD
Compounding this problem, the authors pair this “sharpshooting” with another logical misstep. Their study identified metabolites that were elevated among diabetic patients with lifestyle interventions but not among diabetic patients on metformin monotherapy. Even if we ignore the retroactive hypotheses and pretend that the investigators had determined, in an unbiased manner, that prior evidence supports correlations between these metabolites and CVD, we cannot leapfrog through this series of evidentiary links in order to draw conclusions about metformin and CVD. To do so would constitute an invalid form of logic known as a “syllogistic fallacy,” in which deductive reasoning is inappropriately applied to two facts in order to arrive at a new, unifying conclusion.
Consider this example: driving while intoxicated causes car accidents. Car accidents are positively correlated with the amount of ice on roads. Few would dispute these two facts, so can we then conclude that driving while intoxicated causes ice on roads, or even correlates with ice on roads? Of course not. Open questions on the directionality of correlations and on countless confounding variables prohibit any valid interpretation of a relationship between intoxicated driving and road ice, just as open questions on the directionality of correlations and on countless confounding variables prohibit any valid interpretation of a relationship between metformin and CVD.
Guessing at such relationships is an important part of developing hypotheses, and researchers can and should muse on possible explanations and implications of their results. But those hypotheses must then be subject to independent, rigorous experimentation before they are presented as conclusions. The authors did absolutely nothing to extend their findings into the domain of clinical relevance, collecting no data on incident cardiovascular events or disease diagnosis, yet they repeatedly present the connection between metformin and CVD as far more than mere speculation. They also failed to assess a number of metrics with more established relationships to CVD (e.g., coronary artery calcium), and for the few that they did measure (i.e., LDL-C, blood pressure), the groups did not differ significantly.
What can we conclude
At best, this study can be labeled as a “fishing expedition,” a type of study which is often needed in order to move forward in a given field. Science needs studies that simply set out to see what they can find – testing many variables and reporting which turn out to be significant, which in turn inspires others toward new hypotheses and experiments. But a fishing expedition should never be passed off as a hypothesis-testing endeavor in itself. Wancewicz et al. insisted on trying to convince us why the specific “fish” they caught were precisely the ones they needed and expected, drawing spurious conclusions about their results in the process.
So in deriving value from this “goldmine” study, we’ll have to settle for something more akin to a silver lining. We can conclude virtually nothing about metformin, diabetes, or CVD, but the paper offers important lessons in other regards, as it practically demands to be used as a case study in flawed logic. And for conducting or interpreting scientific research, understanding and recognizing poor research practices is just as valuable as learning good research practices.
For a list of all previous weekly emails, click here.
