Tag Archive for: specification curve analysis

Nutritional Epidemiology: Still Confusing

/in /by

Remember where we began: frustrated with the conflicting studies on nutrition and their impact on our health. The researchers used specification curve analysis to illustrate several issues. The most important point is that there are many ways to analyze large datasets in nutritional epidemiology. Reviewing 15 studies in 24 papers, they found that the number of ways to analyze the data could reach 10 quadrillion (that’s 10,000,000,000,000,000). Obviously, that’s not realistic.

Instead, the approach that could be used by researchers doing these types of studies in any field is to select a randomized sample of different analytical approaches and present the results in the way I did in Tuesday’s Memo. Using that approach showed that fewer than 4% of the studies reached statistical significance. But how much of that could be just dumb luck? Setting the probability of significance at less than .05 (which is most common) means that out of 100 statistical approaches, five could show significance just by chance alone.

This paper addresses a long-standing problem in nutritional research and other areas as well. Researchers who do these types of longitudinal studies already use different analytic techniques in a haphazard way. They just keep chunking data until they find an analytic approach that’s statistically significant, and that’s the one they publish, sort of like a thief checking car doors until he finds one unlocked. Journals won’t publish results that don’t demonstrate significance, even though that would be beneficial for others to find out what not to do. “Publish or perish” just doesn’t work that way.

The Bottom Line

In this series of Memos, I’ve tried to lay out one of the reasons that long-term nutritional studies that look at morbidity and mortality can be flawed, if not contradictory. To be sure, the statistical analyses I’ve talked about are complicated, but that wasn’t my purpose. It’s to let you know that because of the lack of hard and fast rules for outlining the statistical approaches before looking at the data (as is done in randomized controlled trials), the results and the interpretation of those results will always be suspect.

In plain words, never get too excited about longitudinal studies, whether positive or negative. In the coming weeks, I’ll examine some studies on fish oil and multivitamins to illustrate the points I’ve tried to make.

By the way, for those of you really wanting to know whether you should eat red meat based on the analytic study the researchers tested as an example, they stated that there were some holes in the NHANES data that could impact the outcomes they reported. For now, it seems that women may benefit more from eating red meat than men will, but there’s no definitive answer yet.

What are you prepared to do today?

        Dr. Chet

References:
1. https://www.sensible-med.com/p/the-definitive-analysis-of-observational
2. Journal of Clinical Epidemiology 168 (2024) 111278

Nutritional Epidemiology: Specification Curve Analysis

/in /by

Did you look up quadrillion? It’s a 1 with a whole lot of 0s—15 to be exact.

When I finished Saturday’s Memo, the researchers had chosen an area of nutritional epidemiology to focus on: the analytics used to analyze the data. They began with the premise that there are many ways to analyze any data set. They then identified published research studies that examined the consumption of red meat and mortality. They identified 15 publications reporting on 24 studies that examined the effect of red meat on all-cause mortality.

They weren’t doing a meta-analysis to see all the results of all the studies combined; they used a newer technique called specification curve analysis. They identified the type of data used in the analysis, the number of variables, the number of covariates, as well as demographic data. From that information, they then calculated the total number of ways each data set could be analyzed—the specification curve analysis. Turns out that number is 10 quadrillion! That exceeds the capacity of the computing power of a small country, and I can’t even imagine how much electricity that would consume.

They decided to take a randomized sample of the possible ways to analyze the data with specific variables and covariates in each and came up with 1,440 different approaches to analyzing the data. They ran additional tests on the data and eliminated 232 approaches because the data exceeded norms.

Then they ran the remaining analytic approaches on data from several years of the NHANES study. What did they find?

  • The median hazard ratio (HR) was 0.94 for the effect of red meat on all-cause mortality. That means the mortality risk was decreased 6% if the subject ate red meat.
  • HRs ranged from 0.51 to 1.75; 435 approaches yielded HRs more than 1.0 (increased risk) and 773 less than 1.0 (decreased risk). Most analyses showed that eating red meat reduced the HR, and thus reduced the risk of dying.
  • Of all the results, 48 (almost 4%) were statistically significant; of those, 40 indicated that red meat reduced all-cause mortality and 8 that red meat increased all-cause mortality.

Does this mean that eating red meat decreases your risk of dying early? We’re not done yet. We’ll put it all in perspective on Saturday.

What are you prepared to do today?

        Dr. Chet

References:
1. https://www.sensible-med.com/p/the-definitive-analysis-of-observational
2. Journal of Clinical Epidemiology 168 (2024) 111278