Nutritional Epidemiology: Still Confusing

Remember where we began: frustrated with the conflicting studies on nutrition and their impact on our health. The researchers used specification curve analysis to illustrate several issues. The most important point is that there are many ways to analyze large datasets in nutritional epidemiology. Reviewing 15 studies in 24 papers, they found that the number of ways to analyze the data could reach 10 quadrillion (that’s 10,000,000,000,000,000). Obviously, that’s not realistic.

Instead, the approach that could be used by researchers doing these types of studies in any field is to select a randomized sample of different analytical approaches and present the results in the way I did in Tuesday’s Memo. Using that approach showed that fewer than 4% of the studies reached statistical significance. But how much of that could be just dumb luck? Setting the probability of significance at less than .05 (which is most common) means that out of 100 statistical approaches, five could show significance just by chance alone.

This paper addresses a long-standing problem in nutritional research and other areas as well. Researchers who do these types of longitudinal studies already use different analytic techniques in a haphazard way. They just keep chunking data until they find an analytic approach that’s statistically significant, and that’s the one they publish, sort of like a thief checking car doors until he finds one unlocked. Journals won’t publish results that don’t demonstrate significance, even though that would be beneficial for others to find out what not to do. “Publish or perish” just doesn’t work that way.

The Bottom Line

In this series of Memos, I’ve tried to lay out one of the reasons that long-term nutritional studies that look at morbidity and mortality can be flawed, if not contradictory. To be sure, the statistical analyses I’ve talked about are complicated, but that wasn’t my purpose. It’s to let you know that because of the lack of hard and fast rules for outlining the statistical approaches before looking at the data (as is done in randomized controlled trials), the results and the interpretation of those results will always be suspect.

In plain words, never get too excited about longitudinal studies, whether positive or negative. In the coming weeks, I’ll examine some studies on fish oil and multivitamins to illustrate the points I’ve tried to make.

By the way, for those of you really wanting to know whether you should eat red meat based on the analytic study the researchers tested as an example, they stated that there were some holes in the NHANES data that could impact the outcomes they reported. For now, it seems that women may benefit more from eating red meat than men will, but there’s no definitive answer yet.

What are you prepared to do today?