The Study

High Profile Dietary Intervention fails to approach causality and replicability

NOTE: An earlier, May 2020 look at this issue can be found here: Significant changes necessary in order for dietary intervention studies to be causal and replicable


“Inconsistent and contradictory results from nutrition studies conducted by different investigators continue to emerge, in part because of the inherent variability of natural products, as well as the unknown and therefore uncontrolled variables in study populations and experimental designs.”The Challenge of Reproducibility and Accuracy in Nutrition Research: Resources and Pitfalls
Diet and health studies are infamous for conflicting results that fall short of credibly causal relationships. Those studies are plagued by equivocal and often contradictory results, lack of specific causality, replication failures, and significant barriers to translating results into valid clinical recommendations.
As a result, health professionals and their clients are faced with a steady diet of popular articles such as: 5 Foods We Thought Were Bad For Us, Now Turn Out to Be Good.
The same ambivalence spills over into the scientific literature where one study finds evidence for the healthy consumption of nuts, and another is not so sure: Should we go nuts about nuts? 
An exhaustive look at the foundations of this issue can be found at: Goals in Nutrition Science 2015–2020

Teachable moment?

A high-profile 2019 nutritional study headed by NIH research section head Kevin Hall has come under increasing fire for data and protocol shortcomings which make it a teachable moment into why dietary interventipns so often fail to be useful and informative.

The issue was raised widely when research watchdog, Retraction Watch, noted that the paper published in Cell Metabolism— “Ultra-Processed Diets Cause Excess Calorie Intake and Weight Gain” — was accompanied by an editorial that urged caution in interpreting the results.

In addition to broad attention in general media outlets, Retraction Watch on January 22, noted that the study “[H]as been cited 164 times, according to Clarivate Analytics’ Web of Science, giving it ‘highly cited paper’ and ‘hot paper’ designations. (The paper has also been subject to a correction in October of last year.)”

The first detailed expression of concern for the Hall paper came more than two years ago from Edward Archer in PubPeer  in which he detailed four methodological errors.

More recently, on January 21, 2021, two posts — one from Nick Brown and another by Brown in collaboration with Ethan and Sarah Ludwin-Peery — raised numerous questions about data integrity and experimental protocols.

Fundamental Errors in Hall Study Are Common in Most (If not all) Dietary Intervention Studies

The Hall study is non-replicable on its face because it does not provide a list of ingredients (reagents) and procedures (recipes) that would allow a third-party investigator to accurately reproduce it in an independent study.

Second, the Hall study’s results are inconclusive because the Ultra-Processed Foods (UPFs) in the Hall study are cross-contaminated by Plastic-Derived Chemicals (PDCs) such as Bisphenol A and phthalates which exhibit physiological effects similar to those being measured by the investigation (See charts farther down.)

Fundamentally, the Hall study fails because it has failed to observe standard and long-accepted laboratory practices that are basic to the scientific method in general and bench-science standards specifically. (NOTE: Citations and other details on this and the following statements and assertions can be found as part of this study and at this site.)

First of all, in a proper scientific experiment, there is a control phase and an experimental phase. In this case, the control phase would be unprocessed food and the experimental, Ultra-Processed. To posit a valid conclusion, the two legs of the study must be as different as possible.

However, the “UN-processed” stage of the Hall study is contaminated both by the presence of PDCs and the inclusion of foods which contain unknown food additives. PDCs are a hallmark of food processing which means that extreme care must be taken to avoid their unknown contamination effects. These effects can be characterized as adding “noise” to the data. In general, the noisier data is, the greater the likelihood that conclusions will be inaccurate or invalid.

In a dietary intervention studies — as in standard bench experiments — proper experimental design requires that the consumed substances must be treated as reagents with sources and other provenance details fully disclosed to investigators who may attempt to replicate the results. If you do not know the precise composition of your reagents, then you must assume contamination. And contamination of reagents, as noted above, contaminates results.

Regrettably, the Hall study, and other recent dietary intervention studies tolerate excessive contamination which, in turn, creates noise and uncertainty. The use of statistical methods designed to tease reasonable conclusions from the noise offer false comfort that a study conclusion is valid.. There are no certainties in those methofs — only probabilities — that the correct “voice” is the one plucked from the cacophony.

It is far better to start by eliminating as many voices from the room as possible. This is especially relevant since UPFs and PDCs speak with the  same “accents” as defined by their metabolic effects.

Ultra-Processed Foods and Plastic Derived Chemicals produce very similar effects

A post-trial examination of another recent published study — revealed yet another fatal confounding factor: the effects of PDCs and UPFs are nearly identical and completely confounded any causal conclusion that study or Hall’s might want to draw.

This cross-contamination phenomenon is increasingly vital given the rise in micro- and nano-plastics in the food chain and environment.

A closer examination of the data also shows the ubiquitous PDC, Bisphenol A, affects ghrelin, leptin, and adiponectin in consistent ways as the Hall study recorded for Ultra-Processed Foods.

In addition, various disease/syndromes and clinical indicators also show parallels between Plastic-Derived Chemicals and Ultra-Processed Click this link to download Excel spreadsheet with full-sized text

It is not easy to bar PDC noise from the experimental room, but methods so exist (some described here and here) to eliminate most of it. It is scientifically careless not to make every effort to reduce noise.

By contrast, Hall’s experimental auditorium was filled with extreme ambient noise of unknown origin or amplitude that made the desired voice unintelligible.

An examination of a single meal in the Hall study menu as presented in the Supplemental Materials offers a telling glimpse to many of these contaminants and their variations from valid scientific practices.

Notably nearly every meal in that supplemental menu document suffers from significant cross contamination by UPFs PDCs. It is little wonder that the amount of weight gain cited in the Hall study was puzzling and equivocal according to the analyses by Nick Brown and Ethan & Sarah Ludwin-Peery

First of all, the yogurt will have undergone substantially more processing than whole milk. A look at the meal “Unprocessed Menu, Day 2: Lunch” in which “skim milk” is presented as “unprocessed” indicates a lack of understanding about dairy products in general and that skim and reduced-fat milk undergo more processing steps, many of which involve plastic contact.

While it is true that many Plastic Derived Chemicals (PDCs) are lipophilic and may be more prevalent in higher-fat foods, this is yet another ignored unknown in this menu presentation.

Second, the Hall material does not indicate which milk-fat content of the Fage brand yogurt served.

Third, the yogurt was packaged in a plastic container from which PDCs such as BPA and phthalates can leach.

Fourth, while not specified, the Blue Diamond walnuts most likely came packaged in a plastic container which can leach PDCs. In addition, shelled nuts undergo processing steps that expose them to plastic surfaces.

Fifth, no washing or preparation measures were disclosed for the fresh fruit.

Was the rinse tap water or carbon-filtered? Was the fruit in contact with plastic bowls, cutting boards or other items? Was an attempt made to reduce the quantity of polymer/wax-based materials used to help- preserve the fresh apple’s moisture level?  Were the fresh fruits certified organic which reduces potential pesticide contamination? Where were fresh fruit and vegetables acquired?

Sixth, how were the preparation and eating utensils and dishes washed and handled? Were they rinsed in carbon filtered water to remove any cleaning residues from detergents, ethoxylates and other contaminants?

Eighth, commercial table salt contains unknown additives including iodine supplements and anti-caking agents of undisclosed identity or quantity. In addition, common table salt is contaminated in widely varying amounts by microplastics, either from processing or environmental contamination. A preferred substitute for a dietary intervention would be NaCl from a reputable laboratory supply company.

Ninth, was the olive oil first cold press (no chemical extraction), processed with plastic-free apparatus and packaged in glass?

Finally,  none of the meals presented in the supplemental, materials documented procedures, sources of materials, preparation details. In addition, no data on the weight or volume of the foods served and the error tolerance. It’s also worth noting that all quantities should have been recorded in grams. This avoids meniscus errors for liquids as well as packing errors for granulated substances such as sugar, flour, spices etc.

Partial blame may rest with NOVA

The Hall study notes that it followed the the NOVA classification as a guide for defining Ultra-Processed Foods.

Notably, that 2010 proposal admits that its categories are vague and often equivocal. Significantly, NOVA does not include the almost unfathomable number of FDA-approved food additives, most of which are not required to be listed on a food label. Also not included in NOVA is  the presence of PDCs that leach from packaging materials, processing, and food preparation items.

NOVA also acknowledges that the simply act of cooking is processing. This means that the only foods that are UN-processed are those eaten raw.

However, for the purposes of a scientific investigation involving UPFs, the term “minimally processed” should be used instead.
Further, confounding factors involving both UPFs and PDCs should be reduced by obtaining foods either directly from the producer, as near to the top of the food chain as possible, and before the addition of any additives.
Precautions should also be taken during preparation to remove or reduce PDF contamination as in this example, this one, this one, and this one.

Black Box Meals and Non-Replicability

All of the meals in the Hall study are “Black Box Meals” which are standard in dietary intervention investigations.

Black Box means that the investigators in the study have used third parties to prepare the means and have not supervised, directed or participated in the selection, provenance and gathering of ingredients, or in the process of preparation/cooking and serving. As a result, no data or guidance exists for a valid replication.

Palatability and weight gain

Food consumption is affected by palatability. For its part, palatability depends on preparation, food quality. and spicing. This presents significant unknown variable in the Hall study food preparation in its attempt to prove the thesis that UPFs cause people to gain weight because of substances added to make them more palatable and irresistible to eat.
Particularly important is spicing. The healthiest, PDC- and UPF-free meal can be palatable … or unpalatable depending upon the preparation and spicing … and that affects the food appeal which affects the amount consumed.
We do not know from the Hall study whether the meals were standard industrial/high school cafeteria fare, or spiced and prepared so as to compete for palatability with the UPF meals.
This is particularly relevant in Hall’s  “Black Box” kitchen in which we have no composition or preparation data to assess.

In Search of a Replicable, Causal Dietary Intervention Trial

From the foregoing, the only reasonable conclusion that can be reached is that neither the Hall study nor this other recent trial could attribute a specific causal action to PDC’s or UPFs because both are present in study foods, exert almost identical effects and are, therefore, mutually confounding factors.

That situation  further indicates that, in addition to the revised protocol changes we implemented, our study, Hall’s and every other dietary intervention study must incorporate the dosing of a relevant single compound of interest in order to establish causality and to enable clinically valid health decisions.

Quo Vadis?

From “Best practices in nutrition science to earn and keep the public’s trust

Nutrition science research bears additional burdens for rigor and reproducibility. One such burden is the “unscientific beliefs” that result from the casual familiarity everyone experiences with food….

Another burden is the expectations of the “best science” that understandably accompanies research findings with immediate applicability in daily health-relevant decision-making. There is no single solution to overcoming the challenges of obtaining quality evidence to support nutrition recommendations and nutrition policy…,

Solutions will likely require combinations of large and long-term RCTs (), inclusion of dose–response relations in the design of observational studies, Mendelian randomization approaches in nutrition and other approaches.

Further reading

Reproducibility: Revised Stealth Syndromes Human Study Protocol- APPENDIX 1

Jessie P. Buckley, Hyunju Kim, Eugenia Wong, Casey M. Rebholz, Ultra-processed food consumption https://www.sciencedirect.com/science/article/pii/S0160412019317416

Exposure to phthalates and bisphenols in the US National Health and Nutrition Examination Survey, 2013–2014,Environment International, Volume 131, 2019, 105057, ISSN 0160-4120, https://doi.org/10.1016/j.envint.2019.105057.)

Guasch-Ferré M, Bulló M, Martínez-González MÁ, et al. Frequency of nut consumption and mortality risk in the PREDIMED nutrition intervention trial. BMC Med. 2013;11:164. Published 2013 Jul 16. doi:10.1186/1741-7015-11-164

Rohrmann, S., Faeh, D. Should we go nuts about nuts?. BMC Med 11, 165 (2013). https://doi.org/10.1186/1741-7015-11-165 —

Harnly J. Importance of Accurate Measurements in Nutrition Research: Dietary Flavonoids as a Case Study. Adv Nutr. 2016;7(2):375‐382. Published 2016 Mar 15. doi:10.3945/an.115.010470

Johnston BC, Alonso-Coello P, Bala MM, et al. Methods for trustworthy nutritional recommendations NutriRECS (Nutritional Recommendations and accessible Evidence summaries Composed of Systematic reviews): a protocol [published correction appears in BMC Med Res Methodol. 2019 Dec 17;19(1):240]. BMC Med Res Methodol. 2018;18(1):162. Published 2018 Dec 5. doi:10.1186/s12874-018-0621-8

Garza C, Stover PJ, Ohlhorst SD, et al. Best practices in nutrition science to earn and keep the public’s trust. Am J Clin Nutr. 2019;109(1):225‐243. doi:10.1093/ajcn/nqy337

Johnston BC, Seivenpiper JL, Vernooij RWM, et al. The Philosophy of Evidence-Based Principles and Practice in Nutrition. Mayo Clin Proc Innov Qual Outcomes. 2019;3(2):189‐199. Published 2019 May 27. doi:10.1016/j.mayocpiqo.2019.02.005
Satija A, Stampfer MJ, Rimm EB, Willett W, Hu FB. Perspective: Are Large, Simple Trials the Solution for Nutrition Research?. Adv Nutr. 2018;9(4):378‐387. doi:10.1093/advances/nmy030

Ioannidis JPA. Unreformed nutritional epidemiology: a lamp post in the dark forest. Eur J Epidemiol. 2019;34(4):327‐331. doi:10.1007/s10654-019-00487-5

Giovannucci E. Nutritional epidemiology: forest, trees and leaves. Eur J Epidemiol. 2019;34(4):319‐325. doi:10.1007/s10654-019-00488-4

Lappe, Joan M, and Robert P Heaney. “Why randomized controlled trials of calcium and vitamin D sometimes fail.Dermato-endocrinology vol. 4,2 (2012): 95-100. doi:10.4161/derm.19833

Ioannidis JP. Implausible results in human nutrition research. BMJ. 2013;347:f6698. Published 2013 Nov 14. doi:10.1136/bmj.f6698

Blumberg J, Heaney RP, Huncharek M, Scholl T, Stampfer M, Vieth R, et al. Evidence-based criteria in the nutritional context. Nutr Rev. 2010;68:478–84. doi: 10.1111/j.1753-4887.2010.00307.x. [PubMed]
Heaney RP. Nutrients, endpoints, and the problem of proof. J Nutr. 2008;138:1591–5. [PubMed]
Heaney RP, Weaver CM, Blumberg J. EBN (Evidence-Based Nutrition) Ver. 2.0. Nutr Today. 2011;46:22–6. doi: 10.1097/NT.0b013e3182076fdf. [CrossRef
Biesalski HK, Aggett PJ, Anton R, Bernstein PS, Blumberg J, Heaney RP, et al. 26th Hohenheim Consensus Conference, September 11, 2010 Scientific substantiation of health claims: evidence-based nutrition. Nutrition. 2011;27(Suppl):S1–20. doi: 10.1016/j.nut.2011.04.002. [PubMed]