NOTE: An earlier, May 2020 look at this issue can be found here: Significant changes necessary in order for dietary intervention studies to be causal and replicable
“Inconsistent and contradictory results from nutrition studies conducted by different investigators continue to emerge, in part because of the inherent variability of natural products, as well as the unknown and therefore uncontrolled variables in study populations and experimental designs.” — The Challenge of Reproducibility and Accuracy in Nutrition Research: Resources and Pitfalls
A high-profile 2019 nutritional study headed by NIH research section head Kevin Hall has come under increasing fire for data and protocol shortcomings which make it a teachable moment into why dietary interventipns so often fail to be useful and informative.
The issue was raised widely when research watchdog, Retraction Watch, noted that the paper published in Cell Metabolism— “Ultra-Processed Diets Cause Excess Calorie Intake and Weight Gain” — was accompanied by an editorial that urged caution in interpreting the results.
In addition to broad attention in general media outlets, Retraction Watch on January 22, noted that the study “[H]as been cited 164 times, according to Clarivate Analytics’ Web of Science, giving it ‘highly cited paper’ and ‘hot paper’ designations. (The paper has also been subject to a correction in October of last year.)”
The first detailed expression of concern for the Hall paper came more than two years ago from Edward Archer in PubPeer in which he detailed four methodological errors.
More recently, on January 21, 2021, two posts — one from Nick Brown and another by Brown in collaboration with Ethan and Sarah Ludwin-Peery — raised numerous questions about data integrity and experimental protocols.
Fundamental Errors in Hall Study Are Common in Most (If not all) Dietary Intervention Studies
The Hall study is non-replicable on its face because it does not provide a list of ingredients (reagents) and procedures (recipes) that would allow a third-party investigator to accurately reproduce it in an independent study.
Second, the Hall study’s results are inconclusive because the Ultra-Processed Foods (UPFs) in the Hall study are cross-contaminated by Plastic-Derived Chemicals (PDCs) such as Bisphenol A and phthalates which exhibit physiological effects similar to those being measured by the investigation (See charts farther down.)
Fundamentally, the Hall study fails because it has failed to observe standard and long-accepted laboratory practices that are basic to the scientific method in general and bench-science standards specifically. (NOTE: Citations and other details on this and the following statements and assertions can be found as part of this study and at this site.)
First of all, in a proper scientific experiment, there is a control phase and an experimental phase. In this case, the control phase would be unprocessed food and the experimental, Ultra-Processed. To posit a valid conclusion, the two legs of the study must be as different as possible.
However, the “UN-processed” stage of the Hall study is contaminated both by the presence of PDCs and the inclusion of foods which contain unknown food additives. PDCs are a hallmark of food processing which means that extreme care must be taken to avoid their unknown contamination effects. These effects can be characterized as adding “noise” to the data. In general, the noisier data is, the greater the likelihood that conclusions will be inaccurate or invalid.
In a dietary intervention studies — as in standard bench experiments — proper experimental design requires that the consumed substances must be treated as reagents with sources and other provenance details fully disclosed to investigators who may attempt to replicate the results. If you do not know the precise composition of your reagents, then you must assume contamination. And contamination of reagents, as noted above, contaminates results.
Regrettably, the Hall study, and other recent dietary intervention studies tolerate excessive contamination which, in turn, creates noise and uncertainty. The use of statistical methods designed to tease reasonable conclusions from the noise offer false comfort that a study conclusion is valid.. There are no certainties in those methofs — only probabilities — that the correct “voice” is the one plucked from the cacophony.
It is far better to start by eliminating as many voices from the room as possible. This is especially relevant since UPFs and PDCs speak with the same “accents” as defined by their metabolic effects.
Ultra-Processed Foods and Plastic Derived Chemicals produce very similar effects
A post-trial examination of another recent published study — revealed yet another fatal confounding factor: the effects of PDCs and UPFs are nearly identical and completely confounded any causal conclusion that study or Hall’s might want to draw.
This cross-contamination phenomenon is increasingly vital given the rise in micro- and nano-plastics in the food chain and environment.
A closer examination of the data also shows the ubiquitous PDC, Bisphenol A, affects ghrelin, leptin, and adiponectin in consistent ways as the Hall study recorded for Ultra-Processed Foods.
In addition, various disease/syndromes and clinical indicators also show parallels between Plastic-Derived Chemicals and Ultra-Processed Click this link to download Excel spreadsheet with full-sized text
It is not easy to bar PDC noise from the experimental room, but methods so exist (some described here and here) to eliminate most of it. It is scientifically careless not to make every effort to reduce noise.
An examination of a single meal in the Hall study menu as presented in the Supplemental Materials offers a telling glimpse to many of these contaminants and their variations from valid scientific practices.
Notably nearly every meal in that supplemental menu document suffers from significant cross contamination by UPFs PDCs. It is little wonder that the amount of weight gain cited in the Hall study was puzzling and equivocal according to the analyses by Nick Brown and Ethan & Sarah Ludwin-Peery
First of all, the yogurt will have undergone substantially more processing than whole milk. A look at the meal “Unprocessed Menu, Day 2: Lunch” in which “skim milk” is presented as “unprocessed” indicates a lack of understanding about dairy products in general and that skim and reduced-fat milk undergo more processing steps, many of which involve plastic contact.
While it is true that many Plastic Derived Chemicals (PDCs) are lipophilic and may be more prevalent in higher-fat foods, this is yet another ignored unknown in this menu presentation.
Second, the Hall material does not indicate which milk-fat content of the Fage brand yogurt served.
Fourth, while not specified, the Blue Diamond walnuts most likely came packaged in a plastic container which can leach PDCs. In addition, shelled nuts undergo processing steps that expose them to plastic surfaces.
Fifth, no washing or preparation measures were disclosed for the fresh fruit.
Was the rinse tap water or carbon-filtered? Was the fruit in contact with plastic bowls, cutting boards or other items? Was an attempt made to reduce the quantity of polymer/wax-based materials used to help- preserve the fresh apple’s moisture level? Were the fresh fruits certified organic which reduces potential pesticide contamination? Where were fresh fruit and vegetables acquired?
Sixth, how were the preparation and eating utensils and dishes washed and handled? Were they rinsed in carbon filtered water to remove any cleaning residues from detergents, ethoxylates and other contaminants?
Eighth, commercial table salt contains unknown additives including iodine supplements and anti-caking agents of undisclosed identity or quantity. In addition, common table salt is contaminated in widely varying amounts by microplastics, either from processing or environmental contamination. A preferred substitute for a dietary intervention would be NaCl from a reputable laboratory supply company.
Ninth, was the olive oil first cold press (no chemical extraction), processed with plastic-free apparatus and packaged in glass?
Finally, none of the meals presented in the supplemental, materials documented procedures, sources of materials, preparation details. In addition, no data on the weight or volume of the foods served and the error tolerance. It’s also worth noting that all quantities should have been recorded in grams. This avoids meniscus errors for liquids as well as packing errors for granulated substances such as sugar, flour, spices etc.
Partial blame may rest with NOVA
The Hall study notes that it followed the the NOVA classification as a guide for defining Ultra-Processed Foods.
Notably, that 2010 proposal admits that its categories are vague and often equivocal. Significantly, NOVA does not include the almost unfathomable number of FDA-approved food additives, most of which are not required to be listed on a food label. Also not included in NOVA is the presence of PDCs that leach from packaging materials, processing, and food preparation items.
NOVA also acknowledges that the simply act of cooking is processing. This means that the only foods that are UN-processed are those eaten raw.
Black Box Meals and Non-Replicability
All of the meals in the Hall study are “Black Box Meals” which are standard in dietary intervention investigations.
Black Box means that the investigators in the study have used third parties to prepare the means and have not supervised, directed or participated in the selection, provenance and gathering of ingredients, or in the process of preparation/cooking and serving. As a result, no data or guidance exists for a valid replication.
Palatability and weight gain
In Search of a Replicable, Causal Dietary Intervention Trial
That situation further indicates that, in addition to the revised protocol changes we implemented, our study, Hall’s and every other dietary intervention study must incorporate the dosing of a relevant single compound of interest in order to establish causality and to enable clinically valid health decisions.
Nutrition science research bears additional burdens for rigor and reproducibility. One such burden is the “unscientific beliefs” that result from the casual familiarity everyone experiences with food….
Another burden is the expectations of the “best science” that understandably accompanies research findings with immediate applicability in daily health-relevant decision-making. There is no single solution to overcoming the challenges of obtaining quality evidence to support nutrition recommendations and nutrition policy…,
Solutions will likely require combinations of large and long-term RCTs (16), inclusion of dose–response relations in the design of observational studies, Mendelian randomization approaches in nutrition and other approaches.
Jessie P. Buckley, Hyunju Kim, Eugenia Wong, Casey M. Rebholz, Ultra-processed food consumption https://www.sciencedirect.com/science/article/pii/S0160412019317416
Exposure to phthalates and bisphenols in the US National Health and Nutrition Examination Survey, 2013–2014,Environment International, Volume 131, 2019, 105057, ISSN 0160-4120, https://doi.org/10.1016/j.envint.2019.105057.)
Guasch-Ferré M, Bulló M, Martínez-González MÁ, et al. Frequency of nut consumption and mortality risk in the PREDIMED nutrition intervention trial. BMC Med. 2013;11:164. Published 2013 Jul 16. doi:10.1186/1741-7015-11-164
Rohrmann, S., Faeh, D. Should we go nuts about nuts?. BMC Med 11, 165 (2013). https://doi.org/10.1186/1741-7015-11-165 —
Harnly J. Importance of Accurate Measurements in Nutrition Research: Dietary Flavonoids as a Case Study. Adv Nutr. 2016;7(2):375‐382. Published 2016 Mar 15. doi:10.3945/an.115.010470
Johnston BC, Alonso-Coello P, Bala MM, et al. Methods for trustworthy nutritional recommendations NutriRECS (Nutritional Recommendations and accessible Evidence summaries Composed of Systematic reviews): a protocol [published correction appears in BMC Med Res Methodol. 2019 Dec 17;19(1):240]. BMC Med Res Methodol. 2018;18(1):162. Published 2018 Dec 5. doi:10.1186/s12874-018-0621-8
Garza C, Stover PJ, Ohlhorst SD, et al. Best practices in nutrition science to earn and keep the public’s trust. Am J Clin Nutr. 2019;109(1):225‐243. doi:10.1093/ajcn/nqy337
Ioannidis JPA. Unreformed nutritional epidemiology: a lamp post in the dark forest. Eur J Epidemiol. 2019;34(4):327‐331. doi:10.1007/s10654-019-00487-5
Giovannucci E. Nutritional epidemiology: forest, trees and leaves. Eur J Epidemiol. 2019;34(4):319‐325. doi:10.1007/s10654-019-00488-4
Lappe, Joan M, and Robert P Heaney. “Why randomized controlled trials of calcium and vitamin D sometimes fail.” Dermato-endocrinology vol. 4,2 (2012): 95-100. doi:10.4161/derm.19833
Ioannidis JP. Implausible results in human nutrition research. BMJ. 2013;347:f6698. Published 2013 Nov 14. doi:10.1136/bmj.f6698