Login | Cart (0)

10 Tips on Reading Research

Jade Teta ND, CSCS

The world of health fitness and fat loss is a mixed bag. It can be difficult to decipher what is real and what is not. Lately I have noticed a trend where individuals argue over who has the better sources, the most influential guru, or stands for the most noble cause. Because of this, health and fitness discussions can often resemble political and religious conversations. So, I decided to write my thoughts down about research, reading it, making sense of it, and some of the things I think it would be useful for those living the fat loss lifestyle to think about. Here are 10 points to ponder on health and fitness research.

1) Read studies NOT Abstracts. When possible read the actual study NOT the abstracts and NOT the interpretations of the studies from your favorite fat loss guru.

2) Find reliable unbiased and non-extreme resources. When it is not possible to read the actual study, a trusted guru is the next best option. By trusted we mean someone who you know through experience reads the research and is relatively unbiased (it is impossible to be completely unbiased in our mind) and avoids extremes.

3) Consider the study design. Population study?= Get curious, but realize there can be no causation link here and given all the variable in populations just file the info away as “interesting, but correlation means nothing but correlation”. Animal study?= Get even more curious especially if a population study hints towards the same findings, but understand this proves nothing. Human intervention trial? Now get a little more excited…this thing may have potential, but wait for several more well designed studies to corroborate.

4) A science background makes a BIG difference. If you have a background in a science it is extremely helpful. Knowing the mechanism of something being studied (i.e. the fact that the branched chain amino acid leucine has a unique metabolism and interacts with cellular signaling agents like mTOR or AMPK) can help predict or even explain study results. Understanding this is the reason many clinicians try certain agents for certain conditions without a research study guiding them.

5) Dont cherry pick. If you are really after the truth, you will go and read the studies that show the exact opposite of what you are so excited about. If you choose not to continue looking at both sides, it is a good idea to use words like “may” or “my bias” as opposed to definitive statements like “useless”, “never”, or “does not”.

6) There is rarely just one study. Look at the study in the context of the entire body of research on the subject. If 2 studies show no effect but 10 show a positive effect, that should sway your opinion a bit.

7) Take a holistic view. This is where things get tricky. Let’s say you are educated enough to know vitamin E is an antioxidant and even more savy to realize single anti-oxidants can become pro-oxidant in some situations (see how 4 above becomes important?). Now you read a study showing alpha-tocopherol has a negative impact in a smoker’s risk of lung cancer. Does this mean vitamin E is dangerous? Are the population studies showing vitamin E dietary intake through food as protective, wrong? Taking a holistic view means understanding alpha-tocopherol is one form of vitamin E among many and that in natural foods tocopherols come in a mixed form with gamma tocopherol being dominant NOT alpha tocopherol. Taking a broader view now allows you to see, that perhaps, you should now look for intervention trials using mixed tocopherols or gamma tocopherol and see if the same results apply.

8) You don’t know what you don’t know. There are the things you know that you know and also things you know you don’t know. But what about all the things you didn’t know you didn’t know? In reality, the things we know we know and the things we are aware we don’t know are a very small part of the total when compared to all the things we have not yet been exposed to and may not even know existed. This is part of the reason most researchers don’t speak in absolutes and use the word “may” a lot. Because research is an imperfect science and there are just too many things we don’t yet know, some argue that the current methodologies in research are too limited to analyze complex lifestyle interventions involving simultaneous changes in diet, exercise, behavior, and supplement changes.

9) What works today is the subject of research tomorrow. Research is a tool and should be used to evaluate and test our approach to problems NOT define our approach. If you see something that works in your clinical or personal experience, don’t dismiss it simply because a study does not confirm it. The most famous and successful clinicians and coaches get that way by combining their knowledge of the subject and research with what they see in the real world and then extrapolate ahead to generate innovative protocols and procedures that can become game changers.

10) Studies ignore the individual. Research studies can not and never have been able to account for differences in human metabolism. Studies regress to the mean which means they uncover the average response, and this is not always useful for an individual. In fact, in part of the statistical design of studies, outliers (which are simply individual participants that did not conform to the “right pattern”) are removed from the data sets. This practice assures any individual who had a pronounced negative or positive effect to the study intervention is not accounted for. Obviously this practice is often necessary since leaving the outliers in could skew the results. But hopefully you can see this is no small issue and should be considered in analyzing study results in depth.

A great example of this issue comes from research on magnesium supplementation for migraines. Most studies will show magnesium does not work for migraines, but clinicians see that it works very well clinically for a subset of migraine sufferers. Studies miss these individual effects because an individual with a genetic need for increased magnesium who suffers from migraines as a result will get “washed-out” of the data in a study of 100 other individuals for whom supplemental magnesium is not effective. Regression to the mean is the enemy of individualized medicine and is part of the reason for the gap between anecdotal support for a supplement vs. research debunking its use.

A Real-World Example:
A heavily reported study in the media called the POUNDS LOST TRIAL came to the conclusion that there were “no differences in changes in body composition, abdominal fat, or hepatic fat between assigned macronutrient amounts”. In other words, this trial gave differing prescribed proportions of fat, protein, and carbohydrate to participants and followed them over a two year period. So, they found no difference in weight loss based on changes in diet composition. But, looking at the study more carefully reveals this really cannot be inferred from this study for several reasons:

The actual differences in reported macronutrient intake (i.e. what the subjects actually ate) did not differ much. For instance, for protein, the difference was only 2%, not the 10% prescribed in the diet (BTW, many who work in the weight loss field, including me, would consider even the higher amount of protein, which was 20% of total calorie intake, low. My prescribed protein intake for weight loss seekers is 30% on the low end and 40% on the high end).

The urinary nitrogen levels, a marker of actual protein intake in the subjects, was not different. This suggests the protein intake between subjects was not different at all.

The two points of above illustrate just two of many problems with these types of studies. Many feel that these types of studies cannot adequately test what they are attempting to do to for several reasons. This study and all the issues other such studies suffer from was critiqued HERE. And you can get the actual study HERE.