What Makes Health and Wellness Claims Trustworthy
Introduction
Each day, the average member of the public is overtly and covertly bombarded with countless health and wellness claims from numerous directions. Whether it be an acquaintance, an influencer, a public intellectual, or a multinational corporation, it seems that everyone these days is trying to shape our understanding of or our approach to health and wellness.
It goes without saying that some claims are more credible than others. Most claims are not even specific enough to be worthy of your attention: an influencer claiming that a certain hair product “saved their hair” should never be the basis for a product purchase or doctor consultation.
In fact, such claims are often deliberately vague, avoiding the specificity that could land someone in legal trouble if the claim were not backed up by real-world results. At first glance, other claims appear specific and evidence-based but are, in reality, flimsy and hidden behind a veil of scientific jargon.
This blog post will introduce some key concepts that undergird credible health and wellness claims. After reading this post, you should have a better understanding of common terms that come up over and over in trustworthy scientific claims, as well as the scientific logic that underpins them.
Hallmark #1: Control groups
Scientific studies are designed to test hypotheses. Put simply, a hypothesis is a proposal about how the world might work that has not yet been put to the test.
For research regarding health and wellness, a hypothesis typically involves the claim that using an intervention (e.g., treatment) can lead to better health outcomes (e.g., disease cure, improved quality of life).
A control group is an essential component of effective clinical investigations, but is not exactly what its name would suggest. A control group is not a board or committee involved in governing trials. Rather, control groups are a subset of the study population on which the intervention being investigated is not used.
To explain, let’s imagine the situation in which a research outfit would like to test out a new weight-loss drug using 100 study participants. Giving the weight loss drug to all 100 individuals and observing any changes in the 100 participants’ bodies over time is one way to conduct the investigation.
However, dividing the study participants into two equal groups (fifty participants each) and giving the new weight-loss drug to only one of them would be a much better approach.
Why is that? Setting up a group of study participants who are not taking the drug (a control group) allows for direct comparisons between individuals taking the weight loss drug and those not taking the drug.
In other words, control groups establish a baseline that allows direct, apples-to-apples comparisons between a world in which an intervention is applied and a world in which it is not. Without control groups, it is easy for study results to become biased. Oftentimes, these biases make the new intervention look more effective than it really is.
To provide an example of how a control group prevents bias, think back to the example of the new weight loss drug. As you may know, individuals tend to gain weight during the winter and tend to lose weight during the summer. Weight fluctuation throughout the year happens because of seasonal fluctuations in activity levels, cultural events, etc.
Seasonal trends in body weight are one reason a manufacturer might be tempted to start a clinical trial in wintertime (when participants are likely heavier) and conclude it six months later (when participants are likely to be skinnier). Even if the drug does nothing, participants would likely be thinner after taking it simply because of the season.
However, incorporating a control group into the experimental design would help us understand whether the drug is actually effective. Participants in the control group, who do not take the new weight loss drug, may naturally lose weight as the weather warms up.
If the people taking the weight loss drug lose no more weight than those in the control group, researchers would (correctly) conclude that the new weight loss drug is ineffective.
Alternatively, patients taking the new weight loss drug may lose more weight than those not taking the drug. This result would suggest that the new drug contributed to the weight loss (researchers would typically have some understanding of “how” the drug works before beginning the investigation).
The above example shows how including a control group allows the scientific community to plainly see the effect of the new diagnostic tool or treatment itself, “controlling for” the impact of seasonality, expectations, and other factors on study results.
Hallmark #2: Randomization
Randomization is the process by which the subjects of an investigation (typically patients) are divided into two or more groups through random sorting.
It is very important that participants are assigned to study groups at random, not through some predetermined logic or the decision-making of those who organize or run the study.
Failure to randomly assign participants to groups is another source of bias in study results. The reason is that, even if the study design includes a control group, baseline differences between participants in the control and intervention groups can influence the results of the investigation.
Let’s say, for example, that researchers wanted to test the ability of a mobile application that encourages healthy habit formation. Specifically, the application is designed to help users exercise more consistently by giving them reminders and rewards for doing so.
To test the application’s value, it would make sense to gather a group of individuals and observe how their use of the app over weeks and months influences their exercise habits. As discussed in the prior section, it would also make sense to establish a control group of individuals not using the app during the same time period.
Without randomly assigning participants to the two groups, it is possible that individuals with different exercise-related attitudes and experience are unevenly distributed between them. Naturally, this could skew the results of the investigation.
For example, assume for a second that participants with firmly established exercise habits are overrepresented in the group of participants who use the new app. It is possible that these pro-exercise participants are more responsive to the encouragement and rewards of the app and, as a result, develop even stronger exercise habits over the course of the study.
It is also possible that, because these pro-exercise participants already have strongly ingrained exercise habits, they are less responsive to the encouragement of the app. After all, it is probably not realistic to expect that an app can encourage someone who already runs one hour a day to run an additional hour just because the app tells them to.
In either case, the core problem is the same: it is impossible to know for sure the way in which baseline differences in study participants (in this case participants’ exercise habits) might influence experimental results. That is why we must strive to create two study groups with an even mixture of exercise-related habits, age, sex, etc.
This example illustrates the primary reason randomization is used: to distribute participants across the experimental groups in a statistically unbiased way. A balanced distribution helps create a simpler, more compelling “story” about a new technology’s impact when it comes time to interpret study results, making the resulting claim more trustworthy.
Hallmark #3: Blinding
Put simply, blinding is the process of withholding group assignments from experimenters and participants to protect the validity and reliability of study results.
Blinding of participants often entails hiding whether they have been assigned to the intervention group (which receives a novel treatment or other change) or the control group (which does not, for purposes of comparison).
Blinding of experimenters (scientists) also takes place. In this case, they do not know which group any given participant belongs to. Studies in which both experimenters and participants are blinded are considered double-blinded.
In the classic case, double-blinding is achieved in investigations of pharmaceuticals through the use of placebo medicines. The medicines given to the control and intervention groups are made to look identical, but only those given to the intervention group contain the active ingredient under investigation (something that neither doctor nor patient can detect).
Without blinding, study participants (patients) might change their attitudes or health-related behavior in response to knowing whether they are receiving the new technology. To illustrate this point, consider a study designed to test the effect of a new diabetes drug on long-term blood sugar levels.
Without blinding, diabetic patients who know they are receiving the new medicine might become optimistic and feel empowered to engage in more health-conscious behaviors. Alternatively, patients who know they are receiving it might become complacent about blood sugar control, assuming that the drug will manage it on their behalf.
Failure to blind experimenters can also negatively influence the reliability of study results. For example, not blinding a health care provider can unduly influence either how they interact with a patient or how they judge outcomes during the investigation. In the latter case, health care providers often conclude that an experimental drug or device achieves better outcomes than it actually would if used in the real world.
That is why blinding of both investigator and study subject is a core component of investigations whose results can be believed.
While it can be difficult to introduce blinding into investigations due to procedural or logistical challenges, it is still worth attempting; an investigator’s failure to blind when it is perfectly feasible should make you question the scientific rigor of the study and the health and wellness claims that arise from it.
Hallmark #4: Pre-reporting of methods and expectations
Many people are familiar with the role that scientific journals play in the recording and reporting of an investigation’s results. However, reporting that takes place before an investigation begins may be just as important to trustworthy health and wellness science.
One component of pre-reporting is the statistical power calculation. Power calculations estimate how many participants a study needs in order to detect the effect it is looking for, if that effect exists.
For example, if investigators expect that the new technology will have a large impact on study subjects’ health and wellness, fewer patients would need to be recruited into the study to observe that effect. In contrast, more study subjects would need to be recruited if investigators assume that the effect of using the new technology is more modest.
Power calculations are a key tool for avoiding the situation in which a new technology actually does influence participants’ health and wellness outcomes, but the investigation concludes that it doesn’t (what is known as a “type II error”).
Another component of pre-reporting occurs when investigators state what types of data will be collected during the study, what analytical tests will be run on that data, and what results they expect to observe once the analysis is complete.
Pre-reporting this range of information commits investigators to a hypothesis (a prediction about study results) before the investigation is conducted. In other words, it reduces the temptation for investigators to change their hypothesis or analysis after the experiment is over and investigators know what really happened.
Pre-reporting also prevents investigators from “fishing” for noteworthy results. For example, as the field of statistics has become more and more advanced, it has become possible to run an infinite variety of statistical analyses on the results of any one investigation. This can create the temptation for investigators to run many (potentially inappropriate) statistical tests while searching for something noteworthy to report.
Say, for example, that an investigator runs a hundred different statistical tests on a given set of data. It is likely that, by chance alone, a handful of these tests will produce a result that, viewed in isolation, would seem unlikely to occur by chance (suggesting that a noteworthy finding has been unearthed). When so many tests are conducted, such flukes become expected.
That is one reason why investigators are discouraged from running unnecessary tests during the analysis stage. Instead, investigators are expected to pre-report the limited, highly focused statistical analyses they will carry out to test their specific hypothesis or hypotheses.
Putting reasonable boundaries on the range of statistical tests that will be carried out is one way in which pre-reporting of methods can reduce type I errors, which occur when an investigation finds that a new technology has an impact on health and wellness when it is, in fact, ineffectual.
Hallmark #5: Investigator expertise
With sufficient funding and adequate scientific facilities, there are virtually no barriers to carrying out a study of a novel health or wellness technology. That does not mean, however, that every scientist is well-equipped with the knowledge and experience needed to plan and execute a trustworthy investigation.
Rather, it is important that scientists with expertise in a given topic are put in charge of an investigation on that topic. For starters, experts are more familiar with the way a specific health or wellness issue is currently investigated. This knowledge allows them to design studies in which the new technology is tested in a fair and thorough fashion.
Furthermore, experts who have done research in a particular domain are also familiar with the challenges of running experiments in it. For example, research into some areas of health and wellness (e.g., weight loss) is especially susceptible to the attitudes and behavior of study participants. An appropriate study design should take these factors into account.
Lastly, the choice of investigators sometimes reflects the influence of industry. For example, manufacturers of new technologies are sometimes involved in selecting the investigators who will test their technology. A manufacturer’s decision to work with some investigators but not others is not suspect in itself, but it can suggest an effort to avoid rigorous assessment when investigators likely to produce flattering results are chosen.
In some cases, manufacturers actively seek out investigators with a track record of accepting funding from various manufacturers to perform studies across a wide range of unrelated topics. These investigators would not be considered experts in any particular field, but rather vehicles through which manufacturers can produce and publish scientific results that seem trustworthy.
These pliable investigators may be less resistant to the efforts that manufacturers might apply, implicitly or explicitly, to ensure that the investigation paints their technology in a favorable light.
Hallmark #6: Funding transparency
Funding transparency describes the process by which investigators publicly report both their relationships with technology manufacturers and the financial contributions that made the planning and execution of a given investigation possible.
Closely related to investigator expertise, funding transparency is key to judging whether a given investigator or institution can be trusted to conduct the study in a scientifically valid and credible way.
For example, a concerning situation that sometimes arises is one in which a new technology is evaluated by scientists directly employed by its manufacturer, using funding and facilities the manufacturer provides.
Compare that to a situation in which a manufacturer merely grants the investigators access to its new technology, with zero influence on how the investigation is otherwise funded or how it is carried out. This latter case describes the ideal model of academic research at colleges and universities, where scientific grants from neutral government agencies (not technology manufacturers) are the primary source of funding.
Imagine the case in which there are two studies: one in which the manufacturer had heavy influence, and another in which it had little to no influence. Even if the results of the two studies are identical, the study with minimal manufacturer influence is typically regarded as more trustworthy.
That is one reason why any report of an investigation’s results, whether in an academic or marketing context, should disclose the funding structure of the investigation and the relationship between manufacturer and investigator.
Without such financial disclosures, the results presented to the public should be viewed with a great degree of suspicion.
Wellness Takeaways
Scientific research on health and wellness is a complex undertaking, fraught with pitfalls and biases. Investigators may spend years developing an appropriate method to assess a new technology, working to ensure that their study designs satisfy basic standards for scientific rigor and credibility.
That being said, not all research on health and wellness is carried out with the same commitment to seeking scientific truth. Much of the health and wellness research that makes its way into publications, advertisements, and social media is, upon closer inspection, less rigorous than it may seem.
While the checklist below is by no means exhaustive, be sure to consider these factors the next time you are taking a deeper dive into any health and wellness claim.
Did the investigation use a control group? If not, what is the investigator’s justification for not doing so?
Was randomization used to assign participants to each study group? If not, what is the investigator’s justification for not doing so?
Was blinding used to ensure that participants were not aware of which group they were assigned to? If not, what is the investigator’s justification for not doing so?
Before beginning the investigation, did the investigators report their specific hypothesis and the statistical methods they planned to use to test it? If not, what is the investigators’ justification for not doing so?
Was the investigation carried out by well-regarded, experienced subject-matter experts? What other kinds of research do they participate in?
Was the source of research funding disclosed? What type of support (if any) did an industry partner offer the investigators or institution?
What is the relationship between the technology manufacturer and the investigators? Does it seem likely that the technology manufacturer had a large influence during the set-up, execution, or analysis stages of the investigation?
Thank you for your time and attention!
Are you looking for personalized wellness insights on this and other topics?
