Fact or Fiction: Tips to Evaluating Scientific Studies and Evidence

Image courtesy of Priscilla Du Preez via Unsplash.

Have you ever noticed that scientific evidence seems to say something different every day? It’s particularly evident in our current COVID-19 situation, with conflicting information circulating around the prevalence, danger of, care for, and prevention of the virus.  

But COVID seems to be highlighting an ongoing issue—who or what can we trust when it comes to scientific studies? Keeping up with evidence is a sizable challenge. Many people make it their full-time jobs, and barely manage to scratch the surface. That said, even a non-expert (you) can take some simple steps to critically evaluating evidence that will help you manage your health decisions.

Starting Your Search 

Set a defined question.

This is the best way to keep yourself from meandering into any topic that seems interesting. Set the questions you want to answer and pass every topic in your research range by the filter “is this helping me answer my main question?”

Set a time limit for yourself.

It’s very easy to get lost in a research rabbit hole. When you start to dive into a research topic, set a time limit in which you have to make a decision. Bigger decisions deserve more time and attention, but they still deserve boundaries.   

It’s also highly stressful on your brain to be questioning all of your assumptions all the time. Your brain is designed to find patterns, create beliefs and habits from those patterns, and reinforce them over time. Ambiguity and radical shifts in thinking are incredibly stressful on that system. For the sake of your mental health, don’t wait until you have all of the information in existence to make a choice to act on.

Avoid the urge to label things good or bad—there are always multiple sides to the story.

Everything has an up side and a down side. Take tap water for example. The introduction of chemicals like chlorine prevents the spread of deadly pathogens in our water systems (that’s generally considered a good thing). Chlorine is also toxic at certain levels (generally considered a bad thing), which is essential to its disinfecting quality.  

Even in small amounts, it’s a chemical that builds up in our systems over time and creates stress on our bodies and ecosystems. This, like many of our current health and ecosystem challenges, does not have a clear right and wrong, good and bad divide. It’s a trade off, a decision we as a society made that has both positive and negative consequences.

Honing Your Resources

Evaluate your experts—there’s no such thing as an unbiased opinion.

Human beings are not built to be unbiased. We naturally form opinions and hypotheses that our brains will look for evidence to confirm (confirmation bias). It takes conscious effort to refute your own opinions and beliefs.  

That means even the experts are working within a system that might inadvertently confirm their own beliefs. Getting your information from a trustworthy source is therefore key.

So is having an opposite expert in your lineup, someone who you know does their research, but forms opinions that differ from yours and/or the other experts you’ve chosen. Use them to challenge and strengthen the conclusions you draw.

Evaluate the funding.

Science is a business. Studies can only answer questions that have been asked by a scientist with the means (money, time, effort, technology) to answer them. This means that studies that turn out less profitable outcomes are less likely to be invested in.

The lengthy steps to completing a study are even longer if there’s a possibility the results will run counter to currently accepted theories, or pose a risk to long-held beliefs that promote certain medications, practices, or lifestyle changes. Investors don’t want a portfolio full of risky investments where they can’t, with some level of certainty, predict the outcome. They also invest in the hope to profit from the result.

This isn’t evil—it’s capitalism. But it’s another reason to follow the money, and pay attention to what or who is funding the research.

Evaluate the study—each type has its advantages and disadvantages.

There are many different kinds of evidence that are considered with varying degrees of acceptance. Whenever you see the words “new evidence suggests” or “new study says,” they can be referring to multiple types of evidence. Clicking through to find out what type of “study” was conducted, before even reading past the headline, gives a huge amount of context to the claims.

Non-Human Studies

When looking at mathematical models and in-vitro studies, it’s wise to remember in theory there is no difference between theory and practice, while in practice there is. There’s no way to predict all of the factors that will be in effect in a complex system like the human body or an ecosystem. Many things that work out on paper fall apart when applied to real life. Sometimes we can figure out why; other times we’re left only with speculation.

Large Studies and Trials

These are great for generalities—they seek to find results that are applicable to the largest number of people and to smooth over or eliminate outliers for the sake of studying the average. However, they find these generally applicable rules at the expense of finding variations and outliers. When varying results are found across studies, it’s considered a weakening of the evidence, rather than a sign that they’ve simply asked the wrong question (or at least a question that isn’t specific enough). 

Anecdotal Evidence and Single Cases

These are great for exploring possibilities, but usually lack control—meaning there could be any number of factors that contributed to the result that was observed. They are far easier and cheaper to produce, but they leave room for questions and possibilities. It’s when these are assembled en mass over time that they become a stronger form of evidence.  

Before we had “modern” medicine, anecdotal observations were the foundation of healing traditions. Bear in mind these were thousands of years worth of anecdotal evidence. While these healing practitioners couldn’t point to the chemical mechanism that caused the healing effect, they did know that if you contacted certain points on the body, ate certain foods, spices, and herbs, or addressed certain mental challenges that they would create somewhat predictable outcomes for the person’s health. Anecdotal evidence over thousands of years can be considered equivalent to our modern understanding of scientific evidence, even though current healthcare systems give it less weight.

Understanding the Results

“Significant result” probably doesn’t mean what you think it means.

In science, “significant” does not mean large, impressive, or great. When a study uses the word “significant,” it means statistically significant. Statistical significance is a way of measuring the likelihood that an event or effect happened because of random chance. For a clinical study to be considered valid, there must be a 95% or greater statistical significance—meaning that there is less than a 5% chance that the results were sure to be a random occurrence. 

In order for this to happen, the result can be greater (a larger response) and/or the number of people included in the trial who experienced the same result can be higher.

When you read studies yourself, the chance that something is not a random occurrence is called the p-value (the 5%, often written as p = 0.05). Take note of that before diving into the rest of the outcomes.

The absence of evidence is not evidence of absence.

There are questions that we haven’t thought to ask. There are also some hypotheses that we simply don’t have the technology or know how to test yet. For example, we can observe that gravity exists and understand its effect on large objects, but to this day, we can only hypothesize about what causes it. It’s one of the great unsolved mysteries of physics.

This is all to say, our body of scientific knowledge is (and will always be) incomplete. There are limits to our collective knowledge that have not been overcome yet. Just because we haven’t studied it, does not mean it doesn’t exist.

It also takes 15 to 20 years for information to go from scientific validity to clinical application in the healthcare system. This is a multifaceted issue that even the age of the Internet has yet to solve, and can mean that the most cutting-edge research and evidence doesn’t reach the masses for nearly a generation.

There’s always more to the story.

If there’s one thing I want you to take away from this article it’s this: There is always more to the story.

Whenever you hear one new evidence that seems to blow an old theory out of the water… there’s always more to the story.

Whenever you read about experts disagreeing about a core scientific theory… there’s always more to the story.

Especially when a doctor, scientist, journalist, politician, or any other “expert” claims that there’s a clear, obvious, incontrovertible, uniting truth of the universe… there is always more to the story.

This is just the tip of the iceberg when it comes to evaluating scientific evidence, but it can help you get a start on identifying real vs. fake claims for yourself. Keep in mind that not only are there many factors that go into scientific research, there are also many factors that impact how a treatment or suggested use will work on your body. For personalized guidance, I’m here to help. Contact me to set up an exploratory appointment where we can walk through the best plan for your health.