From time to time in this column, I have tried to explain what makes research findings reliable and valid, and what a person with little or no scientific training should look for to make that judgment. I realize this is not the most exiting topic, but today, more and more sources – books, articles and particularly websites – present research results claiming to be scientifically obtained when they are not. And people, who are supporting an ideological cause or fear some technology or product they want eliminated, use those sources to justify their crusade. So its time for a research critique update.
In explaining the difference between research methods that produce valid and verifiable results and those that do not, I have found it useful to distinguish between “coercive” and “fact-finding” research.
Finding valid facts is what scientific research is all about. The studies are designed to prevent value judgments from influencing results and are unbiased as to the outcome. As an example, when testing to determine if there are more health risks associated with one group exposed to a suspected technology, chemical or environmental condition and another group not exposed, every effort is made to ensure that the two groups are identical except for exposure. The hypothesis tested that is there is no difference between the groups, and that the statistical methods employed are to confirm or refute that hypothesis.
Coercive-research, in contrast, is designed to demonstrate that there is a statistical difference in order to support an activist group’s effort to constrain or eliminate, say, a technology or a chemical. This distinction may not seem important, but, statistically, there is a world of difference.
If the test designers are motivated to show a technology is dangerous or a product unhealthy, then they can construct a design that will allow that interest to influence the results and show a difference. It can be done intentionally or through ignorance by the way groups are selected, the number included in each group, by the statistical method used to analyze the results and the statistical parameters chosen to claim significance.
The studies often ignore the need for a plausible physical, biological or physiological mechanism to support such a difference, and there is seldom any attempt to replicate the findings. I would classify a large percentage of research results identified on the Web in support of activist causes as coercive, and people without research training can’t tell the difference.
When one reads a letter to the editor stating that anyone with a computer can access numerous studies proving whatever health effect or technological harm the letter writer wishes to support, the readers should ask two questions: Have the referenced results been obtained from scientifically approved methodology? And are there other studies that refute the results to which the letter writer is referring? The second question is particularly important because activists with a cause pay no attention to evidence that does not support their convictions.
So how does a layperson find answers to the first question? Try and determine who did the research. The most reliable sources are from laboratories at research universities where scientifically approved methodology is required. The most suspect are from labs financed by industry, environmental groups or other activist organizations. As would be expected, those having a stated purpose to support a cause will sponsor research supporting that cause. Also, be very skeptical of research results online under names like “the Institute for Responsible Technology” or “independent science news” that sound scientific but aren’t. Find out if activist groups are the ones supporting the website, and if they do, there is a high probability the research is coercive and methodologically flawed.
Finally, where does one find answers to the second question? Unbiased information about both the benefits and the risks of an issue are obtained from research conducted at university laboratories and are published in peer-reviewed scientific journals and on their websites. Peer review also prevents methodologically flawed studies from acceptance in these publications. I can’t list all the journals, but here are the ones I subscribe too: Science, sponsored by the American Association for the Advancement of Science; Nature, its British counterpart; MIT Technology Review, sponsored by the Massachusetts Institute of Technology; Issues in Science and Technology, sponsored by the National Academies of Sciences; and for economics, the British journal The Economist. If one doesn’t have access to the websites of these journals – some are limited to subscribers – they are usually available at Magpies Newsstand Café at 707 Main Ave. or Durango Public Library.
There are other scientifically sponsored peer-reviewed journals devoted to specific disciplines, but the ones listed are the most respected of the general science publications and most available to the public. If one is interested in obtaining a scientific understanding about the risks of genetic modification, contrails, cellphone towers, nuclear radiation, vaccinations, insecticides in food or other hot-button fears, then these journals are your most reliable source.
Garth Buchanan holds a doctorate in applied science and has 35 years of experience in operations research. Reach him at gbuch@frontier.net.
