In my previous article, I attempted to explain scientific inquiry through the use of experiments, observation and presentation. Of course, gathering and presenting your data and conclusions based thereon is only part of the process. In order for that knowledge to be published and made available to other scientists as well as all other enquiring minds, one must pass one last test: that of peer review.
This is probably the one area where science is most often attacked, and rightfully so. Peer review is one of the most important parts of science, and must therefore be rigorously scrutinised. That said, peer review is often also a soft target for those who for whatever reason wish to discredit science as a system for generating knowledge.
As I pointed out in the first article, scientists are first and foremost human beings, with all that implies. We have the same potential for personal character defects as everyone else. Just because scientists as a group are smart and well-trained doesn’t mean they can’t be nasty, backstabbing, conniving little cheats. There is such a thing as professional jealously, greed, lust for power and fame etc. Most scientists would like to pretend it doesn’t happen, but realistically, we must acknowledge that it does happen.
Enter the peer review system. This system is in place not only to ensure that the scientific papers being published are of a high standard, but also to limit the effect of the darker side of human nature on what is and isn’t published.
Contrary to the popular view of science, peer review doesn’t begin and end upon submission and subsequent publication of scientific papers. It begins much earlier than that. Like I mentioned in an earlier article, scientists don’t work in isolation. Your first line of review are usually your colleagues who work on the same project, or in the same department, or the same building or whatever else the case may be. Their input can be felt right from the get go. Scientists don’t just sit in coffee shops arguing for the sake of argument. They’re bouncing ideas off each other, checking facts, double-checking their reasoning, all in an attempt to refine their ideas, experiments or conclusions. This is one of the main functions of scientific conferences. It’s not just to present papers. It’s also an opportunity for peer review during every step of research, whether you’re still in the process of formulating a hypothesis, busy designing your experimental approach or ready to present your results.
Let’s say our scientist in the previous article passed those initial reviews and is ready to publish his paper. He’ll submit his paper to one of the relevant journals, let’s call this one Geology in Review. Once his article arrives, it’ll be read first by the editor, and then the editor will submit the article to other scientists who work in the same field as our scientist.
Previously, such reviews were done anonymously, but criticism of this approach led to most scientific journals changing this to an open process in which the names some or all of the reviewers are revealed. This was done to cut down on some of the above mentioned “dark side of human nature” events. These events were rightfully used for the wrong reasons by various elements to attempt to discredit the peer review process. However, it could be shown that because these events were picked up, it indicates that the process worked well, and is now improved to work even better.
Scientific research are often multi-disciplinary, meaning it requires inputs from more than one field of expertise. In cases where scientists from different disciplines worked together on one paper, the article will be sent to reviewers from both disciplines.
The job of the reviewer is to check the general quality of the work: whether sound scientific principles were followed, whether the conclusions are supported by the data, and whether the work was well researched and based on solid supportive work. Based on the comments received back from the reviewers, the article may be sent back to the original researcher to correct or improve. This happens much more often than you might think. It is exceedingly rare for any scientific papers to be published upon first submission.
This doesn’t mean that any work thus reviewed and published will be flawless. Remember that our knowledge is not complete. A good example would be Dalton’s paper on the nature of the atom. Dalton’s original paper was accepted, even though today we know that his conception of the atom was rather limited. It is up to other scientists who subsequently used his work to discover whether his research or ideas held merit or not, and often we discover that those initial findings were either wrong, or needed to be updated.
This is of course another criticism of science. “You guys are so often wrong!” That is of course a gross, and rather ignorant, oversimplification. Was Dalton wrong about the nature of matter? No. Was his model of the atom wrong? No. It was just incomplete. It was pioneering work that no one had ever done before and as such, we had limited knowledge on the subject. Dalton’s work was the first increment, but on his work, we now have whole new branches of science. Those scientists were not wrong. Their work just became obsolete in the same way that old man Benz’s 1886 “horseless carriage” eventually became a Mercedes-Benz S-class. Still essentially the same concept but vastly improved in so many ways to the original inventor it would look like magic.
That’s not to say that mistakes cannot be made. However, these mistakes are always found and corrected as new research is done.
So when will a scientific paper not be accepted?
1) When it fails to meet the technical publication standards of the journal
2) When there are flaws in the experimental design (ie, when the experimental design doesn’t directly address the question being asked)
3) When the literature review is insufficient (every article must cite its sources to show that the scientist conducting the study is sufficiently informed on the subject to conduct meaningful research)
4) Errors in logic at any point in the paper
5) When the data does not address the hypothesis
6) When there is insufficient data to support the conclusions being drawn
7) When the data doesn’t support the conclusions at all
This is of course not an exhaustive list.
Of course, there have been allegations of the suppression of dissenting views, and this is one of the reasons why an open approach to peer review is now followed by most of the authoritative scientific journals. With the reviewers known, it’s much harder for such bias to slip through.
The ultimate test of authenticity for a scientific paper is of course when other scientists either use the paper to base their own research on (in which case incorrectly done or false research is easily exposed), or when other scientists use the carefully noted methods to replicate the experiment. If the research was based on observation, then other scientists may go and make their own observations.
Ultimately, there is very little chance of fraudulent, biased or incorrect work from passing undetected, and zero chance for detected flaws to go unchallenged. This is what makes science and scientific publications so authoritative.
(Previous articles in the series: