What most of us see of science is provided by the news; we get blurbs about recent scientific advances, or perhaps a blurb about how a new finding challenges current understanding. I think the ‘blurbs’ scrape the substance from the process, and leave us with the impression that conclusions derived from peer-reviewed scientific studies are equivalent to opinions. They’re not. Here’s why.
I submitted a paper for peer review yesterday. Wow, that was terrifying. After uploading the document to the journal’s website, and filling out all the silly crap on the journal’s webpage, my stomach was in knots when I hit the ‘submit’ button.
It is. To understand why, consider scientific inquiry and the peer-review process:
- You spend a year or two doing experiments and analyzing the results, often with the help of undergrad students, grad students, and collaborators from other institutions. This is my favorite part. These experiments are things that have never been done before … you are flying blind … every day is a high-stakes, poor-odds gamble. It is terrifying, but in a good way, and there are multiple opportunities to kludge with duct tape. It’s creative, it’s crazy, it turns your head inside out to have your mind in the clouds with your feet on the ground; it’s a privilege.
- You write the clearest possible manuscript describing the context, methods, results, and what you make of them. You choose your words carefully so that they are as unambiguous as possible. When I was a grad student, a mentor of mine had the following suggestion for writing technical manuscripts: “Imagine writing ‘Romeo and Juliet’ to submit to a peer-reviewed science journal: They met. They f*cked. They died.” Really. It’s like that. Words are tools in this context, and the fewer, the better.
- You submit the manuscript to a journal.
- The journal editor submits the manuscript to 3-5 anonymous peer scientists who are deemed knowledgeable in the subject matter of the manuscript.
- The peer scientists read the manuscript carefully and critique it.
- The editor collects the anonymous critiques and decides on the next step:
- Accept as is (this never happens to me; it rarely happens to anybody).
- Accept with revision (this is what happens to me on a REALLY GOOD DAY).
- Reject, but encourage resubmission with extensive revision and/or additional experiments. This is bearable when provided sufficient ibuprophen, and maybe a few hours with a whoopee cushion. The process of revision and additional experiments can add a year of effort to the manuscript.
- Flat-out, no-go, reject. FAIL. Bleak, bleak, bleak. You have the hard task of facing the reality that you spent a few years of your time, your students’ time, your collaborators’ time, and a fair bit of taxpayer money on an exercise in futility. If you can’t publish it, the research is essentially useless.
If something gets published in a respected peer-reviewed journal, it’s run the gauntlet. The peer-reviews that place the manuscript in 6a – d above are anonymous, and VERY BRUTALLY FRANK.
My manuscript had been previously submitted, and fell into category 6c above. The first round of reviews was VERY BRUTALLY FRANK. Some comments were helpful. Some comments were misguided and indicated that I needed to write some parts of the manuscript more clearly (they met, they f*cked, they died). Some comments, frankly, hurt.
I spent a year doing more experiments and rewriting the paper. When I resubmitted the paper, I had to include a point-by-point list of the reviewers’ comments, and how I addressed them. My list for this paper was 54 items long. Previous papers that I have written, submitted, and published have had lists over 100 items long. If, like me, you aren’t God’s Gift to Science, this is normal.
To submit a paper for peer review is to throw your ego into a chipper-shredder.
Hopefully the above clarifies how results from scientific studies get published, and why they are not equivalent to opinions.
PS As an amusing side note, the process I’ve described explains why it’s common for scientists to have hobbies like racecar driving, rock climbing, jumping horses, skydiving, surfing, fire-breathing, and motorcycling (I have specific colleagues in mind for each of these; this is not a fictitious list). The perception that we are meek and risk-averse couldn’t be farther from the truth.