I have served on the program committees for many conferences in storage and HPC over my career. Reviewing paper submissions is a lot of work, but Jeff Hammond once told me that he tries to review ten papers for every one paper he authors, and that ratio has stuck with me as a motivator to keep doing this.
When I started out reviewing, it used to take me at least four hours to review a single paper, and I think this is a common pitfall to many early-career researchers. However, serving on the SC technical program committee usually comes with 8-10 papers, which meant that conference alone was a 40+ hour commitment in the month that reviewers are given.
Over the years, I’ve gotten better at reviewing papers, and I’m down to about 1.5 to 2 hours per paper. Here’s how I approach it:
- The day papers are assigned, I space them out over the month. If I’m given 10 papers and three weeks, I schedule time on Monday/Wednesday/Friday each week to review one paper. This is way less stressful than packing all papers into a weekend.
- I do not read papers from front to back. I read the title, abstract, and results, then figure out what else I need to read after that. There are a couple benefits to this:
- If the results don’t appear until page 7 out of 10, the paper is probably mostly introduction and background and not a lot of novelty. Those papers are easier to skim through without the risk of missing much.
- If the methods and results are weak, the paper is weak. There’s no point in spending a half hour scrutinizing the grammar in the introduction if there’s no way the paper is going to be accepted.
- As I jump around the paper, I write comments down in the review form directly. This makes completing the review paperwork more like editing some notes rather than trying to synthesize high-level points about the whole paper post-hoc.
It only takes about 30 minutes of skimming and jumping around to answer the high-level questions of if the paper is good or bad. The rest of the time is documenting examples of what makes it good or bad in the review form, phrasing things constructively, and giving the authors something that they can use to improve the paper.
I’ve also found that, the better a paper is, the harder it is to review. Conversely, good papers take a lot of work.
- I often look out for bad papers when bidding, because I know I can complete a review with meaningful feedback in 30-60 minutes. Finding constructive feedback for a bad paper is easy and doesn’t really require a deep understanding of the subject matter, because bad papers aren’t even at the starting line of the race.
- Finding constructive feedback for good papers requires a higher degree of subject matter expertise, and this is where I often find myself reading references and trying to understand nuanced concepts. Though they take more time to review though, good papers tend to be more enjoyable because I learn something that I otherwise wouldn’t have known.