The well-known writers of fiction and science often report that the best way to be an effective writer and to write a lot is simply to read a lot. Nearly all graduate students and scholars consume an amazing amount of written material. Usually we read professional material because it is necessary scholarship for the specific project that we are currently working on or general reading simply to keep current with the field of study. This reading is usually rapid and interpreted at a surface level. However, for reading to make an essential contribution to becoming an effective writer the reader must think deeply and critically. Many departments or labs hold a “journal club” in which a specific paper is discussed and analysed in a group format. I would estimate that more is learned in a single journal club discussion than in multiple lectures or many rapid and surface readings. Another form of deep and critical reading is the evaluation of articles for scholarly journals. Engaging in peer review is one of the best methods for the development of skills as a writer and scholar is to review articles for professional journals.
There is much criticism about the value of peer review as a measure of worthiness for publication in selective scholarly journals. This occurs most frequently in the social sciences where assumptions are often not clear and social and political agendas implicitly carry more weight than anyone would care to admit. Despite bias and random error I still support peer review as the strongest method of advancing the science and scholarship in any field of study. Although decision-making for publication ultimately lies with the journal editor (or associate editors), peer review not only strengthens the quality of published papers, but also requires the deep and critical reading that inspires productivity and writing skills of the peer reviewers.
When requesting that a scholar be a peer review on a submitted article for publication, journals are maddeningly unclear on exactly what this review should look like. Further complicating matters is that different journals emphasize different types of reviews. Some journals would like the reviewer to suggest an editorial decision such as reject, revise and resubmit, accept with major revisions, or accept with minor revisions; while other journal specify that the peer reviewer is not to make an editorial decision. Some journals provide scoring rubrics that may provide structure, but may not be relevant to the format of the specific paper being reviewed. Are there common features that a peer review should contain that not only provide a valuable service to the profession but can facilitate the deep and critical reading required in the development of an outstanding graduate student and scholar?
The first question is how does a graduate student become a peer reviewer? I know one professor who is on eight editorial view boards and has his graduate students write all of his peer reviews. Although this is ethically sketchy (okay, it is flat out wrong), it is a great opportunity for his students to engage in the peer review exercise. Many journals have a small student review panel that is designed for graduate students to be peer reviewers and have their reviews evaluated and mentored by journal editors or associate editors. Finally, many journals are so overwhelmed with articles and with so few available reviewers that they accept graduate student volunteers as peer reviewers. Thus, for many situations it is a matter of sending information concerning your availability, expertise, and a CV to an editor. My experience as an editor is that the best reviews I get are from graduate students, postdocs, and faculty members within their first three years of the job. The big shots and established scholars are often unavailable; or tend to write short, dismissive, and lazy reviews (there are notable exceptions to this gross generalization). Most editors are excited to have anyone volunteer to be a reviewer.
Many reviewers focus on relatively unimportant details. Emphasis on APA style, manuscript preparation, and grammar usually indicate that the reviewer has not given the paper the deep and critical reading that a strong review requires. Although poor writing can be annoying, as long as the major points are clear, then the writing can be revised and journals have copy editors for a reason. Excessive amounts of energy spent line editing are usually wasted. Manuscript preparation can be identified as a problem, but is rarely a strong reason for rejection.
There are two major areas of focus: internal and external. I follow generally the same process for all reviews. Step one: has the rationale for the research question been established and couched in the current literature. This step involves a logical flow of scholarship. There does not need to be a running history of every single study done on the topic, yet the need for the current study needs to be clear. Step two: the overall research question and specific hypotheses need to be clear and well stated. Step three: methods must be appropriate to answer the research questions and hypotheses. It is surprising how many studies using sophisticated methods and analyses and answer a valuable research question, but the methods do not answer the question that was developed in the introduction. Step four: do the results logically flow from the methods and are the best possible way to answer the research question. Step five: are the conclusions consistent with the results. This is basic article reviewing 101, but I would estimate that over 80% of papers are rejected because the rationale for the study is not established or there is no logical flow to the sections. By the way, being the first article that studies any topic is not a rationale (many times no one else has studied the topic because it was not worth studying). Most commonly, the conclusions do not logically follow from the results. I am also somewhat amused that many conclusion sections include a limitation segment that is overwhelming. Just because the author acknowledges that the paper has significant weaknesses does not mean that those weaknesses cannot be used to reject the paper. This complex and critical part of the review represents the internal aspects of a review and are necessary, but not sufficient, for an article to receive a positive evaluation.
The external part of the review s most difficult for graduate students and new reviewers. This is the “so what” question. The majority of papers I see published within psychology do not seem to have answered the so what question. These are papers that are perfectly internally consistent with logical flow from problem identification to literature review to research question and hypothesis to methods to results to conclusions; yet, the final results do not make a significant contribution to the field. If an article does not advance the field of study, then there is no need for this paper to be published in a scholarly outlet. Leave program evaluations, button sorting and bottle washing, and other tedious minutia to government reports and grant reports rather than publication in a scientific journal. The external part of the review is where experience can be a great benefit. Reviewers need to not only know their own specific and narrow areas of expertise, but understand where the frontier of knowledge is and what type of studies are required to advance that frontier. There is nothing wrong with an inexperienced reviewer seeking guidance or consultation from a more experienced person when preparing an article review.
Here are a few key features to remember when preparing your article review:
- The goal of the review is to improve the paper. Therefore, the tone of the review is to highlight the changes necessary to make the paper a publishable one. My personal style does not focus on struggling to find a strength of the paper or providing moral encouragement. My style tends to be along the lines of, “in order for this paper to be publishable the following changes must be made…” Sometimes those changes are significant and include redesigning of the experiment and collecting new data. However, common comments such as, “This paper should be reviewed by a native English speaker.” Or “This paper was clearly written by a graduate student.” Are not productive and should be avoided. Taking the tone of speaking directly to the author with the goal of improving the paper is most productive.
- Most journals have a section for confidential comments for the editor. This is the space to advise the editor concerning publication status.
- When a rubric is provided by the journal, please use that rubric. Many associate editors and editors adhere closely to the results of the rubric when making final decisions.
- In a recent change, I have begun to attach my name to all reviews. Reviews are a form of scholarship. Reviewers also need to make sure that they are completely accountable for everything stated in a review. This is also a reminder to make all reviews helpful and productive for authors and editors alike. A couple editors have taken my name off of the reviews before sending to the authors. However, being accountable for our work is a good idea.
- Some journals have begun providing feedback to reviewers. This is a wonderful idea. All reviewers could benefit from feedback and improvement in their manuscript reviewing.
- Typically, reviews range in length from one to two single-spaced pages.
- Reviews usually require two to four hours to complete.
- The time that editors require for the completion of a review has been reduced significantly over the last few years. Six weeks for a review used to be the norm. Now the expectations most commonly range from 14 to 30 days. It is better to turn down an opportunity to review a journal, then to not be able to complete the review during the time allotted. A simple “I cannot get this review completed in the time allotted, but would like to review an article in the future” is sufficient.
- As soon as the invitation to review an article is accepted, then schedule approximately one half day to complete the review in your calendar or to do list.
- As mentioned before, it is acceptable to receive consultation or advice from others with expertise in the area. It is also completely acceptable to consult with the associate editor or editor if you have any questions about the preparation of your review. Although reviewers are supposed to be blind as to the name of the author, many times reviewers can figure out who the author is. Under no circumstances should you consult with the author of the paper on any topic.
By conducting a deep and critical review of papers in progress, graduate students can gain insight into preparation of manuscripts, data reporting, and development of research studies. There are a host of lessons that can be learned in a positive direction (e.g., being inspired by an especially creative research design or particularly useful turn of phrase) or in a negative direction (e.g., appreciating how certain forms of expression can be superfluous or misleading). Receiving practice reviewing articles for journals is not only a service to your profession, but serves as an excellent mechanism to improve your research and writing skills.
S. R. Shaw
@Shawpsych
researchtopractice 