As I have said before, I think journals do a much better job than conferences at reviewing papers. This year I got a good perspective on this issue, because I was both an Action editor at TACL (Transactions of the ACL), which is a leading NLP journal; and a Senior Area Chair for the ACL conference, which is a leading NLP conference. The experience reinforced my belief that reviewing is better at journals, and I summarise some of my observations below.

Please note that everything I say is general and probably already known to many of my readers. However I suspect that some readers may be unfamiliar with TACL reviewing and the details of ACL reviewing, and hopefully they will find the below to be useful.

Who reviews papers?

ACL Reviewing

For ACL, authors select a track (such as Generation or Question Answering. Each track has

• two or more Senior Area Chairs (SAC), who are in charge of reviewing for the track. I was SAC for the Reality Check track. SACs are chosen by the overall programme chairs. Incidentally, I’ve always in the past declined SAC invitations, but Reality Check is a great initiative which I wanted to support, so I agreed to be an SAC.
• A set of Area Chairs (AC), each of which is in charge of 10-12 submissions. ACs are established and respected researchers, with expertise in the track. They are chosen by the SACs.
• A set of reviewers, most of which are asked to review 6 papers. Reviewers are very varied, ranging from PhD students to senior researchers. In the past ACL asked MSc students and even undergraduates to review papers, but thankfully this was rare in 2023. At any rate, the reviewer set is mostly created algorithmically, but SACs can adjust it (I added several people who were conscientious and knowledgable but not suggested by the algorithm).

Every paper is assigned to three reviewers and an Area Chair. This is again done algorithmically, with SACs allowed to modify the algorithm’s output (I didnt, largely because I was unfamiliar with most of the reviewers so I could not easily assess their suitability to review specific papers).

The reviewers then write reviews, which authors can respond to. The paper’s AC then reads the reviews and the author’s response, writes a meta-review, and makes a recommendation about whether the paper should be accepted (and if so, to the main conference or to Findings). Note that SACs can override AC recommendations, as can the overall programme chairs.

TACL Reviewing

Each paper is assigned to an Action Editor (like me) based on topic; I usually get papers on generation, evaluation, or cognitive modelling. The Action Editor chooses three reviewers, who usually come from TACL’s Standing reviewer team; these are established researchers who are similar in stature to ACL Area Chairs. The choice is done manually, not algorithmically.

The reviewers then write reviews, and I as Action Editor then make a recommendation as to whether the paper should be accepted (usually after major or minor revisions), or whether it should be rejected.

Comparison

You’ll from the above that TACL reviewers are all experienced researchers, who are manually selected by the Action Editor. In contrast, ACL reviewers are very varied (and include many PhD students), and are mostly selected algorithmically.

Review process?

ACL Reviewing

The ACL reviewing process focuses on deciding whether a paper should be accepted. In other words, it is a “gate” which selects some papers and discards others. Reviewers can make suggestions for improving papers, but authors may ignore these, especially since they have less than a month to make changes to papers (ie, prepare “camera-ready” versions) after receiving the reviews. I personally have largely given up making suggestions for improving papers, since I have so often seen authors ignore my suggestions in the past…

Reviewing is done using a structured review form, with questions such as “Reasons to accept” and “Reasons to reject”. It is also done on a strict timeline. Papers must be submitted by a specific date (20 Jan 2023 for ACL23), reviews are expected to be complete by specific data (10 Mar 2023 for ACL23), authors are notified of acceptance by a specific date (1 May 2023 for ACL23), etc.

There is an alternative submission mechanism, ACL Rolling Review, which allows papers to be submitted at any time, and has more flexible reviewing timetables. Once a paper has gone through ARR, authors can direct it to a specific conference (such as ACL23). SACs then decide whether to accept the paper.

From a time perspective, authors wait 3.5 months to find out if their paper is accepted. If it is accepted, it is formally published when the conference starts (10 July for ACL23) so time from submission to publication is close to 6 months.

TACL Reviewing

A fundamental difference in journal reviewing is that the reviewing process tries to improve papers as well as “gate” them. In my own experience, journal editors on several occasions have thought that my submission was promising but had problems, and then worked with me to improve the paper to address the problems (some examples are in a blog).

Looking specifically at TACL, TACL does not use a structured review form, but rather asks reviewers to comment on a paper. Reviews make suggestions for improving papers, some of them very detailed (I have seen TACL reviews which included suggested Python code) as well as recommending whether to accept a paper.

When a paper has been reviewed, the Action Editor (someone like me), decides whether to

• accept the paper as submitted (which is rare).
• accept a paper if specific corrections are made. Authors have 2 months to make the corrections, but in practice can usually ask for more time if needed. Corrections are checked before the paper is accepted.
• reject the paper as submitted, but encourage a paper to be resubmitted with specific change made. Authors have 6 months to make the corrections. In many (not all) cases, resubmitted papers are reviewed in part by the same people who reviewed the original paper.
• reject paper

Note that if corrections are requested, then they must be made; this is different from suggestions made in ACL reviews, which authors can ignore.

From a time perspective, if a paper is rejected, the author should find out in 2 months. If it is accepted with specific corrections, the paper can usually be published within 6 months from initial submission (ie, comparable to ACL) if authors make corrections promptly. However, if a paper needs to be resubmitted (and is accepted when resubmitted), then it can be a year from initial submission to formal publication.

Comparison

The key difference is that the ACL reviewing process focused on accept/reject decisions, while the TACL reviewing process also tries to improve papers. Which I definitely prefer as an author!

People often say that TACL is slower to publish papers than ACL, but this is only the case if a paper needs to be resubmitted (and even here the timing is similar to resubmitting a rejected xACL paper to a subsequent xACL conference). Otherwise timescale is comparable, and indeed TACL is faster in letting authors know if papers are rejected.

Final thoughts

My experiences as Action Editor and SAC have reinforced my conviction that that the reviewing process is better at TACL, because (a) TACL reviewers are all experienced researchers, and (b) the process focuses on improving the paper as well as accept/reject. So I hope more researchers consider submitting papers to TACL or other journals such as Computational Linguistics!