MAY 20-23, 2024 AT THE HILTON SAN FRANCISCO UNION SQUARE, SAN FRANCISCO, CA
45th IEEE Symposium on
Security and Privacy
For 2024, IEEE S&P will adopt the following process changes. All of these changes stem from the concern that reviewers are continually asking authors and reviewers to do more, well above what we would consider “acceptable.” The numbers support this. Thus far for 2023, the percentage of papers that are accepted is averaging approximately 1.5%, and those accepted the first time with conditional accept revisions is averaging approximately 5%. Under the old model without major revisions, we would have an approximately 6.5% acceptance rate, which is simply unacceptable.
The three changes seek to address this concern from different perspectives, but with a common underlying hypothesis:
If reviewers feel their criticism is publicly documented alongside a published paper, they may be more willing to accept papers with a more traditional set of expectations of what is acceptable at top-tier security conferences.
Reviewers often point out weaknesses in a paper, but not all of them are necessarily fatal. The goal of this change is to remove the incentive for reviewers to ask for “interesting but not crucial to the thesis” changes via a major revision process while still making their voices heard. To achieve this, all accepted papers will be published with a <500 word meta-review. Meta-reviews will be a summary of the strengths and weaknesses of the paper, as seen by the reviewers. Authors will optionally write a <500 word response to the meta-review. Both of these documents will be made available to the public on the conference website next to the paper itself.
Authors that are comfortable with their meta-review can decide to make no changes to their paper. Alternatively, authors will have a time period (e.g., one cycle) in which they can make changes to their submission to have specific weaknesses removed from the meta-review (e.g., conduct a further experiment, the lack of which was identified as a weakness). In the event that changes are not made before the camera ready deadline of the third submission cycle, authors may opt for their paper to appear in the proceedings of the following year. Changes will be returned to the reviewers, who can then remove these weaknesses from the meta-review. A process similar to this is being used in the Computational Linguistics Community.
All accepted papers will have a shepherd, who is responsible for writing the meta-review. The shepherd will determine if the changes warrant removing a comment from the meta-review. In cases of disagreement between reviewers and authors, both parties can indicate their concerns in their document.
This will result in two decision classes: Accept and Reject. Major and Minor revisions will be removed as an option.
The double-blind process is supposed to limit bias in reviewing. However, reviewers often elect to review papers that they have previously rejected at other conferences. Once a reviewer has offered their comments on a paper and elected not to give the authors a major revision, the commitment from the authors to that reviewer should end. This is necessary to prevent both protectionism (i.e., reviewers intentionally killing papers that compete in their area) and also against sloppy reviews (which demand “nice to have” but unnecessary changes to work).
Reviewers will be instructed to not review papers they have previously rejected at other conferences. Our community is large and robust enough that the opinion of any single person should not be necessary for us all to move forward.
Submission of prior reviews was supposed to serve two purposes:
It is unclear if reviewers are even reading prior reviews, making the benefit to authors negligible. While dissuading flipping paper is good, it is not clear that this is having any effect on the submission load. Authors can simply argue that the old reviews were not relevant, and since most reviewers don’t read the prior reviews, this goes unnoticed. Finally, both CCS and NDSS have removed the requirement to submit prior reviews.