We consider the peer-review process as a complex system that appears to defy predictive modelling. Noisy and inconsistent reviewer judgments create a well-documented “black box” problem for applicants, as wide variability in human evaluation makes it difficult for researchers to preemptively identify the weaknesses—both real and perceived—that might reduce an application’s competitiveness. Assuming that the peer review process to be a highly permeable, abstruse and unpredictable system, we explore the potential efficacy of intervention strategies. We investigate the potential of leveraging LLMs to “simulate” potential reviewer critiques and hypothesize that exposing applicants to a comprehensive analysis of potential application critiques ahead of submission might mitigate some of the effects of weak inter-rater reliability.
Monday, March 17th, 2025, 3 – 4:30 p.m. ET | University of Waterloo | DC 1302 (or virtual)
Systems Community of Inquiry 