This Essay responds to thoughtful analyses from Ashley Rubin, Johann Koehler, Geoff Ward, and Fergus McNeill on our book Breaking the Pendulum: The Long Struggle Over Criminal Justice (2017). In particular, we revisit our claim that consensus within the penal field is illusory. Drawing inspiration from our interlocutors, we argue that while recognized actors constantly struggle over the character and scope of criminal justice, they agree (at least implicitly) that certain positions are “unthinkable” and certain actors must remain outside the field. This “conflictual consensus” limits radical transformations of criminal justice. Our revised perspective encourages scholars to analyze how marginal positions and actors become part of the field, as well as the effects they produce while trying to reshape its boundaries. We conclude by sketching out how scholars have extended and revised the agonistic perspective we advance in Breaking the Pendulum and where we might turn next.