This article reports on a large-scale peer-review assessment of the research done in English departments at German universities, organized by the ‘Wissenschaftsrat’. The main aim is to take a critical look at the methodology of the research rating based on a detailed statistical analysis of the 4,110 ratings provided by the 19 reviewers. The focus is on the reliability of the ratings and on the nature of the criteria that were used to assess the quality of research. The analysis shows that there is little variation across raters, which is an indication of the general reliability of the results. Most criteria highly correlate with each other, and only the criterion of 'Transfer to non-academic addressees' does not correlate very strongly with other indicators of research quality. The amount of external funding turns out not to be a good indicator of research quality.