Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters

The third edition of this book was very well received by researchers working in many different fields of research. The use of that text also gave these researchers the opportunity to raise questions, and express additional needs for materials on techniques poorly covered in the literature. For example, when designing an inter-rater reliability study, many researchers wanted to know how to determine the optimal number of raters and the optimal number of subjects that should participate in the experiment. Also, very little space in the literature has been devoted to the notion of intra-rater reliability, particularly for quantitative measurements. The fourth edition of this text addresses those needs, in addition to further refining the presentation of the material already covered in the third edition.

Features of the Fourth Edition include:

• New material on sample size calculations for chance-corrected agreement coefficients, as well as for intraclass correlation coefficients. The researcher will be able to determine the optimal number raters, subjects, and trials per subject.
• The chapter entitled Benchmarking Inter-Rater Reliability Coefficients has been entirely rewritten.
• The introductory chapter has been substantially expanded to explore possible definitions of the notion of inter-rater reliability.
• All chapters have been revised to a large extent to improve their readability.