Validity and Reliability (Statistical Associates Blue Book Series 12)

A study is valid if its measures actually measure what they claim to, and if there are no logical errors in drawing conclusions from the data. There are a great many labels for different types of validity, but they all have to do with threats and biases which would undermine the meaningfulness of research. Researchers disagree on the definitions and types, which overlap. The typology is much less important than understanding the types of questions the researcher should ask about the validity of research.

Reliability is the correlation of an item, scale, or instrument with a hypothetical one which truly measures what it is supposed to. As the "true measure" is not available, reliability must be estimated by correlation with what is assumed to be true. All reliability coefficients are forms of correlation coefficients, but there are multiple types representing different meanings of reliability. One type, for instance, is "internal consistency reliability" which assumes that if all items in a scale truly measure the same thing, they should be highly intercorrelated with each other. Another is "split-half reliability," which assumes that if all items in a scale truly measure the same thing, then a randomly selected set of half the items should correlate highly with another randomly selected set.

New in the 2013 edition:

"Validity" and "Reliability" now combined in one title.
Totally revised with 33% increase in length
Covers SPSS, SAS, and Stata
Coverage of new tests and coefficients.
Over 30 new illustrations

TABLE OF CONTENTS
VALIDITY OVERVIEW 8
Validity: Historical background 9
Convergent validity 11
Do items in a scale converge on a unidimensional meaning? 11
Cronbach’s alpha as a validity coefficient 11
Other convergent validity criteria 12
Simple factor structure 12
Rasch models 12
Average variance extracted (AVE) 13
Common method variance 14
Discriminant validity 15
Do items in a two scales differentiate constructs? 15
Correlational methods 15
Factor methods 16
Average variance extracted (AVE) method 16
Structural equation modeling methods 19
Criterion validity 20
Types of criterion validity 20
Examples 21
Content validity 22
Overview 22
Example of content validity 22
Ecological validity 23
Internal validity 23
Hawthorne effect (experimenter expectation) 24
Mortality bias 24
Selection bias 24
Evaluation apprehension 24
Compensatory equalization of treatments 24
Compensatory rivalry 25
Resentful demoralization 25
Treatment imitation or diffusion 25
Unintended treatments 25
Cross-sectional limitations 26
Instrumentation change 26
History (intervening events) 26
Maturation 26
Mortality 26
Regression toward the mean 26
Test experience 27
External validity 27
Overview 27
Example 27
Statistical validity 28
Reliability 28
Type I errors and statistical significance 28
Type II Errors and Statistical Power 29
Interaction and non-linearity 30
Causal ambiguity 30
Fallacies of aggregation 30
Validity Checklist 31
RELIABILITY ANALYSIS OVERVIEW 33
Reliability: Overview 33
Data 33
Measurement 35
Scores 35
Number of scale items 35
Triangulation 35
Calibration 35
Models 36
In SPSS 36
In SAS 36
In Stata 37
Internal consistency reliability 38
Cronbach's alpha 38
Overview 38
Interpretation 38
Cut-off criteria 38
Formula 39
Number of items 39
Cronbach's alpha in SPSS 39
SPSS user interface 39
SPSS statistical output 41
KR20 46
Cronbach's alpha in SAS 46
SAS syntax 46
SAS output 47
Cronbach’s alpha in Stata 48
Stata syntax 48
Stata output 48
Spearman-Brown reliability correction for test length 49
Other internal consistency reliability measures 51
Ordinal reliability alpha 51
Composite reliability (CR) 51
Armor's reliability theta 52
Spearman's reliability rho 52
Split-half reliability 53
Overview 53
Split-half reliability in SPSS 53
Overview 53
The Spearman-Brown split-half reliability coefficient 54
The Guttman split