Reliability vs Validity (Video)

0:03 Introduction

In this video, we will learn about reliability versus validity in research. Reliability describes how consistently a measurement gives the same result, and validity describes whether the scores support the interpretation you intend, such as a diagnosis or a comparison between groups.

0:23 The target diagram

First, we will look at the target diagram and separate random from systematic error. The usual way to picture the difference between the two properties is a target diagram, in which the center of the target is the true value and each arrow is one measurement. Arrows grouped tightly in the center show a measure that is reliable and valid. Arrows tight but off-center show a measure that is reliable and not valid, because the measurements agree with each other and miss the true value by the same amount. Arrows scattered around the center show low reliability, so you cannot trust a single measurement even when the average of many lies near the true value. When the arrows are scattered and also off-center, the measure is neither reliable nor valid. Consistency and accuracy are separate properties of a measurement, so a study has to establish each property on its own evidence. I request that you watch the video till the end because then we can make more of these videos as the algorithm recommends it to more people because of increased retention time. And that really helps us a lot.

1:26 Random and systematic error

The two properties correspond to two kinds of measurement error, because you can write any measurement as the true value plus a random error plus a systematic error. Random error varies unpredictably from one measurement to the next, and it lowers reliability. Systematic error shifts measurements in one direction, so it can leave reliability high while the scores stay wrong. A bathroom scale that always reads two kilograms more than your true weight gives the same reading on repeated weighing, so the scale is reliable. The reading does not show your true weight, so the scale is not valid for that purpose.

2:04 Four kinds of reliability

Second, we will cover four kinds of reliability. Reliability has several forms, because a measurement can vary across time, across raters, across the items of a scale, or across versions of a test. Test-retest reliability is the agreement between measurements of the same people on two occasions, and you usually report it with an intraclass correlation. Inter-rater reliability is the agreement between different raters who score the same subjects. Cohen's kappa reports inter-rater reliability when the ratings fall into categories, and an intraclass correlation reports it when the ratings are scores. Internal consistency is the degree to which the items of one scale vary together, and Cronbach's alpha is its usual statistic. Parallel-forms reliability is the agreement between two equivalent versions of a test given to the same people. You choose the form that matches the source of error in your study: a study of essay marking depends on inter-rater reliability, and a scale that each participant completes once depends on internal consistency.

3:11 What validity means

Third, we will define validity and look at content validity. Consistent scores still need an interpretation, and validity concerns what the scores mean for the use you intend. The current professional standards for testing define validity as the degree to which evidence and theory support the interpretation of scores for their intended use. Validity therefore belongs to an interpretation and a use. A scale can support screening in adults and fail to support diagnosis in children. Researchers build validity from several kinds of evidence, and the traditional grouping separates content, criterion and construct validity.

3:51 Content validity

Of the three traditional kinds of validity evidence, content validity comes first, because you begin by asking whether the items of a new scale cover the construct. Content validity is the degree to which the items cover the full content of the construct. Experts judge content validity against a written definition of the construct. A depression scale that asks only about sleep misses mood, interest and energy, along with the other symptoms of depression, so its content validity is poor. Face validity asks whether a measure appears on the surface to measure what it claims, and it gives weaker evidence than a systematic review by experts.

4:30 Criterion validity

Fourth, we will cover criterion validity and construct validity. When a trusted external measure exists, you compare your scores with that measure. Criterion validity is how well the scores agree with an external criterion. Concurrent validity compares the scores with the criterion measured at the same time. For example, you compare a short screening questionnaire with a full diagnostic interview given on the same day. Predictive validity compares the scores with a future outcome. For example, you compare scores on an entrance test with the course grades that the same students earn later. The criterion itself needs reliable and valid measurement. A poor reference standard limits what the comparison can show.

5:14 Construct validity

Many constructs, such as anxiety or motivation, have no single criterion, so you test whether the scores behave as the theory predicts. Convergent evidence shows that the scale correlates strongly with other measures of the same construct or of related constructs. Discriminant evidence shows that the scale correlates weakly with measures of constructs it should not reflect. Structural evidence shows that the items group in the pattern the theory predicts, and factor analysis tests that pattern. For example, a new anxiety scale should correlate strongly with an established anxiety scale and only weakly with a vocabulary test.

5:55 Validity evidence across test development

Fifth, we will follow validity evidence across test development. Content, criterion and construct evidence do not arrive at once, because each stage of building a test produces its own kind of evidence. The first stage is a written definition of the construct, made before anyone writes an item. The second stage produces content evidence. You write the items, and a panel of experts rates how relevant each item is to the construct. The content validity index summarizes those ratings. Six experts rate each item from 1 to 4, and the item-level index is the proportion of experts who give a rating of 3 or 4. An item that five of the six experts rate as relevant has an index of 0.83, and the item stays in the scale. An item that three of the six rate as relevant has an index of 0.50, and you revise or drop the item. With six or more experts, an item commonly needs an index of at least 0.78 to stay. The third stage produces evidence from response processes: you run cognitive interviews, in which respondents explain how they understood each item, and you reword the items they misread. The fourth stage produces evidence about internal structure, because you field-test the items on a large sample, run a factor analysis and estimate reliability. The last two stages produce evidence about relations to other variables, from correlations with established measures and with a criterion, and evidence about consequences, from the decisions made with the scores and the effects of those decisions.

7:30 Why reliability limits validity

Sixth, we will see why reliability limits validity and sort measures into reliable, valid, both or neither. Random error weakens any relationship a measure can show, so reliability sets a limit on the validity coefficients you can obtain. Static on a radio broadcast has a similar effect, because the song stays the same while the noise makes the song harder to recognize, and random error hides a real correlation in the same way. The correlation of a measure with another variable cannot exceed the square root of the measure's reliability, so for a scale with a reliability of 0.81, the ceiling is 0.90. To see the effect in a study, take an illustrative example that relates an anxiety questionnaire with a reliability of 0.81 to a sleep-quality questionnaire with a reliability of 0.64. Suppose the true correlation between anxiety and sleep quality is 0.60. You expect to observe 0.60 times the square root of 0.81 times 0.64. The product of the two reliabilities is 0.52, and its square root is 0.72, so the expected observed correlation is 0.60 times 0.72, about 0.43. The correction for attenuation reverses the calculation, and dividing the observed 0.43 by 0.72 recovers 0.60. Use the correction to interpret a result, and report the observed correlation alongside the corrected value. An unreliable measure makes a real relationship look weaker, so reliability is a necessary condition for validity, although reliability alone does not establish validity.

9:16 Reliable, valid, both or neither

The four patterns on the target correspond to situations you meet in real studies. A scale that reads two kilograms heavy is reliable and not valid, and so is a questionnaire whose items consistently measure the wish to give socially acceptable answers instead of the intended trait. Valid interpretations require reliable scores, so a measure with low reliability cannot support valid conclusions about individuals. A measure can be valid for one purpose and not for another, and its reliability can change from one population to another, so you check the evidence for your own population and your own use.

9:53 Evaluating a measure

Seventh, we will evaluate a measure. When you choose an existing instrument for a study, you evaluate both properties from the published evidence. Check which kinds of reliability the developers tested, and whether those kinds match the sources of error in your study. A published figure for internal consistency tells you nothing about agreement between two raters. Check the validity evidence as well, including who developed the items, which criterion the developers used, and which convergent and discriminant correlations they reported. Check that the evidence comes from a population like yours and from the language version you will use. A scale validated in one country can behave differently after translation. Report your own reliability estimates for your sample, because reliability describes scores from a particular sample.

10:43 Evaluating a published measure

The last part evaluates a published measure. The checklist becomes concrete when you apply it to one published-style scale. The scale in this section is illustrative, and its numbers are invented for the demonstration. The scale has 8 items on sleep disturbance in adults, scored from 0 to 32. Eight experts rated the items for relevance, and the item indices ran from 0.88 to 1.00. Cognitive interviews with 12 adults led the developers to reword two items. In the field test the items formed one factor, with loadings from 0.58 to 0.81. Coefficient alpha was 0.86, and the related coefficient omega was 0.87. Over two weeks the intraclass correlation was 0.88, and the standard error of measurement was 1.9 points. The scale correlated 0.71 with an established sleep-quality index, and its correlation with a vocabulary test was only 0.18, as expected for an unrelated construct. Against a clinical diagnosis of insomnia, the area under the ROC curve was 0.84. The area under the curve is the probability that the scale gives a higher score to a person with insomnia than to a person who does not have insomnia. At a cut-off of 12, sensitivity was 0.80, so the scale identified 80 percent of the people with a diagnosis. Specificity was 0.76, so it correctly cleared 76 percent of the people without a diagnosis. The sample was adults aged 18 to 65, using the English version, in outpatient clinics. For a study of adults aged 18 to 65 in an English-speaking outpatient clinic, the evidence supports use of the scale and of the cut-off at 12. For a study of adults over 75 in a translated version, the evidence does not transfer. You translate with forward and back translation, run cognitive interviews, and check the structure and the reliability in the new sample before you rely on the cut-off. The convergent correlation of 0.71 lies below the reliability ceiling, which is the square root of 0.86, about 0.93.

12:56 Recap and quiz

Here is a summary of what you learned. Reliability is the consistency of a measurement, and validity is the degree to which evidence and theory support the interpretation of scores for an intended use. Random error lowers reliability, and systematic error lowers validity while reliability can stay high. Reliability has four common forms, and validity draws on content, criterion and construct evidence. Random error also caps the correlations a measure can show, which is why reliability limits validity. Validity evidence builds up across the stages of developing a test, and the sleep-scale example showed how you weigh that evidence against the population and language of your own study. Now try a question on what you learned. A new depression questionnaire has a Cronbach's alpha of 0.92 but correlates only 0.15 with a full diagnostic interview. Is the questionnaire reliable, valid, both or neither, and what does the pattern suggest? Pause here and try it before the answer appears. The questionnaire is reliable but not valid. The high alpha shows that the items agree with one another, and the weak correlation with the interview suggests that they measure something other than depression, such as general distress.