Translated from the Portuguese original. Read the original (Português)
Every measure of behavior answers two separate questions. The first is about reliability: measuring the same event again, does the same number come out? The second is about validity: does that number speak to what actually matters? These are separate questions, and the order between them matters — a miscalibrated scale reports the same 3 kg too many, with perfect consistency, and a scale does not measure height no matter how well calibrated it is. Between the two sits accuracy: how close the recorded value comes to the true value. In clinical and school practice, these three qualities don't appear on their own: they are built — with clear definitions, trained observers, comparison against an external standard, and agreement checks — or they simply don't exist.
Reliability (or dependability) of a measure is the degree to which repeated measurement of the same event produces the same values. Accuracy is the degree to which observed values correspond to the true values of the event, established by an independent reference procedure. Validity is the degree to which the measure — and the interpretation made of it — corresponds to the phenomenon it is meant to investigate. In Cooper, Heron & Heward, these three constitute the indicators of a measure worth trusting, and the validity of a behavioral measure requires three simultaneous elements: measuring a socially significant behavior directly, measuring the dimension relevant to the question at hand, and ensuring the data are representative of the conditions and moments that matter. In the psychometric framework of the Standards for Educational and Psychological Testing (AERA, APA, and NCME, 2014), validity is not a fixed property of the instrument, but rather the degree to which evidence and theory support a specific interpretation of the scores for a specific use.
The questions that keep getting mixed up
Picture three scales. The first is miscalibrated and always reads 3 kg too high: it is consistent — weigh someone ten times and the same number comes out — but the number is systematically wrong. The second has a faulty sensor and returns 61 kg, 68 kg, 64 kg for the same person in the same minute: here the problem is different, no single reading is of any use. The third is a flawless scale used to answer the wrong question — nobody finds out someone's height by weighing them, no matter how fine the calibration.
These three failures have different names because they call for different remedies. The first is a lack of accuracy (systematic bias), and it is fixed by comparing the instrument against a standard. The second is a lack of reliability, and it is fixed by improving the measurement system and the training of whoever measures. The third is a lack of validity, and no amount of technical rigor fixes it: what is being measured has to change.
The triad of a measure worth trusting
These three qualities form a practical hierarchy: validity defines what is worth measuring, accuracy says whether the number matches reality, and reliability says whether the procedure can be repeated. Explore each one:
Validity — does the number speak to what matters?
It is the first question, and the easiest one to skip. For a behavioral measure to be valid, three conditions must hold at the same time:
- Direct measure of the behavior that matters — observing the actual response of interest, not a convenient substitute. Counting how many tokens the child earned is not measuring whether she learned to ask; a questionnaire score filled out by the school is not the frequency of the behavior at school.
- Relevant dimension — the property measured has to correspond to the real concern. If the complaint is that episodes last too long, counting how many episodes occurred may show no improvement at all, even if each episode dropped from forty to five minutes. See dimensions of measurement.
- Representative data — the observation needs to cover the conditions and times when the behavior actually occurs. Measuring aggression only during the first two hours of the morning, when the episodes actually cluster in the late afternoon, produces a pretty, and false, graph.
Accuracy — does the number match the true value?
A measure is accurate when the recorded value coincides with the event's true value. And the true value is not the supervisor's opinion: it is a number obtained through an independent procedure that is more reliable than the measure being tested — a calibrated instrument, the count from a permanent product (the problems actually solved on the worksheet), or the careful analysis of a recording of the same session.
Checking accuracy, then, means comparing against an external standard, not comparing two observers against each other. It is the only check capable of detecting systematic error: if everyone involved underestimates equally, nothing in the internal comparison will flag the problem. In practice, this is usually done by sampling — some sessions over the course of the monitoring period, not all of them.
Reliability — does the measure repeat?
Reliability is consistency: measuring the same event again and getting the same value. It is the easiest quality to check and the easiest one to confuse with the other two, because it says nothing about being correct. A stopped clock is extremely reliable.
Two logical relationships organize the topic and are worth keeping in mind:
- An accurate measure is necessarily reliable — if every reading coincides with the true value, the readings coincide with each other.
- A reliable measure is not necessarily accurate or valid — it can always miss the same way, or precisely measure the wrong thing.
Hence the order of priority: improving the consistency of a measure that measures the wrong thing is investing work to be wrong more elegantly.
Why the confusion is so commonReliability, dependability, and precision all get used for the same underlying idea — and in everyday use, "reliable" suggests "true," which is exactly what the technical term does not guarantee. Two phrases are worth being suspicious of: "this data is reliable, so you can decide based on it" (reliable without being valid is consistent garbage) and "the measure is super detailed, so it's good" (richness of detail about the wrong variable fixes nothing).
What corrodes the validity of the data
Threats to the validity of a measure aren't rare accidents: they are routine choices, made in good faith, almost always for convenience.
- Indirect measure — measuring a substitute in place of the behavior: impression scales, retrospective reports, recollections from people who spent time with the person. Sometimes it's the only option available, and then it is necessary to say explicitly what is actually being measured — someone's impression, not the occurrence of the behavior.
- Wrong dimension — counting when what matters is duration, timing when what matters is latency, measuring percentage correct when the problem is how long it takes to start.
- Discontinuous measurement — interval methods estimate occurrence, and each one errs in a predictable direction. The problem is not using them; it's treating the estimate as an exact count, or switching methods midway through monitoring and reading the difference as progress.
- Poorly chosen observation period — always observing at the same time, in the easiest context, or only when there's enough staff produces data about that window, not about the person's day.
- Insensitive or limiting scale — when the scale doesn't have the resolution to show the expected change, or imposes artificial ceilings and floors. This is the case with percentages calculated over very few trials.
What corrodes accuracy and reliability
Here the threats live in the measurement system and in whoever operates it — and almost never in bad faith:
- Poorly designed measurement system — vague definitions, too many codes, too many behaviors tracked at once, a data sheet that's hard to fill out in the middle of the action. The more the observer has to judge, the more the number depends on who's observing. The antidote starts with the operational definition.
- Insufficient training — whoever records data needs to practice with examples and counterexamples and reach a performance criterion before the data counts. Data from an untrained observer can't be salvaged afterward.
- Observer drift — the application of the definition slowly shifts over the weeks, without anyone noticing. It's countered with periodic refresher training and recalibration against the recordings used in the original training.
- Expectation and bias — knowing which phase the program is in, or which result is expected, nudges the ratings. It helps to keep whoever records data unaware of the phase and the expected result, and to avoid letting the same person both apply and evaluate their own intervention.
- Reactivity — being observed changes the behavior observed, and knowing one is being supervised changes how one records. Less obtrusive observation, adaptation periods, and unannounced checks reduce the effect.
How it's checked in practice: interobserver agreement
The most widely used tool in applied behavior analysis is interobserver agreement — IOA: two people record the same session independently, and the two records are compared. Independent is the key word: if they talk during the session, or if one sees the other's sheet, the agreement obtained means nothing.
IOA answers three useful questions: is the operational definition clear enough to produce the same number in different hands? Are new observers already ready? Was there drift over the weeks? That's why IOA isn't a one-session ritual: it's collected in samples across all phases of monitoring, with varied pairs of observers.
There are several ways to calculate it, and they are not interchangeable — the same pair of records can come out to 95% by one method and 60% by another. Flip the cards:
For measures of time there are equivalents: agreement on total duration (the shorter duration divided by the longer one) and on duration per occurrence, episode by episode — the latter, again, the most demanding and the most informative.
Agreeing is not the same as being right — and 80% is not a theoremTwo observers trained together, with the same imprecise definition and the same expectation, make the same mistakes the same way and show very high agreement. IOA shows that the measure is reproducible, not that it is accurate; for that, an external standard is needed. As for the threshold: the convention in the literature is to report values starting at 80%, with 90% or higher considered desirable, especially when a clinical decision depends on that data — but that is a convention of the field, not a law of nature. A high number obtained in a single session, by the most lenient method, and always with the same pair of observers, tells you very little. More detail in recording and collecting data.
When the measure is a test, a scale, or a questionnaire
Psychological assessments and standardized instruments — developmental scales, inventories, screening questionnaires — run into the same problem with a different vocabulary, that of psychometrics. Reliability is usually estimated in three complementary ways:
- Test-retest — the same instrument applied twice to the same person, with an interval over which no real change is expected, should produce close results.
- Interrater — different examiners, faced with the same performance, should arrive at close scores. It's the direct relative of IOA.
- Internal consistency — items that are supposed to measure the same thing should behave coherently with one another. This is what indices like Cronbach's alpha estimate — and it's worth remembering that a high alpha indicates interrelated items, not that the instrument measures the right construct.
On the validity side, the 2014 Standards (AERA, APA, and NCME) brought a reframing worth knowing: instead of "types of validity," the field now speaks of a single unified concept supported by different sources of evidence — the instrument's content, the response processes of those who answer it, the internal structure of the items, relationships with other variables (including the ability to predict outcomes), and the consequences of its use. The question is no longer "is this test valid?" but "is there enough evidence for this interpretation, with this population, for this purpose?" An instrument well validated for screening risk at a large scale may be weak for settling an individual diagnosis — and an instrument built in another country needs cultural adaptation and local norms before it holds here.
In Brazil this isn't just a technical recommendation. CFP Resolution No. 31/2022, which regulates the Psychological Test Assessment System (SATEPSI) in Brazil and replaced the 2018 resolution, conditions a favorable review of a psychological test on the presentation of empirical evidence of validity, of reliability estimates for the interpretations, and of a scoring and interpretation system — and sets a maximum period of 15 years for the validity, reliability, and norms studies, counted from approval by the CFP's Plenary. In practice, this gives families a legitimate and simple question to ask: is this instrument approved, with up-to-date studies?
Two other senses of "validity"
The word circulates in more than one sense, and swapping them causes confusion. Besides the validity of the measure, discussed here, there is talk of a study's internal and external validity — whether the result can be attributed to the intervention and not to something else, and whether it generalizes to other people and contexts. These are questions about the design, not about the instrument; and none of it holds up without a decent measure, because a flawless experiment that measures the wrong variable rigorously answers a question nobody asked.
There is also social validity, proposed by Montrose Wolf in 1978: asking the people directly involved whether the program's goals make sense, whether the procedures are acceptable, and whether the effects matter in real life. It is an admittedly subjective measure, and an essential one: a downward-trending graph means nothing if the family, the school, and above all the person themselves don't recognize some improvement there that's worth having.
Test your understanding
Quick quiz
Three questions to lock in the concept. You see the explanation right away.
Further reading
- Cooper J. O., Heron T. E. & Heward W. L. Applied Behavior Analysis. 3rd ed. Pearson, 2020 — ch. "Improving and Assessing the Quality of Behavioral Measurement".
- Johnston J. M. & Pennypacker H. S. Strategies and Tactics of Behavioral Research. Routledge — on true values, calibration, and the properties of measurement.
- American Educational Research Association, American Psychological Association & National Council on Measurement in Education. Standards for Educational and Psychological Testing. AERA, 2014.
- Conselho Federal de Psicologia. Resolução CFP nº 31/2022 — guidelines for psychological assessment and regulation of SATEPSI, in Brazil. satepsi.cfp.org.br
- Wolf M. M. "Social validity: the case for subjective measurement or how applied behavior analysis is finding its heart". Journal of Applied Behavior Analysis, 11(2), 203–214, 1978.
This entry is educational and does not replace assessment, diagnosis or treatment by a qualified health professional. Every person is unique — for a specific case, seek individual guidance.