Skip to content
verifiedEvidence-Based Practice

Internal and External Validity

Two different questions about the same study — and neither one substitutes for the other.

Translated from the Portuguese original. Read the original (Português)

boltIn short

When a study shows that a treatment worked, two questions apply, and they are independent of each other. The first is about internal validity: was the change really caused by the intervention, or could something else — time, maturation, extra attention, a switch in who was measuring — explain the same result? The second is about external validity: does this effect hold up outside that setting, with other people, in other contexts, in the hands of other practitioners, measured a different way? The order between them matters. Without internal validity there is no demonstrated effect — and you cannot generalize what was never established. But high internal validity does not hand you generality for free either: that is what replication delivers. In practice, whoever reads a study needs to ask both questions: did it work because of the treatment? and did it work for someone like the person I am treating?

Technical definition

Internal validity is the degree to which a study's design authorizes the inference that the independent variable — and not extraneous, uncontrolled, or unknown variables — produced the observed change in the dependent variable. In Campbell and Stanley (1963), it is the sine qua non of experimental research: "the basic minimum without which any experiment is uninterpretable — did the experimental treatments, in fact, make a difference in this specific experimental instance?". External validity is the degree to which the demonstrated cause-and-effect relationship holds up under variation of people, settings, treatment variables, and measurement variables; the same authors phrase it as the question "to what populations, settings, treatment variables, and measurement variables can this effect be generalized?". In Cooper, Heron & Heward, the two definitions reach the vocabulary of Behavior Analysis almost intact: internal validity is "the extent to which an experiment convincingly demonstrates that behavior change is a function of the independent variable rather than the result of uncontrolled or unknown variables," and external validity is "the extent to which a study's findings have generality to other subjects, settings, and/or behaviors." Shadish, Cook, and Campbell (2002) fit both into a system of four validities, alongside statistical conclusion validity and construct validity.

Two questions that keep getting mistaken for one another

Picture a school that adopts a new reading program in March and measures the class's performance in November. The scores went up. Question 1: was it the program? Between March and November, other things happened — the children got eight months older, there was a book fair, the teacher changed, two families left the school, and the November test was not exactly the same as the one in March. Each of these is a competing explanation, and the design's job is to close the door on them. That is the question of internal validity.

Now suppose the door was closed properly: there was a comparison class, the measures were equivalent, no one dropped out of the study. Question 2: does that hold at my school? The program was tested at a small private school, with teachers trained by the program's own authors and classes of fifteen students. Yours has thirty-five. No amount of rigor in question 1 answers question 2 — and no similarity of context in question 2 rescues a study that failed question 1.

They are separate questions because they call for separate remedies. The first is solved with control: a comparison group, random assignment, repeated measurement, blinding, a stable baseline, checking procedural integrity. The second is solved with deliberate variation: repeating the study with other people, in other places, with other practitioners — in other words, with replication.

Internal validity was it really the treatment? history maturation regression intervention change the design closes the door on the competing explanations External validity does it hold beyond here? other people other settings other practitioners other measures effect here ? ? ? ? only replication answers these four question marks
On the left, the work of internal validity: blocking the competing explanations until only one remains. On the right, the work of external validity: finding out how far the effect travels. One side does not do the other's job.

The four-validities system

Campbell and Stanley opened the discussion with two validities. The tradition that followed them kept unpacking the question, and Shadish, Cook, and Campbell (2002) consolidated the typology into four, separating what used to come bundled together. Tap the cards to see the question each one asks:

It is worth separating these four from two other senses of the word "validity" that circulate in the same literature, because the confusion is common. This page is about validity of the inference — what the study authorizes you to conclude. Measurement validity is something else: does the number you wrote down track what actually matters? And social validity is something else still: does the change matter in the person's life? A study can get one of these right and the other two wrong.

The catalog of threats

The great practical contribution of this literature was not the definitions, but the lists. Instead of asking the reader to be suspicious in general, it names, one by one, the known ways a study can be misleading. Knowing the names is what turns vague suspicion into critical reading:

The nine from Shadish, Cook, and Campbell (2002)

  • Ambiguous temporal precedence: it is unclear what came first. Did the child improve because support arrived, or did support arrive because she was already improving?
  • Selection: the groups were already different before starting — the threat that random assignment exists to neutralize.
  • History: something outside the study happened during the same period and affected the result.
  • Maturation: people change simply because time passes — they get older, more experienced, more tired.
  • Regression (to the mean): whoever is selected for being at an extreme tends to move toward the middle on the next measurement, with or without treatment.
  • Attrition: whoever drops out of the study does not drop out at random. If it is precisely the people who were not improving who leave, the final result looks artificially good.
  • Testing: taking the test teaches you how to take the test. The second administration goes up because of practice, not treatment.
  • Instrumentation: what changed was the instrument or the observer — a looser criterion at the end of the study produces "improvement" without any new behavior.
  • Additive and interactive effects: threats combine with one another. Selection interacts with history, with maturation, with instrumentation — and the sum is more misleading than any single part.

Where generalization gets stuck

Bracht and Glass (1968) split external validity into population and ecological validity and cataloged what gets in the way of each. Some items from the list:

  • Narrow sample — this is the item on the population side: whoever entered the study does not resemble whoever seeks treatment. The rest of this list belongs to the ecological side.
  • Hawthorne effect: people change because they are being observed and getting attention, not because of the procedure itself.
  • Novelty and disruption: the program works while it is new; the initial enthusiasm does not survive into the second year.
  • Experimenter effect: whoever applied the procedure was the method's author, highly trained and invested in the outcome. With a different practitioner, the effect may shrink.
  • Multiple-treatment interference: the person received several things at the same time; the result may depend on the combination, not on any single component on its own.
  • Pretest sensitization: answering the initial questionnaire drew attention to the topic and laid groundwork for the intervention — outside the study, that groundwork does not happen.
  • Insufficient description of the treatment: if the report does not let you reproduce what was done, there is nothing to generalize. Hence the importance of detailed procedural description.

What changes when the subject is their own control

Petursdottir and Carr (2018) applied this same catalog to single-case designs in Behavior Analysis, and the result is instructive: part of the threats simply do not apply.

  • Selection becomes irrelevant: the comparison is within the participant, not between participants. There are no two groups to be unequal.
  • History, maturation, testing, and instrumentation are well controlled by combining repeated measurement across multiple conditions with replication of the effect — reversal, multiple baseline, alternating treatments.
  • Regression to the mean is a minimal threat, precisely because nothing gets decided from a single isolated measure: there is an entire time series.
  • Attrition matters little, except when the report aggregates the performance of several participants into a single number.
  • What is left demanding vigilance is the observer: drift and bias in whoever records the data call for ongoing agreement checks. See recording and data collection.

In exchange, the external validity of a single case starts out unknown — and is built through systematic replication, with planned variations in participants, settings, and procedures, until you discover where the effect holds and where it stops holding.

The tension between the two — and why it is not a law

There is a real tension. To shield question 1, the researcher tightens control: accepting only participants with a single diagnosis and no associated conditions, standardizing the protocol session by session, running everything at a university clinic with supervised practitioners. Each of these tightening moves pulls the study further away from the conditions where the treatment will actually be used. That is the difference between an explanatory trial — ideal conditions, giving the treatment its best chance to show an effect — and a pragmatic trial, run under real service conditions, to find out whether the effect survives them.

The PRECIS-2 (Loudon and colleagues, 2015a, in the BMJ) turned that distinction into an instrument: nine domains of a trial — eligibility, recruitment, setting, organization, flexibility of delivery, flexibility of adherence, follow-up, primary outcome, and primary analysis — each scored from 1 (highly explanatory) to 5 (highly pragmatic). There is no "correct" score: there is a score that matches the question the trial wants to answer.

Explanatory ↔ pragmatic continuum explanatory ideal conditions pragmatic real service conditions Eligibility Setting Adherence 1 2 3 4 5 explanatory trial pragmatic trial
Three of the nine PRECIS-2 domains, with hypothetical positions, chosen only to illustrate — they are not the score of any real trial. No trial is entirely explanatory or pragmatic: each domain is positioned on its own, and the design is only good if it is consistent with the question the study wants to answer.
verified

How much the sample can shrinkA review on anxiety disorders (Goldstein-Piekarski, Williams & Humphreys, 2016) tallied the exclusion criteria most used in treatment studies: psychosis in 80% of them, substance dependence in 69%, bipolar disorder in 47%, comorbid major depression in 39%, and suicide risk in 36%. Added together, these criteria can exclude up to 92% of people who actually seek treatment. That does not invalidate the studies — it invalidates the lazy reading that the result applies, without further thought, to anyone who walks into the clinic.

But the tension is not a conservation law, where gaining on one side forces a loss on the other. Loosening eligibility criteria widens the population reached without touching random assignment, blinding, or analysis — and those are the gears that sustain internal validity. In a survey of cardiovascular trials, the same team (Loudon and colleagues, 2015b) compared more explanatory and more pragmatic studies using the Cochrane risk-of-bias tool and found no clear difference between them. It is limited evidence — a conference abstract, with few trials — but enough to be wary of the automatic assumption. The trade-off exists in some domains; in others, it is just carelessness dressed up as a methodological choice.

warning

Where the reading slips"The study is weak, but it was done in the real world, so it counts for more in practice." It does not. Without internal validity there is no established effect — and generalizing what was never established is generalizing nothing. That is why Campbell and Stanley called internal validity the sine qua non: it is a condition of entry, not one more item on a checklist of qualities. The reverse is just as misleading: a flawless experiment with eight college students in a lab does not, by itself, authorize a recommendation for children in a public school.

What this changes about how you read a study

Two questions, asked in this order, handle most of the work:

  • Was there something to compare against? A control group, a stable baseline, a condition reversal. Without comparison, "improved after treatment" is an anecdote, not evidence.
  • Who dropped out of the study, and why? Unequal losses between groups undermine the comparison without leaving a trace in the final result — whoever dropped out does not show up in the number you are reading.
  • Who was excluded at the start? The exclusion criteria sit in the methods section and say, in two lines, who the result applies to.
  • Who applied it? The method's authors, a team trained by them, or ordinary service professionals? That changes what you can expect outside the study.
  • Is the procedure described in enough detail to be reproduced? And was there a check that it was applied as described — that is, of procedural integrity?
  • Was there replication? A single study is a start. It is the sum of many studies, gathered in systematic reviews, that supports a recommendation — and it is from that sum that evidence-based practice draws.

For whoever provides treatment, the external-validity question has a very concrete, inescapable version: was this study done with someone like the person in my room? When the answer is no — and it often is — the study does not stop being useful; it becomes a hypothesis, to be tested in that specific case with its own measurement and follow-up. Generalization of learned behavior and data-based decision making build exactly that bridge between what the literature shows and what happens here.

confounding variable competing explanation population validity ecological validity pragmatic trial systematic replication

Test your understanding

quiz

Quick quiz

Three questions to lock in the concept. You will see the explanation right away.

Further reading

  1. Campbell, D. T. & Stanley, J. C. “Experimental and quasi-experimental designs for research on teaching”. In: Gage, N. L. (ed.) Handbook of Research on Teaching. Rand McNally, 1963, p. 171–246 — reissued as a monograph (Experimental and Quasi-Experimental Designs for Research, Rand McNally, 1966).
  2. Shadish, W. R.; Cook, T. D. & Campbell, D. T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin, 2002 — the four-validities system and the nine threats to internal validity.
  3. Bracht, G. H. & Glass, G. V. “The external validity of experiments”. American Educational Research Journal, 5(4), 437–474, 1968. DOI 10.3102/00028312005004437 — distinction between population and ecological validity.
  4. Cooper, J. O.; Heron, T. E. & Heward, W. L. Applied Behavior Analysis. 3rd ed. Pearson, 2020 — definitions of internal and external validity in the vocabulary of Behavior Analysis.
  5. Petursdottir, A. I. & Carr, J. E. “Applying the taxonomy of validity threats from mainstream research design to single-case experiments in applied behavior analysis”. Behavior Analysis in Practice, 11(3), 228–240, 2018. DOI 10.1007/s40617-018-00294-6.
  6. Loudon, K.; Treweek, S.; Sullivan, F.; Donnan, P.; Thorpe, K. E. & Zwarenstein, M. “The PRECIS-2 tool: designing trials that are fit for purpose”. BMJ, 350:h2147, 2015 (2015a). DOI 10.1136/bmj.h2147.
  7. Loudon, K.; Zwarenstein, M.; Sullivan, F.; Donnan, P. & Treweek, S. “Do pragmatic trials trade-off internal validity for external validity?”. Trials, 16(Suppl. 2):O81, 2015 (2015b; conference abstract). DOI 10.1186/1745-6215-16-S2-O81.
  8. Goldstein-Piekarski, A. N.; Williams, L. M. & Humphreys, K. “A trans-diagnostic review of anxiety disorder comorbidity and the impact of multiple exclusion criteria on studying clinical outcomes in anxiety disorders”. Translational Psychiatry, 6(6):e847, 2016. DOI 10.1038/tp.2016.108.
info

This entry is educational and does not replace assessment, diagnosis or treatment by a qualified health professional. Every person is unique — for a specific case, seek individual guidance.

Looking for professional guidance?

Instituto Walden4 is a psychology and education center in Brasília, Brazil, offering evidence-based assessment and care for autism, psychotherapy and development — in person and online. Our team serves in Portuguese; get in touch and we will find the best way to help.

chat Message us on WhatsApp
chat

Conta Walden4

Entrar

Use o e-mail e a senha da sua conta.