D.2. Distinguish between internal and external validity.
In behavior analysis and experimental research, understanding internal and external validity is crucial for evaluating the quality and generalizability of findings.
Internal Validity
Internal validity refers to the extent to which we can confidently attribute changes in the dependent variable to the independent variable, rather than to other factors. High internal validity indicates that the study is well-controlled, with minimized confounding variables that could impact the results. This allows the researcher to conclude that the intervention or treatment caused the observed changes in behavior.
• Example: If a BCBA implements a reinforcement program to increase task completion in a child and observes an improvement, high internal validity would mean that this improvement can confidently be attributed to the reinforcement program rather than other unrelated factors, such as changes in the child’s environment or mood.
• Threats to Internal Validity:
• Maturation: Changes in behavior that occur naturally over time.
• Instrumentation: Changes in measurement methods or tools that affect results.
• History: External events that occur during the study and influence behavior.
• Testing effects: Changes in behavior due to repeated measurement or practice.
External Validity
External validity refers to the extent to which the results of a study can be generalized beyond the specific conditions of the experiment. High external validity means that the findings are applicable to other settings, populations, or times, allowing the results to have broader relevance.
• Example: If a BCBA successfully uses a reinforcement program in a clinic setting to increase a specific behavior, the intervention would have high external validity if it could also produce similar results in other settings (like at home or in school) and with different individuals.
• Threats to External Validity:
• Population: If the sample is not representative of a larger population.
• Setting: If the study setting is very different from real-world environments.
• Time: If results only apply to a specific time period and may not hold in future or past contexts.
Key Distinctions
• Internal Validity is about confidence in the cause-and-effect relationship within the study, minimizing alternative explanations for observed changes.
• External Validity is about the generalizability of the study’s findings to other contexts, people, and times.
Summary
High internal validity allows us to confidently say that the intervention caused the observed change in behavior, while high external validity allows us to apply the findings more broadly. Balancing these two forms of validity is important for effective behavior analysis, as it helps ensure interventions are both effective and applicable in real-world situations.
Scenario 1: Internal Validity in a Classroom Intervention
A BCBA implements a token economy system in a classroom to increase on-task behavior. The students receive tokens for every 5 minutes they stay focused, which they can later exchange for small rewards.
• Consideration for Internal Validity: To ensure high internal validity, the BCBA carefully controls the conditions by consistently delivering tokens at the exact 5-minute intervals and limiting other distractions in the classroom.
• Potential Threats to Internal Validity: The BCBA must be mindful of confounding variables, such as changes in classroom routines, substitute teachers, or peer influence, which could affect students’ on-task behavior independently of the token system.
Scenario 2: External Validity in a Social Skills Training Program
A BCBA creates a social skills program for children with autism in a clinical setting, where they practice greetings, turn-taking, and eye contact. After several weeks, children show improvement in social interactions within the clinic.
• Consideration for External Validity: To determine if the skills learned in the clinic generalize to other environments, the BCBA could observe whether these social behaviors also increase in other settings, such as at school or at home.
• Potential Threats to External Validity: The clinic environment may be very structured compared to a school setting, potentially limiting generalizability. If the program works in the clinic but not in more chaotic or unpredictable environments, it may lack external validity.
Scenario 3: Internal Validity in a Self-Management Intervention
A teenager with ADHD often struggles to complete homework. A BCBA introduces a self-management intervention where the student tracks and rewards their own task completion with breaks after every 20 minutes of study.
• Consideration for Internal Validity: To maintain internal validity, the BCBA ensures the student accurately tracks the time and consistently uses the reward system as instructed. They also control for any other variables that might influence homework completion, such as eliminating background noise.
• Potential Threats to Internal Validity: If the student receives extra tutoring during this period or if their parent starts offering additional support, these factors could contribute to improved homework completion, making it unclear if the self-management intervention alone caused the change.
Scenario 4: External Validity in a Functional Communication Training (FCT) Program
A child with developmental disabilities engages in tantrums when unable to access preferred items. The BCBA implements FCT in the clinic, teaching the child to use a communication device to request items instead.
• Consideration for External Validity: To assess if FCT skills transfer outside the clinic, the BCBA encourages the child’s parents to implement the same intervention at home. If the child uses the communication device at home and school, this supports the intervention’s external validity.
• Potential Threats to External Validity: The child may only be comfortable using the device in the clinic, where it was initially introduced. Differences in settings, routines, or response to the communication device could limit its effectiveness in new environments.
Scenario 5: Internal and External Validity in a Study on Reducing Stereotypy
A BCBA develops a behavioral intervention to reduce stereotypy (repetitive, non-functional behaviors) in a child with autism. They use a differential reinforcement procedure in a controlled, one-on-one clinic setting.
• Internal Validity: To maintain high internal validity, the BCBA controls distractions in the clinic and ensures consistent delivery of reinforcement. They also document any additional variables (such as sleep or diet changes) that could impact the child’s behavior.
• External Validity: For external validity, the BCBA assesses whether the reduction in stereotypy generalizes to other settings (such as the child’s home) and when interacting with others, like family members. If the behavior only reduces in the clinic, it may lack external validity.
Scenario 6: External Validity in a Group Exercise Program
A BCBA designs a group exercise program for adults with developmental disabilities at a day center to increase physical activity. The program includes guided exercises with verbal praise after each session.
• Consideration for External Validity: To check if the increase in physical activity continues outside the day center, the BCBA could collaborate with family members to monitor if participants engage in similar exercises at home.
• Potential Threats to External Validity: The structured environment and social encouragement in the day center may not exist at home, limiting generalization. If participants only exercise when supervised or with group encouragement, the results may not be generalizable to other settings.
Scenario 7: Internal Validity in Teaching Safety Skills
A BCBA teaches safety skills to children in a clinic setting, using role-play scenarios to practice crossing streets safely. The intervention includes verbal prompts and reinforcement for correct responses.
• Consideration for Internal Validity: The BCBA ensures that all role-play sessions are identical to avoid confounding variables. They control for distractions and use consistent reinforcement to improve the likelihood that any improvement in safety skills is due to the intervention itself.
• Potential Threats to Internal Validity: If other adults (such as parents) reinforce similar skills outside the sessions, this additional influence could affect the child’s improvement, reducing confidence in the clinic-based intervention’s effect.
Scenario 8: Internal and External Validity in Functional Analysis (FA)
A BCBA conducts a functional analysis (FA) to identify the triggers for a client’s aggressive behaviors. They control the environment to introduce and remove specific conditions (e.g., attention, escape) while observing the behavior.
• Internal Validity: High internal validity is achieved by systematically controlling the conditions to isolate potential triggers for aggression, ensuring that any changes in the aggressive behavior are due to the conditions set up in the FA.
• External Validity: For external validity, the BCBA needs to consider whether the identified triggers in the clinic also apply to other environments, such as the client’s home or school. If the triggers differ across settings, the findings may not generalize well outside the clinic.
These scenarios offer practical ways for BCBA students to think about the importance of internal and external validity in their interventions and assessments, helping ensure both reliable cause-and-effect relationships and the potential for broader application in real-world settings.
In addition to understanding definitions, examples, and threats, there are a few other key aspects of internal and external validity that are important for BCBAs:
1. Balancing Internal and External Validity
• Trade-Offs: Often, as internal validity increases (more control over variables), external validity can decrease, as the settings become more artificial and less representative of real-world conditions. Conversely, focusing on external validity by studying behaviors in natural settings can introduce more uncontrolled variables, potentially lowering internal validity.
• Goal Setting: BCBAs should determine whether their primary goal is to establish a clear cause-and-effect relationship (emphasizing internal validity) or to produce generalizable results (emphasizing external validity).
2. Control Groups and Experimental Design
• Single-Subject vs. Group Designs: BCBAs often use single-subject designs, which allow detailed examination of behavior within individuals and support high internal validity. However, these designs may have limited generalizability. In contrast, group designs (though less common in ABA) can sometimes enhance external validity by studying larger samples.
• Baseline Measures: Establishing a clear baseline before intervention is critical for assessing internal validity, as it shows how behavior looked before changes were introduced.
3. Replication for Validity
• Direct Replication: Repeating the same study with the same conditions can confirm internal validity by verifying that changes in behavior are consistently linked to the intervention.
• Systematic Replication: Making slight variations in the study (e.g., different participants or settings) allows the BCBA to assess external validity by testing whether results apply across different contexts.
4. Social Validity and Practical Application
• Relevance to Real-World Settings: Social validity refers to the acceptability and practicality of interventions from the perspective of clients, caregivers, and other stakeholders. While this is not a direct measure of external validity, it supports generalization by ensuring the intervention is applicable and useful in typical environments.
• Client and Family Involvement: Including family and community members in intervention training can help improve both social and external validity by encouraging generalization and acceptance of the treatment beyond clinical settings.
5. Ethical Considerations
• Minimizing Harm and Maximizing Benefit: When controlling for internal validity, BCBAs should be mindful of ethical obligations to avoid overly restrictive or unnatural conditions that could impact client well-being.
• Informed Consent and Transparency: When focusing on external validity and conducting interventions in diverse settings, it is crucial to obtain informed consent and communicate the goals, processes, and potential impacts to all participants and stakeholders.
6. Data Collection for Validity Checks
• Frequent and Accurate Data Collection: Reliable data collection methods are necessary to support internal validity. Ensuring that measurements accurately reflect the behavior in question (e.g., operational definitions, interobserver agreement) helps to maintain the study’s credibility.
• Generalization Probes: Using probes in other settings or with different people can help check external validity without altering the main intervention setting, providing insight into whether behaviors will transfer to new contexts.
Summary:
BCBAs should be skilled in assessing both internal and external validity and applying strategies that maintain a balance between reliable, controlled intervention effects and practical, generalizable outcomes. Balancing these elements with ethical and social considerations is essential for effective and responsible behavior analysis.