Randomization
Randomization
Unit Overview
What you will do
Most of us have encountered the idea that randomization is important for proper experimental design, but may have less clear guidance on how to randomize or why randomization is so important. We also might be distracted by common appeals to the “randomness” of winning lottery numbers or other patterned behaviors that feel like pure chance.
Course highlights
Unit Lessons
(lessons marked with an * are included in the "Meeting" version of the unit)
1. Our biased brains
The Extent of the Problem
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
2. "Favored" vs. alternative hypotheses
Motive & Method
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
3. Researcher degrees of freedom
Solbing Rigor Problems
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
4. Mitigating bias through masking
The Toolbox
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
5. How good is your mask?
Power in Numbers
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
Solbing Rigor Problems
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
6. Analytical practices to mitigate bias
Power in Numbers
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
Solbing Rigor Problems
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
7. Data masking in machine learning models
Power in Numbers
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
Solbing Rigor Problems
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
8. Bonus biases that disrupt research
Power in Numbers
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
Solbing Rigor Problems
This unit will explore rigor issues as a cause of unreliable research, and guide you in identifying and seeking out solutions to isolated rigor issues that you may encounter in lab, as well as collective rigor issues that require scientists to work together to solve.
3
Activities
20
minutes
Our biased brains
This lesson introduces the concept of confirmation bias, highlighting how our brains tend to favor information that reinforces existing beliefs. Learn why these biases exist from both psychological and neuroscientific perspectives, and how they can inadvertently affect decision-making in research.
Motive & Method
This lesson focuses on the pitfalls of designing experiments that only test a single, “favored” hypothesis. Learn about and practice developing multiple, mutually exclusive hypotheses to challenge assumptions and reveal hidden biases.
Researcher degrees of freedom
This lesson explains how subjective decisions, ranging from data cleaning to statistical analysis, can unintentionally favor an expected outcome. Learn how “researcher degrees of freedom” can lead to overconfident or skewed results.
Mitigating bias through masking
This lesson introduces the concept of masking (or blinding) in experimental design. Learn how masking can prevent both participants and researchers from inadvertently influencing results with examples from clinical trials.
How good is your mask?
In this lesson, the focus shifts to the evaluation of masking effectiveness. Learn about factors that may inadvertently reveal experimental conditions and how to assess whether masking has been compromised.
How good is your mask?
This lesson introduces differences between exploratory and confirmatory data analysis. Learn how blending these approaches without clear distinction can lead to confirmation bias and discuss strategies to preserve the integrity of research findings.
How good is your mask?
This lesson draws parallels between experimental masking and data leakage in machine learning. Learn about the concept of data leakage and its impact on model evaluation, emphasizing the importance of proper data partitioning.
How good is your mask?
This final lesson broadens the discussion to other biases that can influence research, including conformity bias and cognitive dissonance. Learn how to tie these concepts back to confirmation bias, discovering how multiple biases can compound and affect scientific outcomes.
Unit Overview
Unit Overview
Randomization in experimental design is about systematically and thoughtfully imposing an order onto our treatment of variables. Specifically, randomization is a series of actions that reduce the interference of confounding variables and disturbance variables as you study causal relationships. Intentional action like this doesn’t feel very random, and that’s on purpose.
Activities
minutes
How do I know It’s real?
How do I know It’s real?
Randomization in experimental design is about systematically and thoughtfully imposing an order onto our treatment of variables. Specifically, randomization is a series of actions that reduce the interference of confounding variables and disturbance variables as you study causal relationships. Intentional action like this doesn’t feel very random, and that’s on purpose.
1
Activities
5
minutes