Sample Size

Sample size can seem deceptively simple. It's just a number! Yet, that number has a major impact on what our statistical analysis can and can't tell us about the results of our study. In this unit, we explore the impact sample size can have, learn about various calculation methods and their pitfalls, and explore how reading and reporting about sample size can help us to improve the rigor of our work more broadly. This unit includes advice that applies across research contexts, from small to large n and exploratory to confirmatory work.

Lesson 1

The Biased Brain

Explore how cognitive biases shape our perception and decision-making, and why even trained researchers are not immune.

Lesson 2

Framing Effects & Confirmation Bias

Discover how the way a question is framed can steer conclusions before any data is collected.

Lesson 3

Spotting Bias in the Wild

Apply what you've learned by identifying real examples of confirmation bias in published studies and everyday research scenarios.

Lesson 4

Masking & Blinding Techniques

Learn practical strategies like masking and blinding that researchers use to reduce the influence of confirmation bias.

Lesson 5

Building a Rigorous Research Practice

Bring everything together and develop a personal checklist for designing bias-aware, reproducible research.

Lesson 6

Lesson title

Lesson description

Lesson 7

Lesson title

Lesson description

Lesson 8

Lesson title

Lesson description

Lesson 9

Lesson title

Lesson description

Sample Size
Summary
Sample Size

References:

Sample Size

Adolfi, F., van de Braak, L., & Woensdregt, M. (2024). From Empirical Problem-Solving to Theoretical Problem-Finding Perspectives on the Cognitive Sciences. Computational Brain & Behavior, 7(4), 572–587. https://doi.org/10.1007/s42113-024-00216-6

Albers, C. (2019). The problem with unadjusted multiple and sequential statistical testing. Nature Communications, 10(1), 1921. https://doi.org/10.1038/s41467-019-09941-0

Bazerman, M. H., & Samuelson, W. F. (1983). I won the auction but don't want the prize. Journal of Conflict Resolution, 27(4), 618–634.

Brownell, S. E., Wenderoth, M. P., Theobald, R., Okoroafor, N., Koval, M., Freeman, S., Walcher-Chevillet, C. L., & Crowe, A. J. (2014). How Students Think about Experimental Design: Novel Conceptions Revealed by in-Class Activities. BioScience, 64(2), 125–137. https://doi.org/10.1093/biosci/bit016

Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475

Cano-Corres, R., Sánchez-Álvarez, J., & Fuentes-Arderiu, X. (2012). The Effect Size: Beyond Statistical Significance. EJIFCC, 23(1), 19–23.

Cohen, J. (1977). Statistical Power Analysis for the Behavioral Sciences. Elsevier Science & Technology. https://ebookcentral-proquest-com.proxy.library.upenn.edu/lib/upenn-ebooks/detail.action?docID=1882849

Confirmatory versus Exploratory Research. (2024, April 5). Irving Institute for Clinical and Translational Research. https://www.irvinginstitute.columbia.edu/confirmatory-versus-exploratory-research

Danziger, M., Collazo, A., Dirnagl, U., & Toelch, U. (2022). Balancing sensitivity and specificity in preclinical research (p. 2022.01.17.476585). bioRxiv. https://doi.org/10.1101/2022.01.17.476585

Devezer, B., & Buzbas, E. O. (2023). Rigorous exploration in a model-centric science via epistemic iteration. Journal of Applied Research in Memory and Cognition, 12(2), 189–194. https://doi.org/10.1037/mac0000121

Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (n.d.). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), 200805. https://doi.org/10.1098/rsos.200805

Experimental design Landing page | NC3Rs EDA. (n.d.). Retrieved March 20, 2025, from https://eda.nc3rs.org.uk/index.php/experimental-design

Gigerenzer, G. (2018). Statistical Rituals: The Replication Delusion and How We Got There. Advances in Methods and Practices in Psychological Science, 1(2), 198–218. https://doi.org/10.1177/2515245918771329

Heinsberg, L. W., & Weeks, D. E. (2022). Post hoc Power is Not Informative. Genetic Epidemiology, 46(7), 390–394. https://doi.org/10.1002/gepi.22464

Hoenig, J. M., & Heisey, D. M. (2001). The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897

Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

Jaeger, R. G., & Halliday, T. R. (1998). On Confirmatory versus Exploratory Research. Herpetologica, 54, S64–S66.

Krzywinski, M., & Altman, N. (2013). Power and sample size. Nature Methods, 10(12), 1139–1140. https://doi.org/10.1038/nmeth.2738

May, J., Ketchum, J., & Looney, S. (2016). Simplified Tools for Sample Size Determination for Correlation Coefficient Inference. Proceedings of the Joint Statistical Meetings. https://ww2.amstat.org/meetings/proceedings/2016/data/presinfo231324.cfm

Negida, A. (2020). Sample Size Calculation Guide - Part 7: How to Calculate the Sample Size Based on a Correlation. Advanced Journal of Emergency Medicine, 4(2), e34. https://doi.org/10.22114/ajem.v0i0.344

Reinagel, P. (2023). Is N-Hacking Ever OK? The consequences of collecting more data in pursuit of statistical significance. PLOS Biology, 21(11), e3002345. https://doi.org/10.1371/journal.pbio.3002345

Savitz, D. A., Wise, L. A., Bond, J. C., Hatch, E. E., Ncube, C. N., Wesselink, A. K., Willis, M. D., Yland, J. J., & Rothman, K. J. (2024). Responding to Reviewers and Editors About Statistical Significance Testing. Annals of Internal Medicine, 177(3), 385–386. https://doi.org/10.7326/M23-2430

Schwab, S., & Held, L. (2020). Different worlds Confirmatory versus exploratory research. Significance, 17(2), 8–9. https://doi.org/10.1111/1740-9713.01369

Scott, S., Kranz, J. E., Cole, J., Lincecum, J. M., Thompson, K., Kelly, N., Bostrom, A., Theodoss, J., Al‐Nakhala, B. M., Vieira, F. G., Ramasubbu, J., & Heywood, J. A. (2008). Design, power, and interpretation of studies in the standard murine model of ALS. Amyotrophic Lateral Sclerosis, 9(1), 4–15. https://doi.org/10.1080/17482960701856300

Serdar, C. C., Cihan, M., Yücel, D., & Serdar, M. A. (2021). Sample size, power and effect size revisited: Simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochemia Medica, 31(1), 010502. https://doi.org/10.11613/BM.2021.010502

Sert, N. P. du, Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., Garner, P., Holgate, S. T., Howells, D. W., Hurst, V., Karp, N. A., Lazic, S. E., Lidster, K., MacCallum, C. J., Macleod, M., … Würbel, H. (2020). Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0. PLOS Biology, 18(7), e3000411. https://doi.org/10.1371/journal.pbio.3000411

Sullivan, G. M., & Feinn, R. (2012). Using Effect Size—Or Why the P Value Is Not Enough. Journal of Graduate Medical Education, 4(3), 279–282. https://doi.org/10.4300/JGME-D-12-00156.1

Szucs, D. (2016). A Tutorial on Hunting Statistical Significance by Chasing N. Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.01444

Thibault, R. T., Zavalis, E. A., Malicki, M., & Pedder, H. (2024). An evaluation of reproducibility and errors in published sample size calculations performed using G*Power (p. 2024.07.15.24310458). medRxiv. https://doi.org/10.1101/2024.07.15.24310458

Waldron, S., & Allen, C. (2022). Not all pre-registrations are equal. Neuropsychopharmacology, 47(13), 2181–2183. https://doi.org/10.1038/s41386-022-01418-x

Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1. https://doi.org/10.1017/S0140525X20001685

name here

Instructor guide:

Sample Size

Overview and Introduction

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript