Data · dataset · 2025
pvaluesim.m
Listed in IISH Dataverse
These are matlab files to show the effect of p-hacking through a visual simulation.
Description
It simulates a natural experiments with two groups with the default of n = 20 each, both drawn from the standard normal distribution. Default is 1000 simulations and the first 15 are visualized.
The 95% confidence intervals of the difference between the two group means are shown (green means that zero is included in the 95% confidence interval, red means that zero is not included in the 95% confidence interval). It is recorded how often in the simulation the null hypothesis is rejected at a 5% level. The effect of the following four p-hacking cheats can be simulated: A: Not doing the test once, but test two for two variables and their mean.
Read the rest (3 more)
B: If no no significant effect is found, add 10 cases to each of the experimental groups C: Add tests with a covariate (such as gender) and a test of an interaction effect of gender and the experimental condition. D: Add a third experimental condition and repeat the tests for any pair of experimental condition. In matlab, run pvaluesim to start the GUI.
Note that higher values than 5% are found because essentially multiple tests are done without correcting for the number of tests. This simulation is the implementation of: Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.
Psychological science, 22(11), 1359-1366.
Links
Where it is published
- Dataverse dataset page dataverse.nl/dataset.xhtml?persistentId=doi%3A10.34894%2F58CAUD ↗
landing page · from datasets iisg amsterdam
- DOI doi.org/10.34894/58caud ↗
DOI / persistent id · from datasets iisg amsterdam
Catalogue records · 1
- Dataverse API dataverse.nl/api/datasets/:persistentId/?persistentId=doi%3A10.34894%2F58CA… ↗
metadata API · from datasets iisg amsterdam
Topics
- Stated by source
- Other
- From keywords
- Applied statistics · Mathematics & Statistics · Simulation
Provenance · 1 source records, 10 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| IISH Dataverse | doi:10.34894/58CAUD | 6 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| concepts[field].anzsrc:field:490501 | mapping · datasets iisg amsterdam | vocabulary-mapper@1.0.0 | keywords['Applied Statistics'] |
| concepts[field].dataverse_subject:other | source · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 | /subjects |
| concepts[field].local:field:mathematics-statistics | mapping · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 | /subjects |
| concepts[method].local:method:simulation | mapping · datasets iisg amsterdam | vocabulary-mapper@1.0.0 | keywords['Simulation'] |
| created_date | source · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 | |
| description | source · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 | /description |
| publication_date | source · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 | |
| title | source · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 | /name |
| updated_date | source · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 | |
| version_label | source · datasets iisg amsterdam | connector:datasets_iisg_amsterdam@1.0.0 |