Martin Schweinsberg
Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
Schweinsberg, Martin; Feldman, Michael; Van Assen, Marcel A L M; Bernstein, Abraham; Staub, Nicola; Sommer, S Amy; van den Akker, Olmo R; van Aert, Robbie; Liu, Yang; Althoff, Tim; Heer, Jeffrey; Kale, Alex; Mohamed, Zainab; Amireh, Hashem; Venkatesh Prasad, Vaishali; Bernstein, Abraham; Robinson, Emily; Snellman, Kaisa; Sommer, S Amy; Otner, Sarah MG; Robinson, David; Madan, Nikhil; Silberzahn, Raphael; Goldstein, Pavel; Tierney, Warren; Murase, Toshio; Mandl, Benjamin; Viganola, Domenico; Strobl, Carolin; Schaumans, Catherine BC; Kelchtermans, Stijn; Naseeb, Chan; Garrison, S Mason; Yarkoni, Tal; Chan, CS; Prestone, Adie; Alaburda, Paulius; Albers, Casper; Alspaugh, Sara; Alstott, Jeff; Nelson, Andrew A; Ari?o de la Rubia, Eduardo; Arzi, Adbi; Bahn?k, ?t?p?n; Baik, Jason; Winther Balling, Laura; Banker, Sachin; Baranger, David AA; Barr, Dale J; Barros-Rivera, Brenda; Bauer, Matt; Blaise, Enuh; Boelen, Lisa; Bohle Carbonell, Katerina; Briers, Robert A; Burkhard, Oliver; Canela, Miguel...
Authors
Michael Feldman
Marcel A L M Van Assen
Abraham Bernstein
Nicola Staub
S Amy Sommer
Olmo R van den Akker
Robbie van Aert
Yang Liu
Tim Althoff
Jeffrey Heer
Alex Kale
Zainab Mohamed
Hashem Amireh
Vaishali Venkatesh Prasad
Abraham Bernstein
Emily Robinson
Kaisa Snellman
S Amy Sommer
Sarah MG Otner
David Robinson
Nikhil Madan
Raphael Silberzahn
Pavel Goldstein
Warren Tierney
Toshio Murase
Benjamin Mandl
Domenico Viganola
Carolin Strobl
Catherine BC Schaumans
Stijn Kelchtermans
Chan Naseeb
S Mason Garrison
Tal Yarkoni
CS Chan
Adie Prestone
Paulius Alaburda
Casper Albers
Sara Alspaugh
Jeff Alstott
Andrew A Nelson
Eduardo Ari�o de la Rubia
Adbi Arzi
�t?p�n Bahn�k
Jason Baik
Laura Winther Balling
Sachin Banker
David AA Baranger
Dale J Barr
Brenda Barros-Rivera
Matt Bauer
Enuh Blaise
Lisa Boelen
Katerina Bohle Carbonell
Prof Robert Briers R.Briers@napier.ac.uk
Professor
Oliver Burkhard
Miguel-Angel Canela
Laura Castrillo
Timothy Catlett
Olivia Chen
Michael Clark
Brent Cohn
Alex Coppock
Nat�lia Cuguer�-Escofet
Paul G Curran
Wilson Cyrus-Lai
David Dai
Giulio Valentino Dalla Riva
Henrik Danielsson
Rosaria de FSM Russo
Niko de Silva
Curdin Derungs
Frank Dondelinger
Carolina Duarte de Souza
B Tyson Dube
Marina Dubova
Ben Mark Dunn
Peter Adriaan Edelsbrunner
Sara Finley
Nick Fox
Timo Gnambs
Yuanyuan Gong
Erin Grand
Brandon Greenawalt
Dan Han
Paul HP Hanel
Antony B Hong
David Hood
Justin Hsueh
Lilian Huang
Kent N Hui
Keith A Hultman
Azka Javaid
Lily Ji Jiang
Jonathan Jong
Jash Kamdar
David Kane
Gregor Kappler
Erikson Kaszubowski
Christopher M Kavanagh
Madian Khabsa
Bennett Kleinberg
Jens Kouros
Heather Krause
Angelos-Miltiadis Krypotos
Dejan Lavbi?
Rui Ling Lee
Timothy Leffel
Wei Yang Lim
Silvia Liverani
Bianca Loh
Dorte L�nsmann
Jia Wei Low
Alton Lu
Kyle MacDonald
Christopher R Madan
Lasse Hjorth Madsen
Christina Maimone
Alexandra Mangold
Adrienne Marshall
Helena Ester Matskewich
Kimia Mavon
Katherine L McLain
Amelia A McNamara
Mhairi McNeill
Ulf Mertens
David Miller
Ben Moore
Andrew Moore
Eric Nantz
Ziauddin Nasrullah
Valentina Nejkovic
Colleen S Nell
Andrew Arthur Nelson
Gustav Nilsonne
Rory Nolan
Christopher E O�Brien
Patrick O�Neill
Kieran O�Shea
Toto Olita
Jahna Otterbacher
Diana Palsetia
Bianca Pereira
Ivan Pozdniakov
John Protzko
Jean-Nicolas Reyt
Travis Riddle
Amal Akmal Ridhwan Omar Ali
Ivan Ropovik
Joshua M Rosenberg
Stephane Rothen
Michael Schulte-Mecklenbeck
Nirek Sharma
Gordon Shotwell
Martin Skarzynski
William Stedden
Victoria Stodden
Martin A Stoffel
Scott Stoltzman
Subashini Subbaiah
Rachael Tatman
Paul H Thibodeau
Sabina Tomkins
Ana Valdivia
Gerrieke B Druijff-van de Woestijne
Abstract
In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.
Journal Article Type | Article |
---|---|
Acceptance Date | Jul 1, 2021 |
Online Publication Date | Jun 17, 2021 |
Publication Date | 2021-07 |
Deposit Date | Mar 8, 2021 |
Publicly Available Date | Jun 17, 2021 |
Print ISSN | 0749-5978 |
Publisher | Elsevier |
Peer Reviewed | Peer Reviewed |
Volume | 165 |
Pages | 228-249 |
DOI | https://doi.org/10.1016/j.obhdp.2021.02.003 |
Keywords | Crowdsourcing data analysis, Scientific transparency, Research reliability, Scientific robustness, Researcher degrees of freedom, Analysis-contingent results |
Public URL | http://researchrepository.napier.ac.uk/Output/2750751 |
Files
Same Data, Different Conclusions: Radical Dispersion In Empirical Results When Independent Analysts Operationalize And Test The Same Hypothesis
(3.5 Mb)
PDF
Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/
Copyright Statement
Licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.
You might also like
Making predictions of mangrove deforestation: a comparison of two methods in Kenya.
(2013)
Journal Article
The ecological effectiveness of protected areas: The United Kingdom.
(2006)
Journal Article
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search