In our first post – “Where is the data? Scientific Research”, EkaLore discussed the value of the Supply Chain of Data. This post focuses on the “Replication Crisis”, a failure of that supply chain.
Less well known are ongoing controversies known as the “Replication Crisis” and the use of citations to previous bad research (more than 12,000 citations in newer papers to retracted or discredited work just for the ‘Top 10’ bad research works).
The “Replication Crisis” discussions are found in the fields of psychology, social sciences, and medicine. In some analyses, the majority of research findings published in renowned journals could not be reproduced to reach the same conclusions (for many different reasons). In cases where academic or research misconduct has been retrospectively analyzed, failures to retain data, computer codes, analysis runs, or chronological records have been reviewed. Particularly in this age where almost all research is captured, collated, processed, and analyzed by computing resources, the absence of data artifacts is troublesome. Often computer codes, and data collections, are performed by less skilled technicians (often graduate students or interns). Technical standards, testing, and validation are often left incomplete in these actions compared to experienced data engineers or analysts.
In the medical field, certified biostatisticians are usually present for research studies. However, the retention of the data's analytical and statistical testing and the analytical process's quality often varies wildly (“p-hacking” again). Inspections and reviews of computer codes, data collections, and artifacts (often graphic representations or computer produced imaging) have often shown the need for higher standards and improved retention for archival purposes. The Replication Crisis is especially seen in recent events such as Covid-19 research, research on control-substance abuse, and behavioral psychology.
Citations to prior research are a peculiar problem to deal with. Researchers often refer to prior research as context, justification, or further supporting their hypothesis and conclusions. In patent research and presentation, the references have significant property and monetary implications. In many cases, the domain knowledge runs back decades. In prior decades the terminology, tools, methods, and even the mathematical techniques may vary multiple times. Tracing back a research subject can sometimes feel like a fictional detective as the motivations, ideas, and results of long-dead research is reviewed for relevance and conceptual relationships. Prior research on an ‘innovation’ can result in a controversy of terminology (often in different languages) and usage. Citations to prior research will be an ongoing problem for scientific research even as the rate of publications and public data explodes.
EkaLore specializes in helping companies improve their processes that support the Supply Chain of Data. You can find more posts at www.ekalore.com/ARS If you’d like to talk with us, send us a note at sales@ekalore.com
Comments