Overview of Reusability Standards

In this document we are adopting the National Academies of Sciences, Engineering and Medicine 2019 report on Reproducibility and Replicability in Science definitions of reproducibility and replicability.

Reproducibility means obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis. Replicability means obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.

Reproducibility is a cornerstone of scientific research, especially for transparent evaluation of research claims. Equally important is the potential for scientific procedures to be reused and modified in order to carry out new research. While clear documentation is essential to reusability, new technologies also offer the potential to increase the reproducibility and reusability of model code.

A critical metric of reproducibility is whether a model can consistently reproduce the published results that claim to have been generated by the model. Reproducibility and reusability can also be enhanced if model code is accompanied by explicit workflows that demonstrate how the model generates and transforms its output data into its published findings.

Goals for Reusability Standards

An independent reviewer should be able to easily download, build, and execute a computational model and verify that it meets the reproducibility claims stated by the model author(s).

Minimal Reusability Standards

This set of minimal standards also can be adopted by journals to ensure that submitted publications meet baseline reproducibility and reusability requirements.

Computational models must:

  • meet OMF minimal standards for Accessibility and Documentation
  • include a minimal set of metadata describing the model including: name, version, description, authors and contributors, OSI approved license, a permanent link to a versioned codebase landing page that broadly follows the FORCE11 software citation guidelines
  • include provenance of all input datasets: permanent links to versioned input data dependencies (and outputs where possible)
  • favor open file formats for data inputs and outputs (e.g., CSV, netCDF, YAML/JSON)
  • provide information on all versioned software and system dependencies (operating system, software and system libraries) with relevant software citations for key dependencies
  • include instructions on how to set up a compute environment to compile / interpret / execute the software
    • software with large compute or data requirements can be problematic:
      • provide representative input data samples along with sampling methodology
      • provide durable link to wholetale or other provenance-tracked computation
  • clearly describe what reproducibility measures or metrics are applicable to the given software
  • provide range of acceptable outcomes for software with stochastic components
  • provide descriptions of possible input parameters and expected outputs with units and format (e.g., shape, data type, etc.)
  • include output analyses and workflows
    • scripts to transform raw data -> intermediate -> figures
    • subsets of intermediate data if prohibitive to generate

Ideal Reusability Standards

In order to meet the ideal standards, computational models should:

  • provide durable containerization that follows archival best practices with archived container images
  • provide a link to an active source code repository if still in active development
  • have continuous integration services that compile and run automated unit or integration test suites on a periodic basis
  • include metadata on related research outputs (publications, other software, relationship)
  • additional domain specific standards if any (examples?)

Cyberinfrastructure and Tools for Implementation of Reusability Standards

Build Docker images from research code:

Archive computational pipelines:

OMF may consider building some github template repositories or scaffolding for common modeling frameworks that reduce friction of adoption.

Examples and References for Reusability

Issues / Errata

  • It is fine to use commercial and / or closed source products so long as the researchers' source code is available and the version of the product is specified (e.g., matlab 9.5, AnyLogic 8.4, ArcGIS 10.8)