data_flow.md

Typical flow of data

Source data

• Experimental results
• Large data sets
• Manually collected data
• External

Intermediate

• Derived data
• Tidy data
• Curated sets

Analyses

• Exploratory
• Model building
• Hypothesis testing

Dissemination

• Manuscript, report, presentation, ...

Preserve

• Version data sets
• Backup
• Protect

Reproduce

• Automate your builds
• Use workflow tools (e.g. Snakemake)

Trace

• Multiple iterations.
• Code versioning (Git)

Track

• Through multiple versions

flow of the data is downstream (mostly), but you are going back and forth applies to all data (financial report, lab safety assessment)