Raw Data vs. Processed Data: Best Practices for Storage

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

By the end of this article you will be able to define raw data vs. processed data for your own project, set up a folder structure that keeps the two physically separate, freeze your raw files with checksums, build a scripted processing pipeline that can regenerate every processed file from raw inputs, and run a reproducibility checklist before you publish or hand a project to someone else. You do not need special software to start. A laptop, a text editor, and access to some form of backup storage (an external drive, an institutional server, or a cloud account such as Dropbox or Microsoft Azure) are enough. If you work with high-throughput data, a workflow manager such as Snakemake will pay for itself within the first rerun.

Raw Data vs. Processed Data: Definitions That Matter

Raw data are the files that come directly off an instrument or data collection system before any transformation, filtering, annotation, or quality control. Examples include DICOM image volumes from a scanner, FASTQ files from a sequencer, MedPC output files from an operant behavior chamber, EEG recordings exported from an amplifier, Raman spectral files from a spectrometer, and chromatographic traces from an LC-MS run.

Processed data are any files derived from raw data by a transformation: quality-filtered reads, aligned and variant-called genomic outputs, converted image formats, structured spreadsheets, feature tables, and statistical summaries. The distinction can have layers. In toxicology, new approach methods (NAMs) explicitly recognize multiple data layers between a machine-generated output and the final hit definition, and each transition between layers can involve several processing steps [1]. In imaging, a tool such as MedicalDataHandler converts raw DICOM files into the Nearly Raw Raster Data (NRRD) format for deep learning, which means the "raw" DICOM layer and the converted layer are distinct artifacts with distinct storage needs [2]. In neurophysiology, Meta-EEGs reorganizes raw EEG recordings into a standardized, reduced-volume structured format suitable for AI model input, again producing a separate derived layer [3].

The practical rule: raw data are the evidence, processed data are the claims. A reviewer who wants to check your claims needs both, plus the exact pipeline that connects them.

Why You Keep Both: Reproducible Research Depends on the Raw Layer

Data management covers organization, documentation, storage, sharing, and preservation, and it underpins scientific integrity and reproducibility in experimental biomedicine [4]. Well-organized and well-documented data allow others to validate results and build on them, and they protect your own career when a collaborator, reviewer, or funder asks for the underlying files [4].

The strongest argument for keeping raw data untouched is that the same raw data, processed in different ways, can produce different final outcomes. That principle was spelled out for toxicological NAMs, where a change in any processing step can alter robustness, performance, regulatory readiness, and the final hit call, which is why the processing pipeline itself belongs in the report [1]. The same logic applies everywhere. Methodological variation at each stage of a microbiome workflow, from sample collection to bioinformatic processing, demonstrably affects every downstream step and has undermined reproducibility across studies [5]. If you overwrite your raw files during processing, you lose the ability to test whether a different pipeline would reach the same conclusion.

There is also a volume problem. The Preclinical Addiction Research Consortium phenotyped more than 3,000 rats and accumulated over 100,000 raw data files from operant behavior experiments, which forced the team to design an explicit storage and processing architecture rather than handling files by hand [6]. The Swiss HIV Cohort Study viral sequencing database was built specifically to handle high loads of raw and processed sequencing data, storage solutions, downstream bioinformatic tools, and reproducibility for a cohort of more than 21,000 people [7]. Even a single bovine bioacoustics study produced 90 hours of raw recordings totaling 65.6 GB before any processing [8]. Raw vs. processed data storage is not an administrative chore. It is the load-bearing structure of your project.

Raw vs Processed Data Storage, Step by Step

The following walkthrough assumes a small to medium project, for example a sequencing or imaging study with tens to hundreds of files. Every command shown runs in a Unix-style shell (macOS terminal, Linux, or Windows Subsystem for Linux).

Step 1: Inventory and freeze the raw layer

Four step flow: copy instrument output into raw, checksum every file, make raw read-only, then verify with md5sum -c before processing.

Create a project directory and copy instrument output into a raw/ subfolder. Then compute a checksum for every file so you can prove later that nothing changed. MD5 and SHA-256 are standard checksum algorithms covered in any introductory computing text.

mkdir -p myproject/raw myproject/processed myproject/scripts myproject/metadata
cp /path/to/instrument_output/*.fastq.gz myproject/raw/
cd myproject/raw
find . -type f -exec md5sum {} \; > ../raw_checksums.md5

Expected content of raw_checksums.md5:

3f5a8c42d9e1b770a4c6f2e8d1b90345  ./run1_sampleA01.fastq.gz
7c9e0d1b4a2f6835e7d0c2b1a9f8e6d4  ./run1_sampleA02.fastq.gz

To verify later that the raw layer is unchanged, run md5sum -c raw_checksums.md5 from the project root. Expected output is one line per file followed by OK. If any line reports FAILED, the file changed or was corrupted, and you should restore it from backup before processing anything.

Treat everything in raw/ as read-only from this moment on. On a shared server you can enforce this with chmod -R a-w raw/. On cloud storage, use versioning or an archive tier where overwrites are not silently allowed.

Step 2: Separate raw from processed at the folder level

A single standard hierarchy, where folders reflect the logical flow from raw acquisition to processed outcomes, improves traceability and reduces the risk of misplaced files. This principle was formalized for REDCap-based clinical research projects, and the same structure works for bench science [9]. Use the listing below as a starting template:

myproject/
  raw/
  processed/
  scripts/
  metadata/
  README.md
  raw_checksums.md5

Rules that keep this structure honest:

Never write an output file into raw/.
Name processed files so their origin is obvious, for example sampleA01.trimmed.fastq.gz derived from sampleA01.fastq.gz.
Keep one metadata file per experiment in metadata/, with sample identifiers that match file names exactly.
For multi-site field work, organize raw files by campaign, site, and instrument before any harmonization step. A reproducible R workflow for indoor air quality sensing did exactly this, organizing raw files by campaign, school, and instrument, then standardizing timestamps including daylight saving changes before merging sensor signals with activity logs [10].

Step 3: Document before you process

Write a README.md at the project root covering what each folder contains, who collected the data and when, instrument settings, and the one command needed to regenerate all processed files. Documentation of procedures, data collection, and analysis methods is what makes transparent and accountable research possible [4]. Add a metadata/ table with one row per sample. If a value is missing, write NA rather than leaving the cell blank, so a script can distinguish missing from forgotten.

Step 4: Version the processing pipeline as code

Write every transformation as a script or workflow file rather than performing it by hand in a spreadsheet or GUI. A whole exome sequencing pipeline built in Snakemake, spanning quality control with FastQC and Trimmomatic, alignment with BWA, and variant calling with GATK, shows the pattern: one versioned workflow definition that ties each tool to its inputs and outputs, giving scalability and reproducibility across the full raw-to-annotation path [11].

A minimal Snakefile skeleton for a trimming step looks like this:

rule trim:
    input:
        "raw/run1_{sample}.fastq.gz"
    output:
        "processed/run1_{sample}.trimmed.fastq.gz"
    shell:
        "trimmomatic SE -phred33 {input} {output} "
        "ILLUMINACLIP:adapters.fa:2:30:10 LEADING:3 TRAILING:3 MINLEN:36"

Running snakemake then regenerates every processed file from raw inputs. If you prefer a graphical or configuration-based approach, choose a tool that encodes the pipeline in a shareable declarative form. The TRaP toolkit for Raman spectroscopy stores complete processing workflows in a single configuration file so a collaborator can rerun the identical pipeline on a different instrument platform [12]. The key property is identical regardless of tooling: the pipeline is text, it is version controlled, and it travels with the data.

Step 5: Process into a derived layer, never in place

Run the pipeline so all outputs land in processed/. After the run, checksum the processed layer too:

cd myproject/processed
find . -type f -exec md5sum {} \; > ../processed_checksums.md5

Now record, in the README or a provenance.md file, the pipeline version, tool versions (trimmomatic -version, bwa release), and the date. Many large projects formalize this with a relational database. The operant behavior consortium converted raw MedPC files into structured spreadsheets with scripts, integrated them with standardized experimental and behavioral metadata, and loaded everything into a SQL database on Microsoft Azure, which enabled automated quality control, backups, and daily summary reports [6]. You do not need a database at small scale, but you do need the same discipline: raw input, scripted transformation, derived output, recorded versions.

Step 6: Back up both layers with different expectations

Raw data are irreplaceable. If the samples are consumed or the experiment cannot be repeated, losing raw data ends the project. Processed data are regenerable from raw data plus code, which changes the storage calculus. A three-copy rule (one working copy, one local backup, one off-site or cloud copy) is standard practice in research data management [4]. Cloud services are a practical off-site tier for high-throughput projects, as the behavior phenotyping pipeline demonstrated with Dropbox as the ingest layer and Azure as the database and backup tier [6]. Institutional servers and sequencer-core storage fill the same role in most universities.

Step 7: Verify, then publish or archive

Before submission or archiving, confirm the raw checksums still pass, confirm the pipeline runs end to end from a clean directory, and confirm the README explains everything a stranger would need. For public sharing, match the artifact to the venue. Structured processed datasets from EEG studies have been released on Figshare alongside working code on GitHub, with the raw recordings remaining in their source repositories [3]. The Chromatograms package in R illustrates a useful long-term property: because its storage backends are interchangeable and analysis code stays independent of where data reside, a published peak table plus the raw files is enough to recover the corresponding chromatograms for 4,063 cohort samples years later [13]. Design your storage so that a future reader can make the same round trip from raw files back to your published results.

Storage Options at a Glance

Comparison of raw, metadata and processed layers by contents, change policy, backup priority and whether they can be rebuilt.

Storage layer	What it holds	Change policy	Typical medium	Backup priority
Raw	Instrument output, untouched	Read-only after checksum	Local server plus cloud archive tier	Highest, irreplaceable
Metadata	Sample sheets, protocols, instrument settings	Append or version only	Version-controlled files	High
Processed	QC outputs, aligned files, feature tables	Regenerable, safe to delete and rebuild	Working disk or cloud compute storage	Medium
Pipeline and code	Scripts, workflow files, config files	Version controlled (Git)	Code repository	High, small but critical
Published artifacts	Anonymized processed datasets, code	Immutable after release	Public repository such as Figshare	Handled by repository

Raw vs Processed Data Storage Tips

Checksum on day one, not on submission day. Checksums prove integrity only if they are computed before any file has had a chance to change.
Keep tool versions in the project, not in your memory. A one-line versions.txt saved next to the pipeline costs nothing and answers the question reviewers ask most.
Convert deliberately, and document the conversion. When raw formats are unwieldy, a documented conversion to an analysis-friendly format is fine. MedicalDataHandler, for example, converts DICOM collections into NRRD with patient identifiers and unique identifiers preserved, reducing reliance on custom scripting and promoting reproducible handling [2]. The conversion is a recorded, scriptable step, not an ad hoc drag-and-drop.
Use a lab notebook system that ties raw, code, and documentation together. Fragmented tools and inconsistent documentation undermine reproducibility in data-intensive fields such as neuroimaging. The IntegriLAB framework addresses this by consolidating data, documents, and code repositories into one system with cryptographically signed, immutable audit records of research activity [14]. Any electronic lab notebook with version history gets you most of the way there.
Curate derived variables with explicit definitions. When raw data become analysis variables, the definitions matter. In an electronic health record study preparing data for AI, raw data were carefully curated using validated definitions to create variables such as age, race, gender, and healthcare utilization, and the processed output showed lower missingness and better representation than the raw layer [15]. Write down the definition of every derived variable.
Standardize timestamps immediately. Field and clinical data routinely span daylight saving changes and mixed time zones. The air quality workflow mentioned above standardized timestamps, including daylight saving changes, before aligning sensor signals with activity logs [10]. Do this in the first processing step, not during analysis.

Common Mistakes and Limitations

Processing in place. Overwriting raw files with filtered versions is the single most damaging habit. Once the raw layer is gone, no pipeline can recover it, and the same raw data processed differently can legitimately give different answers [1].
Storing only processed data. A feature table without the underlying raw files cannot be reanalyzed, reprocessed with improved methods, or checked for processing errors. Genomics databases such as the SHCND exist precisely because both raw and processed layers must be retained and managed together [7].
Manual, undocumented transformations. Steps performed by hand in a spreadsheet or GUI are invisible to anyone who was not at the keyboard, including you six months later. Customized, case-by-case code with manual reconciliation of processing steps is a documented bottleneck for reproducibility in spectroscopy [12] and elsewhere.
Checksums without verification. Computing a checksum list once and never running the check again detects nothing. Schedule verification after every move, migration, or lab-server update.
File names that encode too little. final_v3_REAL.xlsx carries no sample identity, date, or processing history. Identifiers should match the metadata table exactly.
Confusing conversion with analysis. Format conversion (DICOM to NRRD, MedPC text to structured tables) is a processing step that must be scripted and versioned like any other [2] [6].
Assuming processed data are always derivable. Some derived products depend on external resources that change, such as reference databases, instrument calibration files, or cloud services. Keep those dependencies versioned alongside the pipeline.
A note on scope. Storage discipline preserves data integrity, but it does not by itself guarantee valid science. Pipeline choices at every stage influence downstream outcomes, so methodology deserves the same scrutiny as storage [5]. If your work involves human subjects or clinical records, follow your institution's data governance requirements for de-identification and access control before sharing anything.

A Reproducibility Checklist

Run this list before submission, group onboarding, or archival.

Raw files are in a read-only raw/ folder and a checksum manifest exists.
md5sum -c raw_checksums.md5 passes for every file.
No processed file shares a folder with raw files.
Every processed file can be traced to a raw file by a scripted step.
The full pipeline runs end to end from a clean directory with one command.
Tool and workflow versions are recorded in the project.
A README explains folder contents, data provenance, and how to rerun the analysis.
The metadata table covers every sample, with explicit NA for missing values.
Derived variables have written definitions.
Timestamps are standardized to one time zone convention across all files.
Three copies of the raw layer exist, with at least one off-site or cloud copy.
Data, code, and documentation are linked from a single entry point, such as a repository or electronic lab notebook record.
Any public release includes the processed dataset, the pipeline, and enough raw data (or a documented path to it) for the round trip from raw to result.

Frequently Asked Questions

What is the difference between raw data and processed data?

Raw data are unmodified instrument or capture outputs such as FASTQ files, DICOM images, or spectral files. Processed data are anything derived from them through filtering, alignment, conversion, annotation, or statistics. The raw layer is the evidence, and the processed layer is the basis of your published claims.

Should I store processed data if I can regenerate it from raw data?

Yes, in most cases. Regeneration can take hours of compute, and some steps depend on external resources that may change or disappear. Store processed data in a separate processed/ folder, treat it as disposable in principle, and keep the pipeline that rebuilds it versioned next to it.

How long should I keep raw data?

Keep raw data for as long as your funder, institution, and journal expect, which for many funded biomedical projects means years past publication. Requirements vary, so check your funding agency's data sharing policy and your institutional data retention schedule, and plan preservation as part of initial data management rather than as an afterthought.

Can processed data from one step be the raw data for the next step?

Yes, and this is normal. Converted imaging formats such as NRRD derived from DICOM, or structured tables derived from instrument text logs, function as raw inputs for later analysis stages. What matters is that every layer transition is scripted, documented, and checksummed, so the chain from true instrument output to final result stays intact.

What metadata should I store with raw data?

Store sample identifiers that match file names exactly, collection date, instrument and settings, operator, protocol version, and any conditions that could affect interpretation. Keep this in a structured table in a metadata/ folder, and document procedures and data collection accurately, since transparent research practice depends on it.

How do I make my processing pipeline reproducible?

Encode every transformation in versioned scripts or a workflow manager such as Snakemake, record tool versions, and prefer pipelines expressible in a single declarative configuration file that a collaborator can rerun on different hardware. Then verify by rerunning the whole pipeline from raw inputs in a clean directory and comparing outputs.

What checksum should I use and how do I verify it?

MD5 and SHA-256 are both standard and widely supported. Generate a manifest with md5sum or sha256sum at the time raw files are frozen, then run the checksum command with the verify flag (md5sum -c) after every transfer or migration. Any FAILED line means the file changed and must be restored from a known good copy.

Where should I publish raw and processed data?

Publish processed datasets and code in venues that issue persistent identifiers, such as Figshare for structured datasets and GitHub for code, and deposit raw data in the domain repository your field expects, for example a sequencing archive for genomic data. Include the pipeline and a README so a reader can travel from your raw files to your published figures without emailing you.