How to Implement a Bioinformatics Data Management Plan

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

By the end of this guide you will be able to implement a bioinformatics data management plan for a real sequencing project: you will know how big your genomics data storage footprint actually is, you will have a documented directory structure and naming convention, a working checksum and backup routine, a metadata capture system, a version-controlled analysis pipeline, and a written plan that satisfies a funder review. Every step includes runnable commands and the output you should expect to see.

You need the following on hand before you start:

A Linux server, workstation, or cloud account where sequencing data lands
Basic shell access and permissions to create directories
samtools installed for alignment file handling
A version control system such as Git for pipeline code
An object storage target or external destination for backups (S3, Google Cloud Storage, institutional storage, or a synced drive accessed through rclone)
Optionally, a data capture platform such as REDCap if your project collects clinical or phenotypic variables alongside sequence data

The plan you build here follows the same logic used by large multi-site programs. The HAPIN trial, a four-country randomized study, managed more than 50 million data points across 80 case report forms and over 4,000 variables by standardizing capture, variable names, and timelines before the first sample was collected [1]. A sequencing project of any size benefits from the same discipline applied at proportionate scale.

What a Bioinformatics Data Management Plan Actually Covers

A data management plan (DMP) is a written document that answers six questions about a project's data: what data will exist, where it will live during the project, how it will be documented, who can access it, how it will be preserved after the project, and what will be shared publicly. In bioinformatics, those answers have to account for three things that distinguish our field from general research data management.

First, volume. Raw sequence output is large and the derived files multiply it. Second, heterogeneity. Molecular diagnostics and multi-omics work produce mixed data types that challenge reproducibility and standardization unless formats and definitions are fixed in advance [2]. Third, sensitivity. Human genomic data carries privacy obligations that a generic lab notebook dataset does not.

Funders increasingly require this document explicitly. Since January 2023, NIH requires researchers submitting proposals to include a Data Management and Sharing Plan, and the NIH Genomic Data Sharing policy has applied to genomic data for years [3]. Even without a funder mandate, a written plan is the difference between a cohort you can reanalyze in three years and a directory of orphaned FASTQ files nobody can interpret.

Step-by-Step Implementation

The ten steps below take you from an empty server to an operating, documented system. Work through them in order. Steps 1 through 4 are planning and foundation, steps 5 through 7 are operational, and steps 8 through 10 cover protection, access, and handoff.

Step 1: Inventory Your Data and Estimate Storage

Worked storage calculation: 2000 samples at 95 GB each gives 190 TB raw, 380 TB with two copies, about 570 TB planned capacity.

Start by measuring what you already have. On the server or share where data currently accumulates, run:

du -sh /data/projects/*/raw

Expected output looks like this:

1.8T    /data/projects/rnaseq_2025/raw
14T     /data/projects/wgs_cohort/raw

Then count the files that define your project scale:

find /data/projects/wgs_cohort/raw -name "*.fastq.gz" | wc -l

With counts in hand, estimate forward-looking genomics data storage needs. As rough planning figures, a 30x whole-genome sequence of a human sample produces on the order of 90 to 100 GB of compressed FASTQ, and the aligned BAM file is of similar magnitude. A CRAM file, which stores alignments against a known reference, typically runs smaller than the equivalent BAM. Exact numbers vary with instrument, read length, and coverage, so treat these as order-of-magnitude estimates and verify with your own first batch.

A simple planning calculation in a text file or notebook:

samples           = 2000
per_sample_fastq  = 95 GB
primary_copy_TB   = 2000 * 95 / 1000        = 190 TB
two_full_copies   = 190 * 2                 = 380 TB
plus_working_set  (alignments, reports)     = add roughly 50 percent
planned_capacity                            = about 570 TB

If your project includes sequencing at multiple sites, budget for it deliberately. A global survey of next-generation sequencing and bioinformatics capacity across 52 countries collected information on instrumentation, throughput, data management, and funding sources precisely because these cost lines are often unplanned, and the expanded genomics costing tool now supports budgeting across a wider range of pathogens and laboratory settings [4]. Storage is rarely the line item that gets cut, but transfer, staffing, and archival maintenance frequently are.

Step 2: Write the Plan Document

Write the plan before the first sample arrives. A workable outline has these sections:

Data description. Data types (raw reads, alignments, variant calls, metadata, images), approximate volumes, and whether any data are derived from humans.
Collection and capture. Instruments, software versions, and the system that records sample-level metadata.
Documentation and metadata. The metadata fields every sample must have, controlled vocabularies, and the sample identifier scheme.
Storage and backup. Where primary, secondary, and archive copies live, and the verification method.
Access and security. Who can read, write, and administer each layer, and how controlled-access human data is gated.
Retention and sharing. Which repository receives which outputs, under what timeline, and what stays restricted.
Responsibilities. Named roles for data steward, pipeline maintainer, and capture-system administrator.

For projects involving communities, especially those contributing samples or health information, plan sharing decisions with those communities rather than for them. Investigators examining data sharing through a community-engaged research lens emphasize that questions about to whom the benefits of sharing accrue, and to whom they do not, deserve explicit attention when human data are involved [3]. Write those decisions into the plan document itself, not into an afterthought appendix.

Step 3: Set Up the Directory Structure and Naming Conventions

Create the project scaffold with a single command so every project on the server looks the same:

mkdir -p projects/wgs_cohort/{raw,trimmed,aligned,calls,reports,metadata,scripts,docs}

Verify the result:

ls projects/wgs_cohort

aligned  calls  docs  metadata  raw  reports  scripts  trimmed

The rule that makes this structure durable is that raw/ is immutable. Read data land there once, checksums are generated immediately, and no pipeline ever writes into that directory. All derived content goes to sibling folders, so a pipeline failure never destroys source data.

Fix a filename convention and enforce it mechanically. For Illumina output, a workable pattern is:

{SITE}_{SUBJECT}_{LIB}_{LANE}_R{1|2}_001.fastq.gz

For example: PER_0142_EX_L001_R1_001.fastq.gz. Enforce the pattern with a check you can run before analysis:

ls raw | grep -E '^[A-Z]{3}_[0-9]{4}_[A-Z]{2}_L00[1-4]_R[5]_001\.fastq\.gz$' | wc -l

If this count does not equal your file count from Step 1, you have nonconforming names, and renaming is easier on day one than in year three. Multi-site studies that adopted shared variable names and harmonized structures before enrollment found that early standardization paid for itself across every downstream analysis [1].

Step 4: Standardize Formats, Metadata, and Identifiers

Fix your file formats in the plan: FASTQ for raw reads, BAM or CRAM for alignments, VCF or BCF for variant calls, and a single tabular format for metadata. Heterogeneous data types are a recognized obstacle to reproducibility and standardization in molecular diagnostics and large-scale bioinformatics [2], and the cheapest time to remove heterogeneity is before collection starts.

For metadata, build a relational table with a primary key that ties every file to a sample. A minimal schema:

CREATE TABLE sample (
  sample_id      VARCHAR(32) PRIMARY KEY,
  subject_id     VARCHAR(32) NOT NULL,
  site_code      CHAR(3) NOT NULL,
  collection_date DATE NOT NULL,
  tissue         VARCHAR(64),
  library_kit    VARCHAR(128),
  instrument     VARCHAR(64),
  raw_read_count BIGINT,
  created_at     TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)

This is the same design principle behind long-lived scientific databases. The FlyEx database, built on a relational database management system, holds 2,073,662 quantitative records alongside 2,832 images from 954 embryos, and its value comes from structured fields that let users query, assess data quality, and download defined subsets [5]. Datasets with thin or inconsistent metadata become patchy and incomplete at scale, which is a documented barrier to large-scale synthesis and reanalysis in any data-intensive field [6].

Generate checksums the moment raw files arrive:

cd projects/wgs_cohort/raw
md5sum *.fastq.gz > ../metadata/checksums_raw.md5

Expected content of the checksum file:

d41d8cd98f00b204e9800998ecf8427e  PER_0142_EX_L001_R1_001.fastq.gz
5d41402abc4b2a76b9719d911017c592  PER_0142_EX_L001_R2_001.fastq.gz

Step 5: Build the Data Capture Layer

If your project collects phenotypes, clinical variables, or survey responses alongside sequencing, put those in a structured capture system rather than in spreadsheets. REDCap is a widely used choice. In the HAPIN trial, the data management core implemented identical REDCap projects at each of four country sites, using shared variable names and translated forms, and field staff collected data offline on tablets through the REDCap Mobile Application before syncing [1]. The transferable lessons for any bioinformatics project are three: one project definition replicated per site rather than four divergent ones, shared variable names across sites, and an offline-capable client where connectivity is unreliable.

For numeric or image-derived data such as quantified expression measurements, a relational database keyed to the sample table from Step 4 works well. SiMCAL, a clustering analysis of microarray time-series data, stored its data in MySQL and implemented the analysis in Python with numerical extensions, a lightweight and still entirely workable pattern for small and mid-sized projects [7]. Whatever you choose, the capture layer should enforce required fields at entry, because retrofitting missing metadata is the most expensive correction in bioinformatics data management.

Step 6: Put Data Cleaning and Quality Control on a Schedule

Cleaning is a procedure, not an improvisation. Document your cleaning steps before analysis begins, then apply them identically to every batch. A published systematic approach to cleaning web-collected research data recommends reviewing completion speed, identifying consecutive identical responses, and checking other behavioral indicators of low-quality entries in a defined sequence, and notes that research teams often fail to report their cleaning processes at all [8]. Report yours, in the plan document and in methods sections.

For sequence data, the equivalent schedule looks like this:

Verify checksums on every incoming batch before any analysis.
Run per-sample read quality summaries and adapter-content checks, and record the results in reports/.
Flag samples with abnormal yield or duplication and resolve before alignment.
Log every exclusion decision in a versioned file, with the reason, so the filter can be reproduced or reversed.

A check that catches corrupted transfers immediately:

samtools quickcheck -v aligned/*.bam

No output means all listed files passed structural checks. Any file named in the output is damaged and must be regenerated from source, never patched in place.

Step 7: Version Pipelines and Track Provenance

Every derived file should be reproducible from raw/ plus versioned code. Put pipeline code in Git, pin software versions, and use a workflow engine so execution order is declared rather than remembered. A typical run with Nextflow looks like:

nextflow run nf-core/rnaseq \
  --input samplesheet.csv \
  --outdir results \
  -profile docker \
  -resume

Expected terminal output begins with:

N E X T F L O W  ~  version 24.xx.x
Launching `nf-core/rnaseq` [reverent_turing] DSL2 - revision: xxxxxxxxxx

The -resume flag reuses completed steps, which makes iterative analysis affordable on large cohorts. After a run, record the exact software and parameters used:

nextflow log -f process,container,script > reports/provenance_run0042.txt

Provenance records belong in the plan document's custody chain. When a reviewer asks how a variant call was produced two years later, the answer should be a file, not a memory.

Step 8: Design Storage, Backup, and Retention

Apply a 3-2-1 pattern to genomics data storage: at least three copies of irreplaceable data, on at least two distinct storage systems, with at least one copy off-site or in a different failure domain. Raw reads are irreplaceable. Alignments and callsets are regenerable and can be treated with lighter redundancy.

Stage	Typical format	Regenerable	Copies and verification
Raw reads	FASTQ (gzip)	No	3 copies, checksums verified after every transfer
Alignments	BAM or CRAM	Yes	1 to 2 copies, quickcheck after generation
Variant calls	VCF or BCF	Yes	2 copies, count and sample checks
Metadata	SQL dump or TSV	Partially	3 copies, checksummed, synced on every change
Code and pipelines	Git repository	No	Remote remote mirror, tagged releases

Convert large BAM files to CRAM where downstream tools allow it, because CRAM stores only differences from the reference:

samtools view -C -T Homo_sapiens.GRCh38.fa aligned/sample.bam > aligned/sample.cram

Sync an off-site copy with checksum verification:

rclone sync /data/projects/wgs_cohort/raw remote:wgs_cohort_raw --checksum MD5 --transfers 8

Expected output ends with a summary line such as:

Transferred:  1.200 GiB / 1.200 GiB, 100%, 95 MiB/s, ETA 0s
Checks: 2000/2000, 100%
Elapsed time: 14.2s

Schedule checksum verification of the archive quarterly, not just at write time, because silent bit rot is a documented failure mode of long-lived storage. Retention periods should match funder and institutional policy, and the plan should state explicitly what happens to the data when the grant ends, including who holds administrative access.

Step 9: Control Access and Security

Access control should mirror data sensitivity. Public reference data and deidentified aggregates can sit on shared infrastructure. Raw human genomic data should sit behind authentication, audit logging, and role-based permissions, with distinct roles for steward, analyst, and administrator. Molecular diagnostics and big data work generate recognized privacy and management concerns precisely because of the scale and sensitivity of the data involved [2].

Multi-site collaborations need this designed in from the start. A federated tumor data system built on grid middleware addressed heterogeneous multi-center data by providing a unified access layer, federated storage, and a multiple-authorization security model with real-time monitoring and load balancing [9]. Most sequencing consortia do not need grid middleware, but they do need the underlying idea: sites retain control of their own data while a common authorization scheme governs who queries what.

For human participants, tie access decisions to consent. Where research is community engaged, sharing plans should reflect participant and community expectations, since expanded data sharing raises distributional questions about who benefits and who bears the risks [3].

Step 10: Share, Archive, and Sustain

Deposit findable outputs in a repository appropriate to the data type and write the deposit into the plan as a deliverable with a date, not an intention. NIH policy now requires a Data Management and Sharing Plan with proposals, and the agency has maintained a dedicated genomic data sharing policy for genomic data specifically [3]. Name the repository, the exact datasets to be deposited, the timeline relative to publication, and any access controls in your plan document.

Sustainability is a budget line, not a hope. Survey evidence from 52 countries shows that laboratory demographics, instrumentation, throughput, data management, and funding sources are the variables national genomics strategies are expected to cost and plan for, and the genomics costing tool was expanded to help countries budget these lines across pathogens and laboratory settings [4]. Mirror that structure locally: give data management its own line in every grant, covering storage, staff time, and repository deposition.

Invest in people as deliberately as in disks. A multi-partner Spanish agreement for leishmaniasis surveillance specifies shared databases, shared analytical tools, and genomic training as explicit commitments of the network, recognizing that a data plan without trained people does not execute [10]. Assign at least one named person per project whose job description includes checksums, backups, and metadata quality, and rotate the knowledge so the plan survives staff turnover.

Common Mistakes and Limitations

Five warning signs: no checksums at ingestion, writing into raw, unversioned pipelines, no named owner, and forgotten transfer costs.

These are the failures that show up repeatedly when teams implement a bioinformatics data management plan, listed so you can check your own project against them.

Treating the plan as a grant formality. A document that names no file formats, no repository, and no responsible person will not survive first contact with a real data batch. Write the plan after Step 1's inventory, not before it.
No checksums at ingestion. Transfer corruption is discovered months later, after the original run folder on the instrument has been overwritten, and the sample is lost. Generate checksums the day files arrive.
Writing into the raw directory. One script bug with an absolute path pointed at raw/ can destroy irreplaceable data. Keep raw/ read-only at the filesystem level where possible.
Unversioned pipelines. Results that cannot be tied to a code version, container, and parameter set are not reproducible, they are anecdotes. Version everything in Git from the first week.
Undocumented cleaning decisions. Teams that clean data ad hoc cannot report what they did, which undermines integrity review of the whole analysis [8]. Sequence your cleaning steps and log every exclusion.
Budgeting hardware and forgetting people. Capacity surveys consistently treat data management and staffing as distinct budgeted activities alongside instruments and reagents [4]. If no one owns the plan, the plan is decoration.
Ignoring metadata until analysis. Missing tissue, date, or batch fields are cheap to prevent at capture and expensive to reconstruct later. Patchy metadata is a primary reason large datasets become incomplete and hard to synthesize [6].
Forgetting egress and transfer effort. Cloud storage carries transfer costs and time for moving large cohorts in and out. Model them during planning rather than discovering them during a repository deposit deadline.
Single-site assumptions in multi-site studies. Sites that each build their own variable names and folder logic spend months reconciling later. Harmonize before enrollment, as large multi-country studies did with shared variable names and replicated project structures [1].
No exit plan for the archive. Data without a named post-project custodian, format migration note, and repository endpoint decay into unfindable storage. State the end-of-grain disposition in writing.

A limitation of any written plan is that it cannot substitute for your institution's specific requirements. Individual projects need review by your institutional data steward, information security office, and legal or compliance team, particularly where human genomic data is covered by funder policy and consent terms.

Frequently Asked Questions

What does a bioinformatics data management plan need to contain?

It needs seven elements: a data description with volumes, a capture and collection method, a metadata and identifier standard, a storage and backup design, an access and security model, a retention and sharing schedule with named repositories, and named responsible people. If any element is missing, that is where the project will fail first.

How much storage do I need for a genomics project?

Estimate from sample count and coverage. A 30x human whole genome produces roughly 90 to 100 GB of compressed FASTQ, so a 2,000-sample cohort needs about 190 TB for raw reads, doubled for redundancy, plus roughly half again for alignments and working files. Measure your first real batch and replace estimates with observed values.

When should I use REDCap instead of spreadsheets for data capture?

Use REDCap, or any structured capture system, whenever phenotypic or clinical variables accompany sequencing, whenever multiple sites collect data, or whenever offline collection is needed. A multi-country trial used identical REDCap projects with shared variable names and offline tablet collection to manage 50 million data points, and spreadsheets cannot enforce that structure.

Should I keep raw data in FASTQ or convert to something smaller?

Keep raw reads in compressed FASTQ permanently and keep alignments as CRAM where your tools support it. CRAM stores alignments relative to a reference and is typically smaller than BAM, while raw FASTQ remains the irreplaceable source that every derived file regenerates from.

How do I handle human genomic data privacy in my plan?

Restrict raw human data to authenticated, role-based access with audit logging, tie access to consent terms, and state in the plan which outputs are public and which remain controlled. Where communities contributed samples, involve them in sharing decisions, since data sharing raises real questions about who benefits from it.

How often should I back up sequencing data and verify checksums?

Write an off-site or secondary copy continuously or per batch, and verify checksums quarterly across the archive plus after every transfer. Backup without verification is not a backup, because silent corruption is discovered only when a checksum run fails.

What are the most common mistakes teams make when implementing a bioinformatics data management plan?

The big five are skipping checksums at ingestion, writing into the immutable raw directory, leaving pipelines unversioned, treating the plan as paperwork rather than assigning an owner, and budgeting storage while forgetting staff time and transfer costs. All five are cheap to prevent and expensive to repair.

Do I need a data management plan for a small or unfunded project?

Yes, at proportionate scale. A one-page plan with a directory convention, checksum routine, backup target, and named custodian covers most small projects. NIH now requires Data Management and Sharing Plans with proposals regardless of project size, and small projects benefit most because they have the least slack to recover from data loss.