# How to Track Data Lineage in Multi-Step Bioinformatics Pipelines

By the end of this article you will be able to answer, for any file your pipeline produces, four questions with written evidence rather than memory. Which raw inputs and reference data went into it. Which tool versions and parameters touched it. Which execution produced it, on what date, on which machine. And how to reproduce or audit it months later. You will need a Linux or macOS shell, git installed, conda or Miniforge installed, and roughly 30 minutes to work through the demo. Nextflow and DVC are optional but recommended, and the walkthrough shows exactly where they fit.

Data lineage in a bioinformatics pipeline is the recorded chain of custody for every file: raw reads in, processed results out, with every transformation, tool version, parameter set, and reference database captured along the way. Without it, a multi-step pipeline becomes untraceable fast. A single bacterial genotyping workflow can chain a dozen or more tools for assembly, typing, resistance screening, and reporting, and modern species-optimized pipelines such as EcoliTyper bundle exactly this kind of multi-tool analysis into one automated run that emits TSV, JSON, and HTML outputs [1]. Each of those tools is a potential link in the lineage chain, and a break anywhere in the chain makes the final report unauditable.

One vocabulary note before we start. In genomics, the word lineage usually refers to ancestry, as in phylogenetic lineages of Mycobacterium tuberculosis classified by SNP-based tools in genomic surveillance studies [2], or lineage classification in amplicon-based resistance prediction pipelines [3]. Data lineage is a different concept. It is an infrastructure and policy concern, not a phylogenetic one. This article is about the [data management](/blog/guides/data-management-basics-principles-processes-and-best-practices) sense only.

## Why Lineage Breaks in Real Pipelines

Three failure patterns show up in almost every lab that has ever rebuilt a result from an old dataset.

First, reference databases drift underneath you. Databases are living resources, and their contents change on release cycles you do not control. UniProtKB recently went through a multi-stage reorganization across releases 2025_04 through 2026_02 in which 125.9 million entries were moved out of UniProtKB into UniParc and tens of millions more were designated for removal in release 2026_02 [4]. If your annotation step ran against release 2025_03 and a collaborator reruns it a year later against 2026_02, the protein sets differ, the counts differ, and nothing in your notebook explains why. The same mechanism applies to taxonomic databases. In one published whole-[genome analysis](/blog/guides/genome-analysis), GTDB-Tk and average nucleotide identity comparison reclassified an isolate that routine phenotypic methods had provisionally identified as a different species within the Klebsiella oxytoca complex [5]. The reclassification was correct and valuable, but it only became auditable because the researchers could point to the exact database and reference genome versions behind it.

Second, the tool versions change. A pipeline paper that compares five read-counting tools and five normalization strategies, as one targeted RNA-seq method comparison did, reaches the conclusion that featureCounts was the most rapid and robust choice among them [6]. That conclusion is only true for the versions tested. If your lineage record says "featureCounts" without a version, and the counting logic shifts across releases, your differential expression calls can shift with it.

Third, intermediate files get deleted or overwritten. On most shared compute systems, work directories are cleaned automatically. If your only record of step 3 lived inside a scratch directory, your lineage chain now has a missing link.

## The Three Layers of a Complete Lineage Record

![Pyramid showing the data layer at the base, the tool layer in the middle and the execution layer at the apex, each with what it captures.](/article-images/illustrations/data-lineage-bioinformatics-pipeline-lineage-layers.webp)

A complete data lineage record for a bioinformatics pipeline has three layers. Every practical tool you will use in this article captures one or two of them, and the step-by-step section wires all three together.

**Layer 1, the data layer.** The identity of every input and output file, captured by content hashes. SHA-256 or MD5 checksums prove that the file a step consumed is byte-for-byte the file you think it was. This layer also covers reference genomes, primer BED files, taxonomy dumps, and training data.

**Layer 2, the tool layer.** Exact software versions, captured as conda environment exports, container image digests, or dependency lock files. A package pinned by version plus a container digest is reproducible. A package pinned by "latest" is a future debugging session.

**Layer 3, the execution layer.** The run itself. Workflow engine version, start and end times, per-task exit codes, parameters, the machine or cluster profile, and the mapping between each output file and the task that produced it.

Tools like TCGA Expedition, which was built to download, extract, harmonize, version, and store all TCGA data plus metadata through a set of Bash, Python, and Java scripts, demonstrate how much engineering a large-scale genomics data storage and versioning layer absorbs when done properly for a petabyte-scale corpus [7]. Most labs do not need that scale, but they need the same principles in miniature.

## How to Track Data Lineage in Multi-Step Bioinformatics Pipelines Step by Step

The following walkthrough builds a small three-step pipeline with full lineage capture. The demo steps are deliberately simple, fastp for read trimming, seqkit for read statistics, MultiQC for aggregate reporting, because the point is the lineage scaffolding around them, not the biology. Every command works as written on Linux or macOS.

### Step 1. Put the code under git before anything else

Lineage without versioned code is incomplete, because the pipeline code itself is an input to every run. Create the project skeleton and initialize git.

```bash
mkdir -p mypipeline/{code,config,data/ref,data/raw,results}
cd mypipeline
git init
git config user.name "Your Name"
git config user.email "you@example.org"
```

If you skip this and later run git commands inside the directory, you will hit the error `fatal: not a git repository (or any of the parent directories): .git`. The fix is exactly what we just did, run `git init` in the project root before anything else.

From now on, every change to code, configuration, or metadata goes through a commit. Commits give you the immutable commit hash that will anchor your run manifests in Step 8.

### Step 2. Pin the software environment

Install the tools into a named conda environment drawn from the Bioconda and conda-forge channels, then export the exact resolved environment to a file.

```bash
conda create -n lineage-demo -c conda-forge -c bioconda fastp seqkit multiqc -y
conda env export -f environment.yml
git add environment.yml
git commit -m "Pin analysis environment with conda export"
```

The exported `environment.yml` contains the full dependency graph with exact build strings, which covers Layer 2 for a conda-based workflow. If you distribute your pipeline as a single conda package instead, as StaphScope does for Staphylococcus aureus genotyping with its integrated MLST, spa typing, SCCmec, AMR, virulence, and plasmid analyses [8], the same export principle applies. Record the package version your users should install.

If you run steps in containers, capture the digest, not just the tag. After `docker pull quay.io/biocontainers/fastp:0.23.4`, run `docker inspect` on the image and copy the sha256 value from the `RepoDigests` field of the JSON output into your configuration. Digests are immutable, tags are not.

A conda version pin that does not exist will fail with `ResolvePackageNotFound: - fastp=0.99` or a similar message naming the impossible package. The fix is to check the actual version list on the Bioconda channel page and re-pin.

### Step 3. Register the raw data with checksums

Raw data is the root of the lineage tree, and genomics data storage policy should treat it as write-once. Compute and store a checksum manifest before any processing.

```bash
sha256sum data/raw/*.fastq.gz > config/raw_checksums.sha256
cat config/raw_checksums.sha256
```

Expected output looks like this, with your own hash values.

```
7d1f4c9a2b8e...  data/raw/sampleA_R1.fastq.gz
0c9e2b8d7a6f...  data/raw/sampleA_R2.fastq.gz
5b3a1f8e2c7d...  data/raw/sampleB_R1.fastq.gz
```

Commit this manifest. From this point, `sha256sum -c config/raw_checksums.sha256` is your one-command integrity check for the entire raw data tier. If a file was truncated in transfer or modified in place, you will see the real `sha256sum` warning:

```
data/raw/sampleA_R1.fastq.gz: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read
```

The fix is to re-transfer the affected file from the sequencing provider or archive, never to regenerate the checksum to match a damaged file.

For projects where raw data is too large for git but you still want per-file identity tracked alongside code, DVC is the standard answer. DVC stores the file content outside git and writes a small `.dvc` metafile containing an MD5 checksum into git, which keeps the lineage pointer versioned.

```bash
dvc init
dvc add data/raw
git add data/raw.dvc .gitignore
git commit -m "Track raw reads with DVC, checksums in git"
```

This approach scales up to shared, versioned data registries. BioBricks.ai packages public biological datasets as modular bricks, where each brick is a DVC git repository containing an extract-transform-load pipeline, and treating data as version-controlled software is the core design decision the platform builds on [9]. You can adopt the same pattern for internal project data without any registry, because DVC works on a plain directory.

### Step 4. Record the reference data as a versioned input

Reference and database files deserve their own checksum manifest, separate from raw reads, because they change on their own schedule.

```bash
cp ecoli_K12_MG1655.fasta data/ref/
sha256sum data/ref/* > config/ref_checksums.sha256
git add config/ref_checksums.sha256
git commit -m "Record reference genome checksum"
```

Write the database release identifiers into a small YAML file that lives in git next to the checksums.

```yaml
reference:
  organism: Escherichia coli K-12 MG1655
  file: data/ref/ecoli_K12_MG1655.fasta
  source: NCBI RefSeq
  accession_note: "Record the exact assembly accession from the download page"
  downloaded: "2026-10-14"
```

Keep the note qualitative if you have not captured the accession yet. What matters is that a year from now, a reader can tell which release was in effect, because database releases reorganize content in ways that change results, as the UniProt reorganization shows [4].

### Step 5. Encode the steps in a workflow manager

Ad-hoc shell scripts executed by hand cannot record the execution layer for you. A workflow manager can, and Nextflow is the most widely used option in this space. Published pipelines in this ecosystem are built as modular Nextflow workflows that automate read processing, variant calling, annotation, and standardized reporting in DOCX and TSV formats, such as the Mycobacteria [Amplicon Sequencing](/blog/guides/amplicon-sequencing) Tool [3]. The modularity matters for lineage because each module boundary is a natural checkpoint where inputs and outputs get hashed.

Create `main.nf` with a single process to start.

```groovy
nextflow.enable.dsl = 2

params.reads = "data/raw/*_R{1,2}.fastq.gz"

process TRIM {
    tag "${meta.id}"
    publishDir "results/trimmed", mode: "copy"

    input:
    tuple path(reads), val(meta)

    output:
    tuple path("${meta.id}_R1_trimmed.fq.gz"), path("${meta.id}_R2_trimmed.fq.gz"), emit: reads
    path("${meta.id}_fastp.json"), emit: report

    script:
    """
    fastp -i ${reads} -I ${reads[10]} \
          -o ${meta.id}_R1_trimmed.fq.gz \
          -O ${meta.id}_R2_trimmed.fq.gz \
          -j ${meta.id}_fastp.json \
          -h ${meta.id}_fastp.html
    """
}
```

Run it with the lineage-relevant flags enabled from the very first run.

```bash
nextflow run main.nf -with-trace results/trace.tsv -with-report results/run_report.html
```

Expected terminal output ends with lines like these.

```
[3a/b2f1c9] Submitted process > TRIM (sampleA)
[8c/4d2e77] Submitted process > TRIM (sampleB)
Completed at 11-Nov-2026 14:02:11
Duration    : 2m 14s
```

The two-character prefixes such as `3a` are the first two segments of each task's work directory hash. Those directories under `work/` contain `.command.sh`, the exact rendered script, plus `.command.err`, `.command.out`, and the task's input and output symlinks. That directory is Layer 3 evidence for one task, generated automatically.

### Step 6. Add the downstream steps and inspect the trace file

Add a `STATS` process that consumes the trimmed reads, and a `MULTIQC` process that consumes the JSON reports. Then rerun with the trace enabled and read the trace file.

```bash
column -t -s $'\t' results/trace.tsv | head -5
```

Expected output, showing the columns Nextflow writes.

```
task_id  process  tag      status     exit  hash      duration
1        TRIM     sampleA  COMPLETED  0     3a/b2f1c9 1m02s
2        TRIM     sampleB  COMPLETED  0     8c/4d2e77 1m05s
3        STATS    sampleA  COMPLETED  0     91/f8a3b0 6s
4        STATS    sampleB  COMPLETED  0     57/c1d9e2 6s
```

The `hash` column is the lineage key. Nextflow computes it from the process name, the session identifier, the resolved inputs including file content, and the container or environment. If you change a single byte of the input FASTQ or swap the container image, the hash changes, which means the lineage record changes with it. This is why cached reruns are themselves a lineage feature. Run the same command with `-resume` and Nextflow either confirms each task from cache, proving the inputs are unchanged, or silently recomputes the task, proving something did change. Either way, the trace file documents what happened.

When a task fails, the trace records it and the terminal prints the real Nextflow message.

```
Process `TRIM (sampleA)` terminated with an error exit status (1)
```

The actual cause sits in `work/3a/b2f1c9/.command.err`. A very common entry there is `sh: line 1: fastp: command not found`, which means the container or environment for that process did not contain the tool, so the fix is in the environment declaration, not the data.

To audit past runs after the fact, use `nextflow log`, which lists every session identifier with its start time and command line, and `nextflow log -f status=ERROR` to filter failed runs.

### Step 7. Embed lineage inside the outputs themselves

Reports that travel away from the pipeline should carry their origin with them. The pattern used by species-level genotyping pipelines is to emit structured machine-readable files alongside human-readable ones. EcoliTyper compiles all results into an interactive gene-centric HTML report together with TSV, JSON, and plain text files, so that any consumer of the report can be handed the matching structured record [1]. On a 16-core system it processed 60 E. coli genomes in approximately 129 minutes, and the runtime characteristics of a pipeline are themselves metadata worth recording for capacity planning [1].

In the demo, add a post-run script that writes a run manifest. This manifest is the single file you hand to a reviewer or paste into a paper's data availability statement.

```bash
python3 <<'EOF'
import json
import subprocess

commit = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()
manifest = {
    "run_id": "2026-11-11_mypipeline_demo",
    "pipeline_commit": commit,
    "nextflow_version": "record from `nextflow -v`",
    "inputs": {
        "raw_checksums": "config/raw_checksums.sha256",
        "reference_checksums": "config/ref_checksums.sha256"
    },
    "environment": "environment.yml",
    "trace": "results/trace.tsv",
    "report": "results/run_report.html",
    "samples": ["sampleA", "sampleB"]
}
with open("results/manifest.json", "w") as fh:
    json.dump(manifest, fh, indent=2)
print("wrote results/manifest.json")
EOF
```

Expected output:

```
wrote results/manifest.json
```

If your pipeline generates per-tool version files the way nf-core pipelines do with a `versions.yml`, reference that file in the manifest too.

### Step 8. Tag the run in git

Tie the run to a code state that will never change.

```bash
git add results/manifest.json
git commit -m "Record run manifest for demo run"
git tag -a run-2026-11-11-demo -m "First full lineage-complete run"
```

Now `git rev-parse run-2026-11-11-demo` recovers the exact commit, the manifest in that commit lists the checksum files and trace, and the checksum files verify the data. That is a closed loop across all three layers.

### Step 9. Version the results, not just the inputs

For outputs you need to keep reproducible, use DVC the same way you used it for raw data.

```yaml
stages:
  trim:
    cmd: fastp -i data/raw/sampleA_R1.fastq.gz -I data/raw/sampleA_R2.fastq.gz -o results/trimmed/sampleA_R1_trimmed.fq.gz -O results/trimmed/sampleA_R2_trimmed.fq.gz -j results/trimmed/sampleA_fastp.json
    deps:
      - data/raw/sampleA_R1.fastq.gz
      - data/raw/sampleA_R2.fastq.gz
    outs:
      - results/trimmed/sampleA_R1_trimmed.fq.gz
      - results/trimmed/sampleA_R2_trimmed.fq.gz
      - results/trimmed/sampleA_fastp.json
```

Run `dvc repro trim` and `dvc status`. The `dvc dag` command prints the stage dependency graph, which is your pipeline-level lineage at a glance. DVC skips stages whose dependencies have not changed, so a `dvc status` that reports `Data and pipelines are up to date.` is positive evidence that outputs still match their recorded inputs.

### Step 10. Run the reproducibility drill

The final step is a test, run once per project milestone. Archive the project, delete the results, restore, and rebuild.

```bash
git clone <your-repo-url> rebuild-test
cd rebuild-test
dvc pull
conda env create -f environment.yml
conda activate lineage-demo
nextflow run main.nf -with-trace results/trace_rerun.tsv
sha256sum -c config/raw_checksums.sha256
diff <(cut -f4 results/trace.tsv) <(cut -f4 results/trace_rerun.tsv)
```

If every line reports `OK` and the status columns match, your lineage record is not just complete, it has been proven sufficient to reproduce the result on a clean machine. Document the drill itself in the repository README so the next person runs it too.

## Key Options and What Each Records

| Mechanism | What it captures | Where the record lives | Main gap to close |
|--|--|--|--|
| `sha256sum` manifest | Content identity of every file | `config/*.sha256` in git | Does not explain how files were made |
| `conda env export` | Full resolved software environment | `environment.yml` in git | No hardware or OS base layer |
| Container digest | Immutable tool image identity | Recorded in config or manifest | Only covers containerized steps |
| Nextflow `-with-trace` | Per-task status, exit code, content-based hash, duration | TSV in `results/` | Files under `work/` are ephemeral, copy what you need |
| Nextflow `.command.sh` | Exact rendered script per task | `work/` directory | Lost if work dir is cleaned, publish copies early |
| DVC | MD5 per data file and stage DAG | `.dvc` files and `dvc.yaml` in git | Requires discipline to `dvc add` new outputs |
| Run manifest JSON | Ties commit, environment, inputs, and trace together | `results/manifest.json`, committed | Manual to write unless scripted |
| W3C PROV or RO-Crate | Standardized machine-readable provenance graph | JSON-LD alongside results | Needs adoption effort, worth it for shared infrastructures |

## Reading and Verifying a Lineage Record

![Flowchart tracing an output back through its work directory, trace file, checksum manifests, environment file and git history.](/article-images/illustrations/data-lineage-bioinformatics-pipeline-verify-record.webp)

Answering "where did this file come from" should take under five minutes with a complete record. Work backward through the chain.

Start at the output. If it was published by a Nextflow task, find its work directory path from the log or the publish location, open `.command.sh` to see the exact command with resolved paths, and open `.command.err` and `.command.out` to see what the tool actually printed. Next, look up the task's hash in `results/trace.tsv` to get status, exit code, and duration, and confirm the run's session ID against `nextflow log`. Then walk to the inputs named in `.command.sh` and verify each one against the checksum manifests with `sha256sum -c`. Finish in the environment file to confirm tool versions, and in the git history to see whether the code or configuration changed between the run of interest and today.

For aggregate verification, MultiQC parses the per-sample JSON outputs that tools like fastp write and consolidates them into one report across all samples, which doubles as a fast sanity check that the expected number of samples flowed through every step [6]. When a pipeline's final outputs matter, verify them against something independent of the lineage record. The targeted RNA-seq method comparison mentioned earlier did exactly this, evaluating its optimized pipeline outputs with clustering and pathway analyses against full RNA-seq on the same tumors, treating agreement as the correctness criterion [6].

Lineage verification extends to analytical choices, not just file movement. In machine-learning pipelines for genomic prediction, published work shows that inflated performance estimates arise when phylogenetic structure leaks between training and test partitions, and that clade-aware data partitioning is the corrective [10]. The train-test split is therefore a lineage-critical step. Record the split itself, not only the model output.

## Bioinformatics Data Management Beyond the Run

Lineage tracking works best inside a broader bioinformatics data management setup with a few standing policies.

Separate raw data into a write-once tier. Nothing downstream ever writes into the raw directory, and every file there has a checksum manifest from the day it lands. This mirrors the architecture used at much larger scale, where TCGA Expedition obtained, versioned, and archived data with command-line access at high-performance computing facilities and security controls conforming to federal best practices [7]. Your version of that tier might be a read-only directory on a file server or an S3 bucket with object lock, but the policy is the same.

Keep intermediate data tiered by cost. Raw reads are irreplaceable and archived. Intermediates are reproducible from raw data plus the lineage record, so they can live on cheaper scratch storage and be deleted without losing lineage, provided the trace, commands, and checksums were copied out first. Published results are the smallest tier and live with their manifests in the repository.

Adopt naming conventions that carry sample identifiers, dates, and pipeline names without spaces. Every tool in the chain, from the FASTQ provider to the reporting step, passes these names through, and a rename halfway through a pipeline silently breaks the connection between trace rows and files.

For teams pulling many heterogeneous public datasets into one analysis, versioned data registries reduce the manual integration work substantially. The BioBricks.ai evaluation reported that assembling multi-dataset analytic cohorts dropped from days to minutes compared with bespoke scripts, and that automated provenance tracking strengthens FAIR compliance in such systems [9]. Even if you never use a registry, the underlying pattern, data as versioned artifacts with recorded ETL steps, transfers directly to internal projects.

## Common Mistakes and Limitations

**Citing tools and databases without versions.** "Ran with fastp and the latest UniProt" is not a lineage record. The UniProt reorganization removed and moved tens of millions of entries across releases in a matter of months [4]. Pin versions, checksums, and release identifiers for every external resource.

**Assuming a database version implies a stable result.** When GTDB-Tk and ANI-based reassignment moved an isolate from one Klebsiella species to another [5], the correct conclusion was that taxonomy assignment is a function of both the genome and the database release. Record the release, and expect assignments to be revisited after major database updates.

**No checksums on raw data.** Transfer errors, silent truncation, and well-meaning hand edits all happen. Without a baseline manifest, the first symptom is usually a downstream tool crashing with a confusing error rather than a clean integrity failure.

**Letting the work directory be the only provenance store.** Nextflow's `work/` directories are excellent automatic provenance and terrible long-term storage. Publish or copy the command scripts, logs, and trace file to a persistent location as part of the pipeline itself, not as an afterthought.

**Lineage gaps at hand-run boundaries.** The most common break in practice is between an interactive Jupyter notebook or R session and the formal pipeline. The notebook reads a file, transforms it, writes another file, and none of it appears in the trace. Either move the step into the workflow, or record the notebook file, its kernel state export, and its input and output checksums manually in the manifest.

**Trusting that a successful rerun proves correctness.** A rerun that matches proves reproducibility of the recorded procedure, nothing about whether the procedure was right. Method choices need biological validation, which is why the RNA-seq panel study evaluated its optimized pipeline against full RNA-seq outputs on the same challenging tumor samples rather than just rerunning it [6].

**Conflating biological and data lineage in documentation.** A surveillance paper reporting that lineage 4 and lineage 3 strains predominated among tuberculosis isolates, with county-level structure in western Kenya [2], is describing phylogenetic lineages. Keep those discussions out of your data management records, and keep data management records out of your phylogenetics sections. Mixed terminology confuses reviewers and sometimes your own team.

**Forgetting that resource accounting is lineage too.** Runtime and memory claims belong in the record, because they let a future user predict cost. StaphScope completed all genotyping analyses for 24 S. aureus genomes in an estimated 10 to 14 minutes on a 16-core, 16 GB RAM system [8]. That sentence in a README is provenance.

**Recording lineage you never test.** A lineage system that has never survived the Step 10 rebuild drill is a hypothesis. Run the drill before the paper deadline, not during it.

## Frequently Asked Questions

### What exactly is data lineage in a bioinformatics pipeline?

It is the recorded chain of custody connecting final outputs back through every transformation to the raw inputs, including tool versions, parameters, reference database releases, and execution metadata. A complete record lets you prove which inputs produced any file and reproduce the run.

### What is the difference between data lineage and provenance?

Provenance is the full history of an individual data object, who created it and from what, at every step. Data lineage usually refers to the end-to-end path of data through the pipeline, often visualized as a graph. In practice the terms overlap heavily and most labs track both with the same tools.

### Do I need DVC if I already use git?

Yes, if any data file exceeds a few megabytes or you want checksum-linked reproducibility for outputs. Git versions small code and metadata well but handles large FASTQ and BAM files poorly. DVC keeps the data outside git and commits a small checksum pointer file into git instead, so you get both.

### What should I record about a reference genome or database?

The file itself with a SHA-256 checksum, the source and release identifier or accession, the download date, and any filtering or preprocessing you applied. Database releases change their contents on their own schedule, so the release identifier is as important as the sequence file.

### How does Nextflow's resume feature relate to lineage?

Nextflow computes a content-based hash for each task from its inputs, code, and container. With `-resume`, unchanged tasks run from cache and changed tasks recompute, so the cache behavior itself documents what changed between runs. The trace file records the outcome for each task.

### What is the minimum viable lineage setup for a small lab?

Git for code, a conda environment export or container digest for tools, SHA-256 checksum manifests for raw data and references, a workflow manager trace file per run, and a one-page manifest JSON tying them together. All five fit in a plain repository and take under an hour to set up.

### How do I track lineage inside Jupyter or R notebooks?

Notebooks are the weakest link because they are interactive. Commit the notebook file with outputs cleared, record the input and output file checksums in the run manifest, and consider moving recurring analysis steps into the formal workflow manager where they get automatic provenance.

### How does lineage tracking help when a database update changes my results?

With a pinned release identifier and checksum, you can demonstrate that the difference comes from the database update rather than a coding error, rerun against the old release if it is archived, and state in the methods exactly which release produced which result. Without lineage, the change looks like an unexplained discrepancy.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What exactly is data lineage in a bioinformatics pipeline?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "It is the recorded chain of custody connecting final outputs back through every transformation to the raw inputs, including tool versions, parameters, reference database releases, and execution metadata. A complete record lets you prove which inputs produced any file and reproduce the run."
      }
    },
    {
      "@type": "Question",
      "name": "What is the difference between data lineage and provenance?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Provenance is the full history of an individual data object at every step. Data lineage usually refers to the end-to-end path of data through the pipeline as a graph. In practice the terms overlap and most labs track both with the same tools."
      }
    },
    {
      "@type": "Question",
      "name": "Do I need DVC if I already use git?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Yes, if any data file exceeds a few megabytes or you want checksum-linked reproducibility for outputs. Git handles small code and metadata well but not large FASTQ and BAM files. DVC keeps data outside git and commits a small checksum pointer file into git instead."
      }
    },
    {
      "@type": "Question",
      "name": "What should I record about a reference genome or database?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "The file with a SHA-256 checksum, the source and release identifier or accession, the download date, and any [preprocessing](/blog/data-analysis/what-is-data-preprocessing) you applied. Database releases change contents on their own schedule, so the release identifier matters as much as the sequence file."
      }
    },
    {
      "@type": "Question",
      "name": "How does Nextflow's resume feature relate to lineage?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Nextflow computes a content-based hash for each task from its inputs, code, and container. With resume enabled, unchanged tasks run from cache and changed tasks recompute, so cache behavior documents what changed between runs. The trace file records the outcome per task."
      }
    },
    {
      "@type": "Question",
      "name": "What is the minimum viable lineage setup for a small lab?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Git for code, a conda environment export or container digest for tools, SHA-256 checksum manifests for raw data and references, a workflow manager trace file per run, and a one-page manifest JSON tying them together. The whole setup fits in one repository."
      }
    },
    {
      "@type": "Question",
      "name": "How do I track lineage inside Jupyter or R notebooks?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Notebooks are the weakest link because they are interactive. Commit the notebook with outputs cleared, record input and output file checksums in the run manifest, and move recurring steps into the workflow manager where provenance is automatic."
      }
    },
    {
      "@type": "Question",
      "name": "How does lineage tracking help when a database update changes my results?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "With a pinned release identifier and checksum, you can show that the difference comes from the database update rather than a coding error, rerun against the archived old release if needed, and state exactly which release produced which result in your methods."
      }
    }
  ]
}
</script>

## Sources

1. [EcoliTyper: a species-optimized computational pipeline for comprehensive genotyping and surveillance of Escherichia coli.](https://pubmed.ncbi.nlm.nih.gov/42265598/)
2. [Whole-genome sequencing analysis of phylogenetic relationships and lineage diversity of Mycobacterium tuberculosis in Western Kenya.](https://pubmed.ncbi.nlm.nih.gov/42920567/)
3. [Mycobacteria Amplicon Sequencing Tool: automated resistance prediction and lineage classification for Mycobacterium tuberculosis.](https://pubmed.ncbi.nlm.nih.gov/42044525/)
4. [UniProt data pruning and the growing invisibility of the dark proteome.](https://pubmed.ncbi.nlm.nih.gov/42889534/)
5. [Whole-genome analysis of a multidrug-resistant Klebsiella michiganensis environmental isolate from an orthopedic ward in Mwanza, Tanzania reveals IncF-family plasmid replicon signatures associated with resistance determinants.](https://pubmed.ncbi.nlm.nih.gov/41957580/)
6. [Optimizing bioinformatic workflows to extract clinically usable gene expression data from targeted tumor RNA sequencing panels: comparison with total RNA-seq in cancer samples.](https://pubmed.ncbi.nlm.nih.gov/42529613/)
7. [TCGA Expedition: A Data Acquisition and Management System for TCGA Data.](https://pubmed.ncbi.nlm.nih.gov/27788220/)
8. [StaphScope: a species-optimized computational pipeline for rapid and accessible Staphylococcus aureus genotyping and surveillance.](https://pubmed.ncbi.nlm.nih.gov/41645058/)
9. [BioBricks.ai: a versioned data registry for life sciences data assets.](https://pubmed.ncbi.nlm.nih.gov/40880880/)
10. [A leakage-aware genomic prediction pipeline for meropenem resistance in Klebsiella pneumoniae using transformer-based resistome representation learning.](https://pubmed.ncbi.nlm.nih.gov/42683100/)