# Handling Large Binary Files in Git: Best Practices

Plain Git stores every version of every file as a complete object in its history, and binary scientific data such as BAM alignments, microscopy stacks, flow cytometry files, and model checkpoints defeat the delta compression that keeps text repositories small. The rule of thumb for choosing an approach is simple. If a file is smaller than a few megabytes and changes rarely, commit it directly. If a file is tens of megabytes or larger, or it grows with every experiment, keep the raw bytes outside the Git object database and let Git track pointers, checksums, and metadata instead.

This guide walks through each option for handling large binary files in Git, compares them on the criteria that decide real projects, gives worked command sequences, and lists the mistakes that cause the most data loss and repository bloat in scientific settings.

## Why Large Binary Files Break Plain Git

### How Git actually stores files

Git is a content-addressable object database. Every file you commit becomes a blob object identified by the SHA-1 hash of its content. Git compresses each blob with zlib and, during packing, tries to compute deltas so that a new version of a text file stores only the changed lines. That delta compression is why a repository with ten years of source code edits can clone in seconds.

Binary files break this model in two ways. First, many scientific formats are already compressed at a higher entropy level, and lossless compression has hard limits set by the information entropy of the data. Data with high entropy, such as random values or output of an earlier optimal compression pass, cannot be compressed further [1]. A gzip-compressed FASTQ, a BAM file, a zipped archive of images, or a binary model checkpoint is therefore stored nearly at full size inside the Git object database. Second, Git's delta algorithm rarely finds long matching byte runs between two versions of a compressed binary file, so a one-line change to an upstream script that regenerates a 2 GB result file produces a second 2 GB blob. Ten regenerations later, your repository has grown by 20 GB and every clone transfers all of it.

The same problem appears at the infrastructure level. Data-intensive scientific analysis already suffers when large datasets must move from storage to compute, and I/O transfer of big files is a documented performance bottleneck in scientific workloads [2]. Git cloning is just another data movement path, and pushing gigabytes of binary blobs through it makes every collaborator pay the cost on every clone, fetch, and branch checkout.

### What happens on shared hosting

Hosting platforms impose their own limits on top of Git's mechanics. GitHub warns when a single file exceeds roughly 50 MB and blocks files larger than 100 MB, values published in GitHub's own documentation. A researcher who commits a 300 MB TIFF will have the push rejected, and removing the file afterwards requires rewriting history on every branch. GitHub for researchers works well for code, small reference tables, figures in vector or lightly compressed form, and notebooks, but it is not a raw data store, and its terms of service ask users not to use repositories as a backup or content delivery network for large datasets.

### The clone tax

Every full clone downloads every blob in the history. A lab that grows from two members to twelve multiplies that cost. Graduate students on laptops, continuous integration runners, and collaborators on slow institutional links all feel it. The cost compounds because Git histories are append-only by design. You cannot shrink a repository without rewriting history, which invalidates every existing clone and requires coordination across the whole team.

## Options for Handling Large Binary Files in Git

There are five practical options for git binary files and scientific data, plus a sixth structural option of moving the data out of file form entirely. The table below is the core comparison. Each option is then explained on its own terms.

| Option | Bytes in Git history? | Typical scale | Reproducibility story | Main drawback |
|--|--|--|--|--|
| Plain Git commits | Yes, fully | Up to a few MB per file | Version and data in one object graph | Repository bloat, clone slowdowns |
| Git LFS | No, pointer files only | Tens of MB to low GB per file | Pointers pinned to exact blob IDs | Server quotas, extra setup for every clone |
| git-annex | No, symlinks plus keys | GB to TB, multi-terabyte archives | Location tracking plus per-file checksums | Conceptual overhead, more states to learn |
| DVC | No, .dvc meta files | GB to TB, pipeline-aware projects | Meta files pin hashes, stages link steps | Another tool and cache directory to manage |
| Checksum manifest plus external store | No, text manifest only | Any size | Manifest hashes verified on demand | You build and enforce the workflow yourself |
| Database or queryable format instead of files | N/A | Any size | Schema and queries versioned in Git | Requires data engineering up front |

### Option 1: Plain Git for small, stable binaries

Small binaries that never change are the cheapest case. Reference genome index summaries, a one-time lookup table, a small trained model from a finished sub-study, or a frozen figure exported as PNG can live in ordinary Git commits. Git compresses them with zlib, and because they never get new versions, no delta problem arises. The practical ceiling is a few megabytes per file and a low count of such files, because every blob you add is permanent from the perspective of every future clone.

The pros are zero extra tooling, one hosting platform, and a single object graph that ties data to code to commit messages. The cons are that the ceiling is low and easy to exceed by accident, and that a naive later commit of a regenerated 500 MB file poisons the repository permanently unless someone performs a history rewrite.

### Option 2: Git LFS

![Flowchart of LFS setup: install, track BAM and TIFF patterns, commit .gitattributes, add the large file, pull on clones, verify tracking.](/article-images/illustrations/git-binary-files-scientific-data-lfs-setup-flow.webp)

Git Large File Support (LFS) replaces large files in the repository with small pointer files that record the file's SHA-256 hash and size. The actual bytes go to an LFS storage server, which can be hosted by GitLab, a self-hosted LFS server, or commercial providers. Git hooks installed on each clone download the real files transparently on checkout.

Setup is three commands at project start:

```
git lfs install
git lfs track "*.bam"
git lfs track "*.tif"
git add .gitattributes
git commit -m "Track BAM and TIFF files with LFS"
```

The `git lfs track` command writes patterns into `.gitattributes`, the file that tells every clone which paths are LFS-managed. From then on, `git add results.bam` stores a pointer in Git and uploads the bytes to the LFS store. Collaborators run `git lfs pull` after cloning, and `git lfs ls-files` lists what is managed.

The pros are a near-invisible workflow once installed, clean integration with pull requests and branches, and pointers that keep clones fast. The cons are real for scientific teams. LFS storage and bandwidth are metered separately from repository hosting on commercial platforms, and quotas are far smaller than typical sequencing or imaging outputs. A lab generating hundreds of gigabytes per month will hit limits quickly or pay for data transfer. Self-hosting an LFS server removes the quota problem but adds an infrastructure component someone must back up and maintain. There is also a historical trap. Files committed before LFS was enabled remain full blobs in history, so LFS must be adopted at project start or paired with a history migration using `git lfs migrate import`.

### Option 3: git-annex

git-annex manages large files by storing them outside the Git object database keyed by checksum, while Git tracks a symlink and the annex metadata. Each clone is an "annex" that records which repositories hold which file content, so the system knows where every version lives even when no single machine has all of it. Files can live in bare directories on a lab server, on removable drives, on cloud object storage through special remotes, or on a colleague's laptop.

The basic workflow:

```
git annex init "lab workstation"
git annex add sequencing_run.bam
git commit -m "Add run 2024-03-01 alignments"
git annex sync
```

A collaborator retrieves the content on demand:

```
git annex get sequencing_run.bam
```

The strongest published evidence for git-annex in a research setting comes from the gitOmmix approach for clinical data warehouses, which combines Git versioning with git-annex large file handling and a PROV-O provenance knowledge graph to trace a clinical interpretation back to the patient sample through supporting data and analyses [3]. The authors designed gitOmmix specifically because [whole genome sequencing](/blog/guides/whole-genome-sequencing), RNA analyses, and complex imaging produce datasets that standard clinical data systems cannot version or link longitudinally, and they rely on git and git-annex for the versioning layer that makes that tracing possible [3].

The pros are scalability to terabytes, independence from any vendor's quota, checksum-verified content, location tracking across many machines, and a design that suits long-lived multi-site archives. The cons are a steeper learning curve than LFS, more file states to understand (for example, a file can be present, or a pointer whose content is elsewhere), and the discipline required to keep at least two copies of every annexed file so nothing depends on a single dead laptop.

### Option 4: DVC

Data Version Control (DVC) is built for machine learning and pipeline-oriented science. Like git-annex, it replaces large files with small meta files, but DVC adds the idea of stages: commands that produce outputs from inputs and parameters, tracked in the same Git repository. The meta files pin content hashes, so any commit in Git identifies an exact state of the data.

The workflow:

```
dvc init
dvc add data/results.bam
git add data/results.bam.dvc
git commit -m "Add alignment results"
dvc remote add -d s3store s3://mylab-bucket/data
dvc push
```

A collaborator restores the data state matching any historical commit:

```
git checkout v1.2-analysis
dvc pull
```

The pros are that data versions and pipeline definitions live in the same commit history, remote storage options include S3-compatible buckets and institutional servers, and reproduction of a full analysis becomes a single `dvc repro` command. The cons are that DVC assumes a pipeline mindset that some wet-lab collaborations do not need, the cache directory consumes disk space on every machine, and the tool is one more dependency that a five-year-old archive must still be able to run.

### Option 5: Checksum manifests with an external store

The simplest robust option is to keep Git purely for text and put the bytes somewhere built for them: an institutional server, an S3-compatible bucket, a designated external drive, or a disciplinary archive. Inside Git you commit a plain-text manifest that records, for every data file, a cryptographic hash, the file size, and the storage location. Cryptographic hashing lets any user later confirm that a file is unmodified by regenerating its hash and comparing it to the reference value, and reading the file in chunks rather than loading it whole makes this practical for files of any size [4].

A minimal manifest workflow on any Unix-like system:

```
sha256sum data/results.bam >> checksums.sha256
git add checksums.sha256
git commit -m "Record hash for March alignment run"
```

Verification later, on any machine that has fetched the data from the external store:

```
sha256sum -c checksums.sha256
```

The pros are maximal transparency (the manifest is human-readable and diffable in pull requests), no client-side tool for collaborators beyond standard shell utilities, and freedom to choose storage by cost and policy. The cons are that nothing is automated: every person must remember to add hashes, update the manifest, and actually upload the bytes, and the workflow depends on institutional discipline rather than software enforcement.

### Option 6: Move the data out of file form entirely

Some binary formats exist mainly because files are the default container, not because they serve the analysis. Variant Call Format (VCF) files and their binary counterpart BCF are the standard containers for gene sequence variant data in bioinformatics, and they require specialized tools and scripts for any query. Relational database systems can store and query large-scale genetic datasets with performance competitive with specialized state-of-the-art tools while adding the full query and integration capability of database management systems [5]. When the deliverable of a project is queryable variant data rather than a file to be shipped, loading it into a relational store and versioning the loading pipeline, schema, and query scripts in Git removes the huge binary from the repository while improving analysis access.

The same principle applies to image data. Storing per-image metadata in Git while the pixel data lives in an imaging archive or object store keeps the repository reviewable. The general rule is to version in Git whatever a human reads, reviews, or diffs, and to version by reference whatever only a program consumes at scale.

## Choosing Between Options: The Criteria That Decide

![Decision path asking whether a file is a few megabytes, tens of megabytes, or hundreds of gigabytes, leading to plain Git, Git LFS, or git-annex.](/article-images/illustrations/git-binary-files-scientific-data-size-triage.webp)

File size and growth rate separate the field immediately. Under a few megabytes, plain Git wins on simplicity. Tens of megabytes to a few gigabytes per file with moderate volume suits Git LFS on a hosted platform. Hundreds of gigabytes to terabytes, or any requirement to span multiple institutions and storage tiers, favors git-annex. Pipeline-driven machine learning projects favor DVC because data versions and pipeline stages share one commit history.

Team size and hosting constraints matter next. A two-person collaboration can share bytes over any channel and verify them with a manifest. GitHub for researchers works well when large files route through LFS within the platform's metered quotas, but quota math must be done before the first large upload, not after the quota error. A consortium archiving decades of clinical omics data needs vendor-independent storage with provenance, which is exactly the design space gitOmmix targets with git and git-annex [3].

Provenance depth is the criterion scientists most often underestimate. A pointer file or hash proves which bytes a commit referenced. A provenance graph proves which sample, which instrument run, which parameter set, and which intermediate analysis produced them. If reviewers, collaborators, or regulators will ever ask how a result traces back to raw data, choose an option that supports a provenance layer rather than retrofitting one, and consider the alignment of version history with a provenance knowledge graph demonstrated in the gitOmmix work [3].

Long-term integrity and archival requirements come last but bite hardest. Ten years from now, someone must be able to confirm that the archived bytes still match what was published. Hash verification against a recorded reference is the basic mechanism [4]. For repositories that must be provably immutable, there are approaches that anchor Git repository state to external trust infrastructure, such as systems that combine Git with blockchain timestamping to create verifiable integrity and authenticity checkpoints for archived repositories, demonstrated on public repositories containing reports and data about COVID-19 spread [6].

## Worked Scenarios

### A small sequencing lab

A lab of four runs one Illumina flowcell per month. FASTQ files are 15 GB each and never change after generation. Analysis outputs, a per-sample count matrix of 2 MB and a QC report of 200 KB, do change as methods improve. The right split is to keep FASTQ files entirely out of Git on a lab server with a checksum manifest committed to Git, keep the small count matrices and reports in plain Git because they are small and meaningfully versioned, and keep the pipeline code in the same repository. Nobody needs LFS here, and nobody should ever commit a FASTQ.

### A multi-site clinical imaging study

Three hospitals contribute imaging studies linked to longitudinal patient records. Files are hundreds of megabytes each, data cannot leave institutional control, and the clinical question requires tracing interpretations back through supporting data to the sample. This is the gitOmmix shape exactly: git-annex for large file versioning, Git for the metadata, and a provenance knowledge graph aligned with the version history so longitudinal and source-tracing queries are possible [3]. A hosted LFS quota is irrelevant because the bytes never leave institutional storage in the first place.

### A machine learning subproject inside a larger grant

A postdoc trains models on features derived from a shared dataset. Checkpoints are 800 MB each, and a dozen training runs happen per week. The shared dataset is read-only for everyone. The right tools are DVC for the checkpoints and pipeline stages, with the read-only dataset distributed once through an external store and referenced by hash. Committing checkpoints to plain Git would add gigabytes per week to the history, and LFS would work but would hide the pipeline structure that DVC records natively.

### A methods paper with supplementary data

A paper needs one 40 MB processed result file and a 5 MB figure archive attached to the repository for reviewers and readers. GitHub blocks nothing at these sizes, and the files will never change. Plain Git commits are acceptable, though pushing both files to a disciplinary archive with a DOI and recording hashes in the repository is better for citation and preservation. Hash verification tools that compare generated hashes against a known reference value exist precisely for confirming that a distributed file was received unmodified [4].

## Checksums, Compression, and What Actually Helps

Two technical points decide many small choices, so they deserve their own treatment.

Checksums belong in every workflow regardless of which option you pick. A SHA-256 hash recorded in a committed manifest converts any later dispute about file integrity into a one-command check. Tools that verify integrity by comparing a freshly computed hash against a reference handle files of any size by reading them in chunks, so verification cost scales linearly and stays practical for multi-gigabyte scientific files [4]. Record the hash at the moment the file is finalized, before the file leaves the machine that produced it.

Compression only helps on data with low entropy. Lossless compression exploits redundancy, and its ceiling is set by the information content of the message, so previously compressed or high-entropy data cannot shrink meaningfully no matter which algorithm is applied [1]. For scientific teams this has a concrete implication. Committing a `.tar.gz` of a results directory to Git buys almost nothing because the gzip layer already removed the compressible redundancy, and every subsequent regeneration of that archive produces a nearly unrelated blob. If compression is going to matter, it happens before the file reaches Git, in the choice of format (for example, CRAM over uncompressed SAM for alignments), not in Git's own zlib layer.

For repositories that must serve as citable, tamper-evident archives, integrity protection can be layered on top of Git itself. Systems that integrate Git with blockchain timestamping create verifiable synchronization points, so a repository can be proven to be in a specific state at a specific time, and closed storage units can make an archived repository effectively immutable for sharing [6]. This is overkill for a [lab notebook](/knowledge/molecular-biology/lab-notebook) repository, but it is a real pattern for regulatory or publication-critical archives.

## Common Mistakes and Limitations

Committing the large file first and adopting tooling second. The single most expensive mistake is pushing a multi-gigabyte file to a shared remote and then deciding to adopt LFS, git-annex, or a manifest workflow. The blob is already in history. Fixing it requires a history rewrite with a tool such as git filter-repo or the BFG Repo-Cleaner, followed by force-pushing every branch and requiring every collaborator to re-clone. Adopt the large-file policy before the first commit, not after the first quota error.

Forgetting `.gitattributes` on LFS projects. If a collaborator clones before pulling the branch that contains `.gitattributes`, their client may commit large files as ordinary blobs, silently defeating the whole setup. Verify with `git lfs ls-files` after the first few commits from a new machine.

Treating Git as a backup system. Git versioning is not a backup. Annexed files can exist as pointers on your machine while their content sits on one retired laptop. Every option in this article requires at least two independent copies of the raw bytes on separate storage systems, with the copies themselves verified by checksum on a schedule.

Regenerating result files casually. A binary produced by a pipeline should be versioned once per meaningful state, not on every script save. If the pipeline is deterministic, the commit that pins the pipeline version and the hash of the output is worth more than five intermediate binary snapshots.

Assuming the pointer proves the data. A pointer file or manifest entry records an intention. The bytes must exist somewhere, be reachable, and verify. A repository full of pointers to a departed postdoc's personal drive is not a dataset.

Individual cases vary by institution, funding agency, and data-sharing agreement, so the storage and access-control decision for human-subject data needs review by your local data governance process before any bytes are placed on any platform.

## Frequently Asked Questions

### What file size is too large for plain Git?

Files in the tens of megabytes start to slow clones noticeably, and GitHub blocks any file above 100 MB with a warning starting at 50 MB. Treat those as hard platform boundaries and treat a few megabytes as the comfortable design ceiling for ordinary commits.

### Should I use Git LFS or git-annex for sequencing data?

Use LFS if your files are in the tens of megabytes to low gigabytes range and you want a workflow that feels like ordinary Git on a hosted platform with quota headroom. Use git-annex for terabyte-scale collections, multi-site storage, or any project where the bytes must stay under institutional control, which is the pattern validated by the gitOmmix approach for genomic and imaging data [3].

### Can I just compress my data files before committing?

Usually no, because most large scientific formats are already compressed, and lossless compression cannot reduce data whose information entropy is already high [1]. Committing a gzipped archive mostly guarantees that each new version becomes a fresh full-size blob in history.

### How do I remove a large binary file I already committed?

You must rewrite history with a tool such as git filter-repo or the BFG Repo-Cleaner to strip the blob from every commit, then force-push all branches and have collaborators re-clone. Removal is never a simple forward commit, because Git keeps every historical object until a rewrite deletes it.

### Does GitHub work for researchers with large datasets?

GitHub for researchers works well for code, small derived tables, figures, notebooks, and LFS-managed files within metered quotas, but raw multi-gigabyte datasets belong in an external store referenced from the repository by hash and location. Plan the quota before the first large upload rather than discovering the limit mid-project.

### How do I prove a dataset has not been altered after publication?

Record a SHA-256 hash for each file in a versioned manifest at the time of finalization, and verify on demand with a chunked hashing tool that regenerates and compares hashes [4]. For archives that require provable immutability and timestamping, Git-based systems that anchor repository state to a blockchain provide verifiable integrity checkpoints over time.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What file size is too large for plain Git?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Files in the tens of megabytes start to slow clones noticeably, and GitHub blocks any file above 100 MB with a warning starting at 50 MB. Treat those as hard platform boundaries and treat a few megabytes as the comfortable design ceiling for ordinary commits."
      }
    },
    {
      "@type": "Question",
      "name": "Should I use Git LFS or git-annex for sequencing data?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Use Git LFS if your files are in the tens of megabytes to low gigabytes range and you want a workflow that feels like ordinary Git on a hosted platform with quota headroom. Use git-annex for terabyte-scale collections, multi-site storage, or projects where the bytes must stay under institutional control."
      }
    },
    {
      "@type": "Question",
      "name": "Can I just compress my data files before committing?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Usually no, because most large scientific formats are already compressed, and lossless compression cannot reduce data whose information entropy is already high. Committing a gzipped archive mostly guarantees that each new version becomes a fresh full-size blob in history."
      }
    },
    {
      "@type": "Question",
      "name": "How do I remove a large binary file I already committed?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "You must rewrite history with a tool such as git filter-repo or the BFG Repo-Cleaner to strip the blob from every commit, then force-push all branches and have collaborators re-clone. Removal is never a simple forward commit, because Git keeps every historical object until a rewrite deletes it."
      }
    },
    {
      "@type": "Question",
      "name": "Does GitHub work for researchers with large datasets?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "GitHub works well for researchers for code, small derived tables, figures, notebooks, and LFS-managed files within metered quotas, but raw multi-gigabyte datasets belong in an external store referenced from the repository by hash and location."
      }
    },
    {
      "@type": "Question",
      "name": "How do I prove a dataset has not been altered after publication?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Record a SHA-256 hash for each file in a versioned manifest at the time of finalization, and verify on demand with a chunked hashing tool that regenerates and compares hashes. For archives that require provable immutability and timestamping, Git-based systems that anchor repository state to a blockchain provide verifiable integrity checkpoints over time."
      }
    }
  ]
}
</script>

## Sources

1. [LOSSLESS BINARY DATA COMPRESSION METHODS FOR INCREASING THE PRODUCTIVITY OF SOFTWARE SYSTEMS](https://doi.org/10.32782/it/2025-1-22)
2. [ODDS: Optimizing Data-Locality Access for Scientific Data Analysis](https://doi.org/10.1109/TCC.2017.2754484)
3. [Enhancing clinical data warehousing with provenance data to support longitudinal analyses and large file management: The gitOmmix approach for genomic and image data.](https://pubmed.ncbi.nlm.nih.gov/39952627/)
4. [File Hash Integrity Checker for Digital Forensics and Data Verification](https://doi.org/10.46610/joits.2026.v12i03.003)
5. [Relational Data Models for Genetic VCF data](https://doi.org/10.14778/3749646.3749674)
6. [Combining Git and Blockchain for Trusted Information Sharing](https://doi.org/10.1109/ACCESS.2024.3418749)