# Git LFS Tutorial for Researchers: Step-by-Step Guide By the end of this Git LFS tutorial for researchers, you will be able to install Git Large File Storage, configure a repository so that multi-gigabyte FASTQ, BAM, VCF, and imaging files are stored as lightweight pointer files, push a repository to GitHub without hitting file size limits, and retrofit an existing repository that already contains large data. You will also learn the daily commands your collaborators need, the exact error messages that appear when configuration goes wrong, and how to verify that LFS is actually doing its job. To follow along you need Git (version 2.0 or newer, available from git-scm.com), a terminal, roughly 200 MB of free disk space for the practice exercises, and a GitHub account if you plan to push to a remote. Everything in this guide works the same on macOS, Windows (PowerShell or Git Bash), and Linux. ## Why Researchers Need Git LFS in the First Place Git was designed in 2005 for the Linux kernel source tree, where the largest objects are text files a few hundred kilobytes long. Every commit in plain Git stores a complete snapshot object for every changed file, and every clone downloads the full history of those objects. For a laboratory pipeline this becomes a problem quickly. A single whole-genome sequencing run produces FASTQ files measured in tens of gigabytes. A month of microscopy produces image stacks that dwarf the entire Linux kernel history. Version control for large data is a recognized requirement in modern clinical and omics infrastructure, and published approaches such as gitOmmix pair Git with a dedicated large-file extension precisely because plain Git cannot manage genomic and imaging data files on its own [1]. If you commit a 10 GB BAM file to plain Git, three things happen. Your local `.git` directory grows by roughly 10 GB per version of that file. The push to your remote stalls or fails. And every collaborator who clones the repository downloads every historical version of that BAM file forever, even if they only need the code. Git LFS (Large File Storage) exists to solve exactly this class of problem, and it has become the default answer for Git for scientists who want reproducible pipelines without duplicating terabytes of data. ## How Git LFS Works ![Diagram of a tracked large file going through the clean filter into a pointer, then the smudge filter bringing content back on checkout.](/article-images/illustrations/git-lfs-tutorial-for-researchers-pointer-lifecycle.webp) Git LFS replaces large file content in the repository with a small text pointer file. The pointer contains the file's size and a SHA-256 object identifier. The actual binary content lives in a separate LFS object store, either on your Git hosting service (GitHub, GitLab, Bitbucket all support LFS) or on a self-hosted LFS server. The mechanism has two parts: 1. **Filter drivers.** When you commit a file that matches a tracked pattern, the LFS `clean` filter replaces the content with a pointer. When Git checks the file out, the `smudge` filter fetches the real content from the LFS store and writes it to your working directory. Git LFS installs these filters by rewriting your `.git/config` and installing hooks when you run `git lfs install`. 2. **The `.gitattributes` file.** This file, which is itself committed to the repository, declares which filename patterns are handled by LFS. Anyone who clones your repository gets the tracking rules automatically, which is what makes LFS work across a whole research group. A pointer file looks like this: ``` version https://git-lfs.github.com/spec/v1 oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b5606b1b3d5b1b06f2e1f3a7 size 2147483648 ``` That 40-byte pointer is what Git stores in history. The 2 GB file it represents lives outside the object graph. A reviewer checking out an old release gets the exact same bytes through the smudge filter, which is what preserves reproducibility while keeping the repository itself small. This separation of versioning metadata from bulky content is the same design principle that clinical data-management systems build on when they link Git history to provenance graphs for omics workflows [1]. ## Installing Git LFS Install the client before you do anything else. If Git later reports that it cannot find the `lfs` subcommand, a missing or broken installation is almost always the cause. **macOS** (with Homebrew): ``` brew install git-lfs ``` **Windows**: Download the Windows installer from git-lfs.github.com, or install with winget: ``` winget install GitHub.GitLFS ``` **Ubuntu/Debian** (package cloud repositories provide current versions): ``` curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash sudo apt-get install git-lfs ``` Verify the installation: ``` git lfs version ``` Expected output: ``` git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.5) ``` Any version from 3.x onward is current enough for this tutorial. ## Step-by-Step: Your First Repository with Git LFS This walkthrough builds a small project with one real data file so you can see every moving part. ### Step 1. Initialize Git LFS on your machine Run this once per user account, not once per repository: ``` git lfs install ``` Expected output: ``` Updated git hooks. Git LFS initialized. ``` This writes `filter.lfs.clean`, `filter.lfs.smudge`, and `filter.lfs.process` entries into your global `.gitconfig` and installs the `post-checkout`, `post-commit`, `pre-push`, and `post-merge` hooks into new repositories. Existing repositories need the hooks refreshed, which happens the first time you run any `git lfs` command inside them. ### Step 2. Create the project repository ``` mkdir rnaseq-pipeline && cd rnaseq-pipeline git init ``` Expected output: ``` Initialized empty Git repository in /home/you/rnaseq-pipeline/.git/ ``` ### Step 3. Track your large-file patterns Tracking is declarative. You tell LFS which filename patterns to redirect, and it writes the rules into `.gitattributes`: ``` git lfs track "*.fastq.gz" git lfs track "*.bam" git lfs track "*.vcf.gz" git lfs track "*.cram" ``` Each command prints a confirmation: ``` Tracking "*.fastq.gz" ``` Open `.gitattributes` and you will see lines like this: ``` *.bam filter=lfs diff=lfs merge=lfs -text *.cram filter=lfs diff=lfs merge=lfs -text *.fastq.gz filter=lfs diff=lfs merge=lfs -text *.vcf.gz filter=lfs diff=lfs merge=lfs -text ``` The `filter=lfs` clause routes these files through the clean and smudge filters. The `diff=lfs` and `merge=lfs` clauses stop Git from trying to compute textual diffs or merge binary content. The `-text` suffix disables end-of-line conversion, which would otherwise corrupt binary objects. **Quote your patterns.** On Windows cmd and in most shells, unquoted wildcards get expanded by the shell before Git ever sees them, which silently tracks the wrong things. Always write `git lfs track "*.fastq.gz"` with the quotes. ### Step 4. Commit the tracking rules before adding any data The order here matters more than beginners expect. Commit `.gitattributes` before the large files arrive: ``` git add .gitattributes git commit -m "Track large sequence and alignment files with Git LFS" ``` Expected output: ``` [master (root-commit) a1b2c3d] Track large sequence and alignment files with Git LFS 1 file changed, 4 insertions(+) create mode 100644 .gitattributes ``` If you add large files first and track them afterward, those files are already stored as full Git blobs and LFS will not retroactively shrink them without a history rewrite (covered in the migration section below). ### Step 5. Add a large file and commit it For practice, generate a legitimately large file rather than hunting for real patient data: ``` head -c 1200000000 /dev/urandom > mock_run_R1.fastq.gz ls -lh mock_run_R1.fastq.gz ``` Expected output: ``` -rw-r--r-- 1 you you 1.2G Nov 11 10:02 mock_run_R1.fastq.gz ``` Now stage it and inspect what Git actually recorded: ``` git add mock_run_R1.fastq.gz git lfs ls-files ``` Expected output: ``` * 4d7a214614 * mock_run_R1.fastq.gz ``` The leading `*` means the real content is checked out in your working directory. The hexadecimal string is the first characters of the SHA-256 object identifier, not the Git blob hash. If instead you see a minus sign in that first column, the pointer is in the index but the content is not present locally. Commit as usual: ``` git commit -m "Add mock sequencing run data" ``` Inspect what actually entered history: ``` git show HEAD --stat ``` You will see the file listed, but its committed size is around 130 bytes, the size of the pointer, not 1.2 GB. Run `cat mock_run_R1.fastq.gz | head -c 120` if you are curious, and you will see the pointer text on your terminal. The real bytes live in `.git/lfs/objects/`. ### Step 6. Push to GitHub Create an empty repository on GitHub, then: ``` git remote add origin git@github.com:your-lab/rnaseq-pipeline.git git push -u origin main ``` Expected output includes an LFS upload line: ``` Uploading LFS objects: 100% (1/1), 1.2 GB | 45 MB/s, done. Enumerating objects: 5, done. ... To github.com:your-lab/rnaseq-pipeline.git * [new branch] main -> main ``` The pointer blob goes through the normal Git transfer. The 1.2 GB object goes to GitHub's LFS storage through the LFS API, which uses HTTPS on port 443. This distinction matters for firewalled computing clusters, where the Git protocol may be permitted while other transfer methods are not. ## Verifying That LFS Is Configured Correctly Three commands give you a complete health check. ``` git lfs env ``` This prints your LFS endpoint, the local storage directory, and the filter configuration. Confirm the `Endpoint` line points at your remote (for GitHub, `https://github.com/your-lab/rnaseq-pipeline.git/info/lfs`) and that `Git LFS` support shows as enabled. ``` git lfs ls-files ``` Every file that should be LFS-managed must appear here. If a 3 GB VCF is missing from this list, it was committed before tracking and exists as a full Git blob. ``` git lfs status ``` This compares pointer state between the index, the working copy, and HEAD. It flags the classic silent failure, where the index contains real file content instead of a pointer. The error you will eventually encounter in that situation, usually on push, is: ``` Encountered 1 file(s) that should have been pointers, but weren't: mock_run_R1.fastq.gz ``` The fix is renormalization, covered in the mistakes section below. ## Cloning, Pulling, and Partial Downloads When a collaborator clones an LFS-enabled repository, LFS fetches the objects automatically during checkout, provided they have run `git lfs install`. Two patterns matter for bandwidth-constrained labs. **Clone without large files, fetch selectively later:** ``` GIT_LFS_SKIP_SMUDGE=1 git clone git@github.com:your-lab/rnaseq-pipeline.git cd rnaseq-pipeline git lfs pull --include="results/rep1/*" ``` On Windows PowerShell the environment variable syntax differs: ``` $env:GIT_LFS_SKIP_SMUDGE=1 git clone git@github.com:your-lab/rnaseq-pipeline.git ``` The `-include` option accepts comma-separated path patterns, so a collaborator who only needs one sample's alignment can pull just that object. `-exclude` works the inverse way and is handy for skipping a `raw/` directory entirely. **Convert an already-cloned working copy:** ``` git lfs fetch git lfs checkout ``` `git lfs checkout` replaces pointer files with real content for everything already fetched. If a collaborator reports seeing pointer files instead of data in their working directory, this two-command pair is the fix. ## GitHub for Researchers: Limits You Should Plan Around GitHub for researchers works well with LFS, but the quotas shape your repository design. According to GitHub's published documentation, files larger than 100 MB are rejected by the plain Git endpoint, files between 50 MB and 100 MB produce a warning, and individual LFS objects are capped at 5 GB. Free accounts include 1 GiB of LFS storage and 1 GiB of monthly LFS bandwidth, with data packs available for purchase beyond that. Three practical consequences follow. First, a single LFS object must stay under 5 GB, so long-read BAM files or large image stacks may need to be split (for example, into per-chromosome CRAM files). Second, bandwidth quotas are consumed by every clone and checkout, which is an argument for the `GIT_LFS_SKIP_SMUDGE` pattern on shared cluster accounts. Third, raw sequencing data generally does not belong in Git LFS at all, because quota math gets ugly fast. Reserve LFS for processed or derived artifacts that are genuinely part of the analytical record, and keep primary data in institutional storage referenced by checksums. Large-file handling strategies for clinical omics data reach the same conclusion, treating the version-control layer as metadata and provenance rather than bulk storage [1]. ## Retrofitting an Existing Repository Many groups discover LFS after their repository is already bloated. The `git lfs migrate` command family rewrites history to move large blobs into LFS. Check the damage first: ``` git lfs migrate info --everything ``` Expected output: ``` *.fastq.gz 4.3 GB 12 files *.bam 8.7 GB 3 files ``` This reports which file extensions occupy the most history. Then rewrite: ``` git lfs migrate import --include="*.fastq.gz,*.bam" --everything ``` The `-everything` flag applies the rewrite across all branches and tags rather than just the current branch. Expect this to take minutes to hours depending on repository size, and expect every commit hash after the first touched commit to change. That rewrite has consequences: 1. Every collaborator must reclone or carefully follow the repo-rehome instructions that `git lfs migrate` prints when it finishes. 2. Open pull requests may need to be rebased. 3. Old commit hashes in papers, issue trackers, and workflow provenance records now dangle. If your group records commit hashes in analysis logs, annotate the migration itself with a note explaining the hash discontinuity. Run `git lfs migrate info, everything` again afterward to confirm the large blobs no longer appear. Then force-push the rewritten branches and tags to your remote, and confirm the repository size dropped on the hosting service's settings page. ## Key Commands Reference | Command | What it does | When to use it | |--|--|--| | `git lfs install` | Enables the clean and smudge filters for your user account | Once per machine, after installing the client | | `git lfs track "*.ext"` | Adds a pattern to `.gitattributes` | Before the matching files are first committed | | `git lfs ls-files` | Lists files currently managed by LFS | Verifying configuration after commits | | `git lfs status` | Compares pointer state across index, worktree, and HEAD | Diagnosing "should have been pointers" errors | | `git lfs pull, include="path/*"` | Downloads only selected LFS objects | Bandwidth-limited collaborators and clusters | | `GIT_LFS_SKIP_SMUDGE=1 git clone URL` | Clones without downloading any LFS content | Quick code-only clones, CI jobs | | `git lfs migrate import, include="*.ext", everything` | Rewrites history to move blobs into LFS | Retrofitting an existing repository | | `git lfs lock FILE` | Marks a binary file as exclusively checked out | Teams editing binary artifacts concurrently | | `git lfs fetch, all` | Downloads LFS objects from every ref | Archiving a repository before decommissioning | ## Git LFS Tips for Research Teams These tips come from the failure modes that recur in laboratory settings. **Treat `.gitattributes` as policy, not convenience.** Decide as a group which extensions are LFS-tracked, commit those rules early, and resist one-off `git lfs track` calls for single files. A consistent pattern list means every clone behaves identically across laptops and clusters. **Use file locking for binary artifacts that people edit.** Text files merge. A 200 MB `.pptx` figure deck or a proprietary instrument output does not. Git LFS file locking (`git lfs lock figures/figure3.pptx`) warns other team members that the file is checked out. Forgetting to lock, then pushing a clobbered binary, is one of the most common LFS-driven arguments in a lab. **Tag releases so reviewers can reconstruct results.** When a manuscript is submitted, tag the commit (`git tag v1.0-manuscript`) so the exact pointer states, and therefore the exact data versions, are recoverable. This reproducibility chain, from patient sample through analysis to published interpretation, is the explicit goal of provenance-aware Git systems in clinical genomics [1]. **Record checksums for primary data.** For files too large or too sensitive for LFS, store a SHA-256 checksum and the accession or archive location in a small committed text file. The repository then documents *which* data produced each result without hosting the data itself. This is the model used by large-data extensions to Git more broadly, including git-annex-based systems [1]. **Budget your quota before you push, not after.** One 4 GB BAM upload consumes most of a free GitHub LFS storage allotment. Compute expected object sizes (`du -sh data/`) and compare against your plan before the first push. ## Common Mistakes and Limitations **Tracking a file after committing it.** `git lfs track` only affects future commits. The tell-tale error on push is the "should have been pointers" message shown earlier. Fix it with `git add, renormalize .` followed by a commit, which re-applies the current `.gitattributes` rules to already-staged content. If the file is deep in history, use `git lfs migrate import` instead. **Committing `.gitattributes` late or never.** If `.gitattributes` is missing from the repository, collaborators clone fine but their LFS filters never engage, and their pushes embed full blobs. Verify the file exists on the default branch before onboarding anyone. **Unquoted wildcards on Windows.** `git lfs track *.fastq.gz` without quotes lets the shell expand the pattern, which on some systems tracks nothing at all or tracks the wrong filename. Always quote. **Assuming LFS files merge.** LFS files do not diff or merge as text. Two researchers editing the same tracked binary produce a conflict that Git cannot resolve semantically. Use locking, or keep such files one-writer-at-a-time. **Hitting bandwidth quota mid-analysis.** On GitHub's free tier, a lab of ten people cloning a repository with 1 GiB of LFS objects exhausts the monthly bandwidth allowance on the second full round of clones. Design around it with selective pulls and shared caches, or upgrade the plan. **Expecting LFS to be an archival system.** LFS stores the *current* set of objects your refs point to, plus whatever you have fetched. Deleting a branch can orphan its LFS objects on some hosting services. Before deleting history, run `git lfs fetch, all` and keep a complete local copy of `.git/lfs/objects/`. **Putting primary human genomic data in a third-party-hosted LFS store.** Data governance matters. Human subject data may carry consent and regulatory constraints that prohibit transfer to a commercial hosting service. Keep such data in institutional storage and reference it by checksum, which is the pattern clinical provenance systems adopt for patient-linked omics data [1]. **A general limitation.** Git LFS adds a network dependency to every checkout and can fail in ways plain Git does not, especially behind authentication proxies on HPC clusters. For small text files, plain Git remains the better tool. Use LFS surgically, for the files that genuinely need it. ## Frequently Asked Questions ### What file sizes justify using Git LFS in a research project? Use Git LFS for any file over roughly 10 MB that changes over time, and for essentially all binary analysis outputs regardless of exact size. GitHub warns about files above 50 MB and rejects files above 100 MB, so anything approaching those thresholds belongs in LFS before it ever touches a remote. ### Does Git LFS work with GitHub free accounts? Yes, Git LFS works on free GitHub accounts, which include 1 GiB of LFS storage and 1 GiB of monthly bandwidth. Individual LFS objects can be up to 5 GB. Labs that exceed the free quota buy data packs in 50 GB blocks, so plan repository content before pushing multi-gigabyte files. ### Can I convert an existing repository that already contains large files? Yes, use `git lfs migrate import, include="*.ext", everything` to rewrite history and move existing blobs into LFS. Every commit hash after the first modified commit changes, so collaborators must reclone and you should note the hash discontinuity in any workflow logs that reference old commits. ### How is Git LFS different from git-annex? Both extend Git to handle large files by storing content outside the Git object database, but they take different approaches to where content lives and how it is fetched. git-annex is widely used in large-data scientific pipelines, and the gitOmmix system for clinical omics [data management](/blog/guides/data-management-basics-principles-processes-and-best-practices) is built on git-annex for exactly this purpose [1]. Git LFS is simpler, has first-class support on GitHub, and is the better default for most lab repositories. ### Will collaborators without Git LFS installed be able to clone the repository? They can clone, but they will see small pointer files instead of data, and any push they make risks embedding full blobs into history. Every collaborator must run `git lfs install` once before their first commit. Treat that step as part of onboarding. ### How do I clone a repository without downloading the large files first? Set the environment variable before cloning: `GIT_LFS_SKIP_SMUDGE=1 git clone URL`. You get all the code and pointer files immediately, then pull specific objects later with `git lfs pull, include="path/*"` when you actually need them. ### What does a pointer file in my working directory mean, and how do I fix it? ![Decision path for a pointer file in the working directory, leading to fetch and checkout, renormalize, or checking the remote.](/article-images/illustrations/git-lfs-tutorial-for-researchers-pointer-triage.webp) A pointer file where real content should be means the smudge filter did not run or the object was never fetched. Run `git lfs fetch` followed by `git lfs checkout` to replace pointers with content. If the object is missing from the remote, the original push from the machine that created it never completed. ### Should raw sequencing data live in Git LFS? Usually no. Raw reads are large, immutable, often subject to consent or regulatory constraints, and better housed in institutional archives or domain repositories referenced by checksum inside the repository. Reserve Git LFS for processed artifacts, figures, and intermediate files that are genuinely part of the versioned analytical record. ## Sources 1. [Enhancing clinical data warehousing with provenance data to support longitudinal analyses and large file management: The gitOmmix approach for genomic and image data.](https://pubmed.ncbi.nlm.nih.gov/39952627/)