# .gitignore for Scientific Data: Template and Examples By the end of this article you will be able to write a `.gitignore` file that keeps raw sequencing reads, alignment files, and other large scientific data out of your Git repository, while keeping the small text files that make your project reproducible. You will also be able to debug rules that are not working, rescue files that Git already tracks, and decide file by file whether something belongs in Git, in Git Large File Storage, or in a data archive. What you need on hand: - Git installed (version 2.20 or later is comfortable, anything from the last several years works). Check with `git, version`. - A terminal. On Windows, Git Bash or Windows Terminal works. - A project directory with at least a few files in it, ideally one large data file and one small text file so you can see the difference in behavior. - A GitHub or GitLab account if you plan to push to a remote. A local repository works without one. Everything in this article runs on plain Git. No plugins, no paid tools. ## Why scientific data breaks Git Git records a snapshot of every tracked file at every commit. It is designed for text files that change line by line, such as source code, because it can store each version as a compact set of line differences. That design assumption fails for scientific data in three specific ways. First, data files are enormous. A single paired-end whole-genome sequencing run for one human sample produces FASTQ files in the range of tens to hundreds of gigabytes after compression. A BAM file holding aligned reads for that sample is the same order of magnitude. Git stores a full copy of every file blob in its object database, so a repository containing raw data grows by the size of the data on every commit that touches it. Second, data files are binary and compressed. Git's diff and merge machinery works on lines of text. A gzipped FASTQ file looks like noise to it, so Git cannot show you a meaningful diff, and a merge of two branches that both touched a data file cannot succeed. You gain nothing from versioning the file and pay the full storage cost. Third, hosting platforms enforce hard limits. According to GitHub's documentation, pushing a file larger than 100 MB is blocked, files larger than 50 MB generate a warning, and GitHub advises keeping repositories below 1 GB total. A single BAM file exceeds all of that. When researchers first put a bioinformatics project on GitHub for their own use, the push typically fails with an error like this: ``` remote: error: GH001: Large files detected. remote: error: File results/aligned.bam exceeds GitHub's file size limit of 100.00 MB. ! [remote rejected] main -> main (pre-receive hook declined) error: failed to push some refs to 'github.com:yourname/rnaseq-project.git' ``` At that point the large file is already baked into your local history, and removing it requires rewriting history, which is painful. The entire purpose of a `.gitignore` for scientific data is to prevent that situation before it happens: declare what Git should never touch, then let Git version the things it is good at, namely code, configuration, documentation, and small metadata. ## How .gitignore works A `.gitignore` file is a plain text file, one pattern per line, placed inside your repository. Git reads it and refuses to automatically track anything that matches. The Git manual (`man gitignore`, also published on the official Git website) defines the exact matching rules, which apply to every `.gitignore` file on every platform. Three layers of ignore rules exist, and they apply in this order: 1. Patterns in `.gitignore` files inside the repository. These are versioned and shared with every collaborator. This is the layer you will almost always use. 2. Patterns in `.git/info/exclude`. These apply only to your local clone and are never committed. Useful for personal scratch files you do not want to impose on collaborators. 3. A global ignore file set with `git config, global core.excludesFile`. These apply to every repository on your machine. Useful for editor swap files and operating system clutter. Within any one file, the rules follow these principles: - The last matching pattern wins. If two rules match the same file, the one further down the file decides. - A line starting with `#` is a comment. Use comments generously so future you understands the intent. - A line starting with `!` negates the previous match, re-including a file that an earlier pattern excluded. - A trailing slash on a pattern matches directories only. `results/` matches a directory named `results`, never a file named `results`. - A leading slash anchors the pattern to the directory containing the `.gitignore` file. `/config.yaml` matches only the `config.yaml` at that level, not one nested deeper. - A pattern without a slash matches at any depth. `*.log` matches `run.log`, `logs/run.log`, and `a/b/c/run.log`. One restriction matters more than any other in scientific projects: if a pattern excludes a whole directory, Git does not even descend into that directory, so a later `!` negation cannot rescue a file inside it. The fix is to ignore the directory's contents with a wildcard instead of the directory itself. This difference between `results/` and `results/*` is the single most common source of confusion in a gitignore for scientific data format questions. Both stop the directory contents from being tracked, but only `results/*` leaves the door open for a negation such as `!results/README.md`. ### Pattern syntax the Git manual defines | Pattern | Matches | Example | |--|--|--| | `*.fastq.gz` | Any file with that suffix, at any depth | `data/run1/sample.fastq.gz` | | `results/` | Any directory named `results`, at any depth, and everything inside it | `results/`, `sub/results/` | | `/results/` | Only the `results` directory at the top level of the repository | `results/` but not `sub/results/` | | `results/*` | Everything directly inside `results`, but Git still descends into `results` | allows `!results/README.md` to work | | `data/**/tmp` | `tmp` at any depth under `data` | `data/run1/tmp`, `data/a/b/tmp` | | `logs/*.log` | `.log` files one level below `logs` | `logs/run.log` but not `logs/old/run.log` | | `sample?.txt` | Any single character in place of `?` | `sample1.txt`, `sampleA.txt` | | `*.[hc]` | Character class, matches `.h` or `.c` here | C-style glob, rarely used in data work | | `!important.csv` | Re-includes `important.csv`, overriding an earlier match | must come after the excluding rule | ## A .gitignore template for scientific projects Here is a complete `.gitignore` template you can drop into the root of a bioinformatics or data-analysis repository. Every line is commented so you can delete sections that do not apply. This is the gitignore for scientific data example I start every project with, then trim. ```text # ============================================================ # Scientific data project .gitignore # Order matters. Later rules override earlier ones. # Negations (!) go at the bottom. # ============================================================ # --- Raw and derived sequence data. Never commit these. --- data/raw/ data/interim/ *.fastq *.fastq.gz *.fq.gz *.fasta *.fa *.fna *.sam *.bam *.bam.bai *.cram *.crai *.vcf *.vcf.gz *.bcf *.csi *.tbi *.bigWig *.bigBed *.2bit *.dnusrar # --- Single-cell and other large binary formats --- *.h5 *.h5ad *.zarr/ *.loom *.rds *.RData # --- Reference genomes and indices (download these, do not commit) --- references/ *.gtf *.gff *.bt2 *.bt2l *.his2.idx *.amb *.ann *.bwt *.pac *.sa # --- Pipeline work directories and logs --- .snakemake/ work/ .nextflow/ .nextflow.log* *.log *.err *.out cluster_logs/ # --- Results regenerated by the pipeline --- # Use results/* rather than results/ if you plan to # re-include specific files with ! later. results/* output/* reports/* figures/* # --- Python --- __pycache__/ *.py[cod] *.egg-info/ .ipynb_checkpoints/ .venv/ venv/ .mypy_cache/ .ruff_cache/ # --- R --- .Rhistory .Rproj.user/ *.Rproj # --- Conda and environment state --- .conda/ envs/.built/ # --- Editors and operating system noise --- .DS_Store Thumbs.db *.swp .idea/ # --- Credentials and access keys. A leak here is a real incident. --- .env *.key *token* credentials*.json config/secrets.yaml ncbi_api_key.txt # ============================================================ # Negations. Always last, so they override everything above. # ============================================================ !results/README.md !metadata/samples.csv !*.md5 ``` Two details in that template deserve explanation. The `!metadata/samples.csv` line protects your sample sheet in case a broad rule such as `*.tsv` or `*.csv` is added later. Sample sheets, checksum manifests, and README files are the small text files that give your repository its scientific value, and losing one to an over-broad wildcard is a classic mistake. The `!*.md5` line assumes you generate checksum manifests, which is the standard trick that lets a repository live without its data and still prove data integrity. You generate it with a command like this after receiving or regenerating the data: ```bash cd data/raw md5sum * > ../../metadata/raw_data_md5.txt cd ../.. ``` Commit `metadata/raw_data_md5.txt`. Any collaborator who downloads the data from its real home, whether that is a sequencing core server, SRA, or an institutional store, runs `md5sum -c metadata/raw_data_md5.txt` inside the data directory and gets `OK` for every file. The repository then documents the data without carrying it. ## What to keep in Git instead of the data The files that make an analysis reproducible are almost always small text files. Git versions them beautifully, and GitHub for researchers works best when the repository is exactly these files and nothing else. - **Sample sheet and metadata.** `samples.csv`, `metadata.tsv`, a data dictionary describing every column, and any phenotype table. - **Workflow definitions.** A `Snakefile` for Snakemake, `main.nf` plus `nextflow.config` for Nextflow, or a `run_all.sh` for plain shell pipelines. These files describe exactly how raw inputs become results, which is the core intellectual content of the project. - **Environment specifications.** `environment.yml` for conda, `requirements.txt` for pip, a `renv.lock` for R, or a Dockerfile plus its image tag. Commit the specification, never the installed environment itself. A conda environment directory is easily several gigabytes of hard-linked binaries. - **Configuration.** `config.yaml`, parameter files, genome build identifiers, and a `README.md` explaining how to run everything. - **Small code-level outputs.** Unit test fixtures, a tiny synthetic test dataset under a few megabytes so CI can run, and small QC summaries such as a `multiqc_report.html` if it stays under a few megabytes. - **Checksum manifests**, as described above. The pattern to internalize: Git holds the recipe, the archive holds the ingredients. ## File formats: commit, ignore, or route elsewhere ![Decision path asking whether a file is small text, then whether it is regenerable, ending in commit, ignore or archive.](/article-images/illustrations/gitignore-for-scientific-data-track-or-ignore.webp) | Format | Typical size for one mammalian sample | Decision | |--|--|--| | `.fastq.gz`, `.fq.gz` | tens to hundreds of GB | Ignore. Submit to SRA or ENA for publication. | | `.sam` | larger than BAM, plain text | Ignore, and compress to BAM in the pipeline. | | `.bam`, `.cram` plus indices | tens to hundreds of GB | Ignore. CRAM halves the size but still far exceeds Git limits. | | `.vcf.gz` | hundreds of MB for whole genome, MBs for targeted | Ignore in raw form. A filtered, small final callset can go to Git LFS or an archive. | | `.h5ad`, `.zarr`, `.loom` | GBs for single-cell matrices | Ignore. Deposit in a repository such as Zenodo or GEO. | | `.fasta`, `.gtf` for a reference genome | GBs uncompressed | Ignore. Provide a download script with a URL and checksum. | | Sample sheet `.csv` | kilobytes | Commit. | | `environment.yml`, `Snakefile`, `*.nf` | kilobytes | Commit. | | QC report HTML | a few MB | Commit if small, otherwise ignore and archive. | | Figure PDFs and PNGs for a manuscript | MBs | Commit final versions. Ignore intermediate drafts. | | Jupyter `.ipynb` without outputs | kilobytes to a few MB | Commit after stripping outputs. | ## Numbered walkthrough: setting up a new project correctly This walkthrough shows the exact sequence for a new project, with expected output at each step. Doing the `.gitignore` first is the whole trick, because ignore rules only prevent Git from starting to track a file. They never untrack one. **Step 1. Create the project and initialize Git.** ```bash mkdir rnaseq-project && cd rnaseq-project git init ``` Expected output: ``` Initialized empty Git repository in /home/you/rnaseq-project/.git/ ``` **Step 2. Create the directory structure and a sample data file.** ```bash mkdir -p data/raw metadata scripts results head -c 100000000 /dev/urandom > data/raw/sample1_R1.fastq.gz printf "sample_id,condition\nsample1,control\nsample2,treated\n" > metadata/samples.csv printf "# Analysis pipeline\nRun: snakemake --cores 4\n" > README.md ``` That `head` command creates a fake 100 MB file to simulate real sequencing data. On macOS, use `head -c 100000000 /dev/urandom` the same way or `mkfile -n 100m data/raw/sample1_R1.fastq.gz`. **Step 3. Add the .gitignore before anything else.** Paste the template from earlier in this article into a file named exactly `.gitignore` at the repository root. Note the leading dot. `gitignore.txt` does nothing. Verify the name: ```bash ls -la | grep gitignore ``` Expected output: ``` -rw-r--r-- 1 you staff 2.1K Nov 11 10:00 .gitignore ``` **Step 4. See what Git intends to track.** ```bash git status ``` Expected output: ``` On branch main No commits yet Untracked files: (use "git add ..." to include in what will be committed) .gitignore README.md metadata/samples.csv nothing added to commit but there are untracked files ``` Notice that `data/`, `results/`, and every wildcard-matched file type are absent from the list. Git is already honoring the rules. This is the confirmation that your gitignore for scientific data is doing its job, and it is the step worth screenshotting for your own records. **Step 5. Test a specific rule with git check-ignore.** When you are unsure whether a path is ignored and which rule is responsible: ```bash git check-ignore -v data/raw/sample1_R1.fastq.gz ``` Expected output: ``` .gitignore:7:*.fastq.gz data/raw/sample1_R1.fastq.gz ``` Read that as: line 7 of `.gitignore`, pattern `*.fastq.gz`, matched this path. If the command prints nothing and exits with status 1, the file is not ignored, which means `git add` would pick it up. **Step 6. Stage and make the first commit.** ```bash git add . git commit -m "Initial project scaffold: ignore rules, sample sheet, pipeline docs" ``` Expected output: ``` [main (root-commit) a1b2c3d] Initial project scaffold: ignore rules, sample sheet, pipeline docs 3 files changed, 12 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 metadata/samples.csv ``` Three files. No data. The 100 MB fake FASTQ never entered the object database. **Step 7. Confirm nothing large is tracked, now and forever.** ```bash git ls-files ``` Expected output: ``` .gitignore README.md metadata/samples.csv ``` Make it a habit to run this before your first push. It is the ground truth for what will leave your machine. **Step 8. Add the remote and push.** ```bash git remote add origin git@github.com:yourname/rnaseq-project.git git push -u origin main ``` Because no file exceeds even the 50 MB warning threshold, the push completes without any of the large-file errors described earlier. ## Fixing files Git already tracks If you or a collaborator ran `git add data/` before the ignore rules existed, the files are in the index and in history. Adding them to `.gitignore` changes nothing, because ignore rules apply only to untracked files. Git will keep reporting changes to them. The fix for files not yet pushed: ```bash # Remove from the index while keeping the file on disk git rm -r --cached data/ git commit -m "Stop tracking raw data" ``` Expected output: ``` rm 'data/raw/sample1_R1.fastq.gz' delete from index/ remove from working tree? no, --cached keeps the file ``` In practice the message reads as a list of `rm 'path'` lines followed by a commit summary. The `-cached` flag is the part people forget. Without it, `git rm` deletes the actual data files from your disk, which for unrecoverable sequencing data would be a genuinely bad day. Run the command, then immediately run `ls data/raw/` to confirm the files are still there before you do anything else. If the large file has already been pushed to a remote, removing it going forward is not enough, because it remains in every historical commit and every clone pays for it. The standard fixes are `git filter-repo` or the BFG Repo-Cleaner, both of which rewrite history to drop the large blobs, followed by a force push and a note to collaborators to re-clone. Coordinate that rewrite with your team, since it invalidates every existing clone. ## Git Large File Storage for files that must be versioned Some scientific files genuinely belong under version control: a final filtered VCF for a paper, a small reference panel, a locked analysis dataset a few hundred megabytes in size. Git Large File Storage (LFS) exists for this. LFS replaces the file in the Git history with a small pointer and stores the actual bytes on a separate LFS server. GitHub supports LFS, with storage and bandwidth quotas on the account tier, so check the current GitHub Docs figures for your plan before committing to it. The setup is deliberately simple: ```bash git lfs install git lfs track "*.vcf.gz" git add .gitattributes git add results/final_calls.vcf.gz git commit -m "Add final filtered callset via LFS" ``` Expected `.gitattributes` content after `git lfs track`: ``` *.vcf.gz filter=lfs diff=lfs merge=lfs -text ``` Two cautions. First, `git lfs track` must run before the file is added. A file committed as a normal blob stays a normal blob forever, and untracking it later does not shrink history. Second, `gitattributes` itself must be committed, otherwise collaborators' clones will not know which files are LFS pointers and they will end up with tiny pointer files instead of data. Anyone helping set up Git for a lab should treat `.gitattributes` as a required repo file whenever LFS is in play. LFS is not a substitute for a data archive. Reviewers and other researchers cannot easily download a single file out of an LFS-backed repository, and LFS data does not get a DOI. Deposit publication datasets in a generalist or domain archive, and use LFS only for files that need to move with the code. ## Checking and debugging your ignore rules Three commands cover essentially every debugging need. **Which rule is ignoring this file?** ```bash git check-ignore -v results/figures/plot1.pdf ``` A line showing the source file, line number, and pattern means the file is ignored. Silence means it is not. **What does Git see, including ignored files?** ```bash git status --ignored ``` The output splits into `Ignored files` and normal sections, which makes it obvious whether Git agrees with your mental model of the repository. The list of ignored files should read like your data directory. If something appears under `Untracked files` that you expected to be ignored, a pattern is wrong. **What is actually tracked right now?** ```bash git ls-files ``` This is the definitive answer to "what would I lose if a collaborator cloned this repo." If a large or sensitive file appears here, deal with it before pushing. **Dry-run an add.** Before staging anything broad, preview it: ```bash git add -n data/ ``` The `-n` flag prints what would be added without adding it. If the list includes anything over a few megabytes, stop and refine the rules. ## Common errors with their exact messages and fixes ![Flowchart branching on whether the rule is results/ or results/* and whether the negation can rescue a file.](/article-images/illustrations/gitignore-for-scientific-data-negation-branch.webp) **Error: Git refuses to add a file you actually want.** ``` The following paths are ignored by one of your .gitignore files: results/README.md hint: Use -f if you really want to add them. hint: Turn this message off by running hint: "git config advice.addIgnoredFile false" ``` Cause: a pattern matches the file. If the file should be tracked, fix the pattern, usually by changing `results/` to `results/*` and adding a negation at the bottom of the file. Using `git add -f` works once but re-introduces the same fight on every future add, so fix the pattern instead. **Error: an ignored file still shows modifications in git status.** There is no error message, which is what makes this confusing. The file was committed before the ignore rule existed. Fix with `git rm -r, cached ` as shown above, then commit. **Error: a negation rule does nothing.** ```text results/ !results/README.md ``` `results/` excludes the directory wholesale, so Git never looks inside it and never evaluates the negation. Change the first line to `results/*`. This is the documented behavior in the Git manual's NEGATED PATTERNS section, and it accounts for most "my exclamation mark rule is broken" reports. **Error: push rejected for large files** (`GH001: Large files detected`, as shown earlier). The blob is already in local history. Convert to LFS only if the file belongs under version control, otherwise rewrite history with `git filter-repo`, add the ignore rule, and force push. **Error: `fatal: pathspec 'data' did not match any files`.** You ran a command such as `git add data/` from the wrong directory, or the directory name is spelled differently. Run `pwd` and `ls` to confirm where you are and what the directory is actually called. Git commands resolve paths relative to your current directory, not the repository root. ## Common Mistakes and Limitations - **Writing the .gitignore after the first commit.** The most expensive mistake in this whole domain. Ignore rules are preventive, not corrective. In a lab onboarding document, put "create the .gitignore before git init" at the top of the Git section. - **Over-broad wildcards that catch small files you need.** `*.csv` and `*.tsv` look convenient until they swallow your sample sheet, a 40 KB peak calling summary, or the multiQC input table. Prefer directory-scoped rules such as `data/raw/*` over file-type rules, and keep negations at the bottom for the small files that must survive. - **Assuming an ignore rule hides data that was already pushed.** Removing a file in a new commit does not remove it from history. Anyone with the repository URL can check out an old commit and read the file. For credentials leaked to a public remote, rotate the key immediately, because the leak is permanent as far as any cached clone is concerned. - **Trusting .gitignore for security.** It is a convenience mechanism, not an access control. Secrets belong in environment variables or a secrets manager, referenced by code that never contains the values. - **Notebook outputs sneaking past ignore rules.** You commit the `.ipynb` and ignore nothing, but the notebook itself carries 40 MB of embedded base64 plot images from the last run. Strip outputs before committing, either with `jupyter nbconvert, clear-output, inplace` on each notebook or with the `nbstripout` tool wired into a pre-commit hook. (The `-` there is Git-free notebook flag syntax, noted so you copy the command exactly.) - **Forgetting that collaborators have their own global ignore files.** If a teammate's `core.excludesFile` ignores something you track, only their machine is affected. Debug their setup with `git config, get core.excludesFile`, and keep project-critical rules in the committed `.gitignore`, never in personal config. - **Treating the repository as the data archive.** Even with LFS, a Git hosting platform is not a preservation system. Publication data belongs in a repository that issues a persistent identifier, such as SRA for sequence reads, GEO for expression matrices, or Zenodo for anything else, with the accession written into the repository README. - **Ignoring results entirely when the paper depends on them.** The final, frozen, analysis-ready versions of key results and figures should exist somewhere durable and citable. A reasonable pattern is to ignore all intermediates, commit final figures, and archive the full result set at publication time. One general limitation deserves its own sentence: no ignore file can decide scientific judgment calls for you, and individual projects, particularly those with human genomics data under controlled-access policies, need institutional review of where data may live and who may see it. ## Frequently Asked Questions ### Should raw sequencing data ever go into a Git repository? No. Raw reads are far too large for Git's storage model and exceed platform file limits, and versioning a binary blob provides no diff or merge value. Keep raw data on institutional storage or a domain archive, reference it in the repository through a manifest of checksums and accession numbers, and let Git version the code and metadata that process it. ### Why is my ignored file still showing changes in git status? The file was tracked before the ignore rule existed, and ignore rules only affect untracked files. Remove it from the index with `git rm -r, cached path/` while keeping the files on disk, commit that change, and the rule takes effect from then on. ### Does adding a file to .gitignore remove it from repository history? No. A file committed in the past remains in every historical commit, and anyone who clones the repository can retrieve it. To truly remove it you must rewrite history with a tool such as `git filter-repo` and force push, then have collaborators re-clone. ### How do I ignore a whole results directory but still track one file inside it? Ignore the contents with `results/*` instead of excluding the directory with `results/`, then place `!results/README.md` at the bottom of the file. Negation works against `results/*` because Git still descends into the directory, whereas `results/` makes Git skip it entirely. ### What is the difference between .gitignore, .git/info/exclude, and a global ignore file? The committed `.gitignore` applies to everyone who clones the repository. `.git/info/exclude` applies only to your local clone and is never shared. The global file configured with `core.excludesFile` applies to all of your local repositories. Put project rules in the committed file and personal clutter in the other two. ### How can I tell which ignore rule is affecting a specific file? Run `git check-ignore -v path/to/file`. It prints the source file, line number, and pattern responsible, or prints nothing if the file is not ignored. Pair it with `git status, ignored` to review everything Git is excluding at once. ### Should I commit Jupyter notebooks, and what about their outputs? Commit notebooks, but strip their outputs first so embedded images and large result objects do not bloat the repository. Clear outputs with `jupyter nbconvert, clear-output, inplace notebook.ipynb` before committing, or automate it with a tool such as nbstripout in a pre-commit hook. ### If the data is not in the repository, how do collaborators and reviewers get it? Commit a manifest that lists every data file with its checksum, its source URL or archive accession, and a script that downloads it. Collaborators retrieve the data from the original source and verify integrity with `md5sum -c`, and for publications you deposit the data in an archive that issues a persistent identifier and cite that accession in the README.