# How to Organize a Bioinformatics Project Directory You will finish with a project root that separates raw data, references, metadata, scripts, temporary files, results, reports, releases, and archived material. You will know where FASTQ, FASTA, BAM, BED, VCF, GFF, HDF5, and k-mer files belong, how to checksum raw data, how to keep Git from swallowing multi-gigabyte reads, and how to hand a project to another analyst without a verbal tour. Have a Bash shell, Git, gzip, awk, and either `sha256sum` or `shasum` on hand. FastQC, MultiQC, and samtools are useful for the checks in this walkthrough. Keep the raw sequencing files, reference genome, sample sheet, and any existing analysis notes nearby. The goal is not a perfect folder taxonomy. The goal is a layout that makes each input, script, output, and decision easy to find and hard to confuse. ## Why the folder layout is part of the experiment Bioinformatics projects fail quietly when the folder layout is ad hoc. A file named `final2.bam` has no experimental meaning. A sample sheet in a personal email has no version. A trimmed FASTQ saved over the original raw FASTQ destroys the ability to rerun the analysis from the start. Reproducibility problems in bioinformatics often come from inconsistent analysis processes and a lack of standardization, which is one reason containerized environments and recorded dependencies matter [1]. A directory structure cannot fix a bad aligner, but it makes provenance visible. File formats add another failure point. Bioinformatics software relies on specialized formats, and tools often lack robust testing for correct input and output handling. Interoperability problems waste time and can lead to undetected errors in scientific results [2]. A project directory that separates raw data, reference files, working files, and results gives you a place to validate each format before it enters the next step. The layout also sets the boundary between what is disposable and what is evidence. `05_work/` can be deleted. `06_results/` should be reproducible from scripts and inputs. `01_raw_data/` should never change. `08_releases/` should identify exactly what was handed to a collaborator or attached to a manuscript. Those distinctions are more useful than deep nesting. ## A practical bioinformatics project directory structure Use one project root with ten top-level directories. Numeric prefixes keep directory listings in workflow order and prevent the layout from becoming a debate about alphabetical aesthetics. | Directory | Purpose | Track in Git? | Retention | |--|--|--|--| | `00_admin/` | README, checksums, environment files, tool versions, manifests | Yes, except large logs | Keep | | `01_raw_data/` | Immutable FASTQ, raw mass spectra, original images, received archives | No | Keep, with backup | | `02_reference/` | Genome FASTA, GFF or GTF annotation, transcript sets, indexes | Usually no, record source and checksum | Keep or re-download | | `03_metadata/` | Sample sheet, experimental design, protocols, batch map | Yes | Keep | | `04_scripts/` | Shell, Python, R, Nextflow, Snakemake, and notebook files | Yes | Keep | | `05_work/` | Trimmed reads, temporary BAM, scratch files, intermediate caches | No | Delete or archive selectively | | `06_results/` | Final alignments, count matrices, variant calls, peak tables | Small summaries only | Keep | | `07_reports/` | FastQC HTML, MultiQC report, logs, run summaries | No or small text only | Keep | | `08_releases/` | Versioned tarballs, manifests, checksums | No | Keep | | `09_archive/` | Superseded scripts, old metadata, notes, retired outputs | Small text only | Keep | This is a convention, not a law. A mass spectrometry project may replace `01_raw_data/fastq/` with `01_raw_data/mzml/`. A single-cell project may add `06_results/matrices/`. A structural biology project may add `02_reference/structures/`. Keep the top-level roles stable across projects so your hands know where to go. ### Naming rules that prevent most confusion Use lowercase ASCII letters, digits, underscores, and hyphens. Avoid spaces, parentheses, brackets, accents, and punctuation that shells interpret. A path with a space breaks naive `for` loops and command-line tools that split on whitespace. A path with a colon can break tools on some systems. Use sample IDs consistently. `S001`, `S002`, `S003` is clearer than `sample1_final`, `Sample_1_redo`, and `S-1`. Pair read files with the same stem: `S001_R1.fastq.gz` and `S001_R2.fastq.gz`. Put the reference build in the file name, for example `GRCh38.fa` or `GRCh38.gtf`. Use ISO dates such as `2026-11-10`. Use version labels such as `v1`, `v2`, and `v3` instead of `final`, `final2`, and `really_final`. Make raw data read only. This is a technical guardrail, not a personality test. If a tool or a tired analyst tries to write into `01_raw_data/`, the permission error is the desired outcome. ## Bioinformatics project directory structure step by step The commands below assume a Bash shell on Linux or macOS. Run them from the parent directory where you want the project to live. ### Step 1: Create the project root and Git repository ```bash export PROJECT="brca_rnaseq_2026" mkdir -p "$PROJECT"/{00_admin,01_raw_data,02_reference,03_metadata,04_scripts,05_work,06_results,07_reports,08_releases,09_archive} cd "$PROJECT" git init printf '05_work/\n06_results/\n08_releases/*.tar.gz\n' > .gitignore ``` Expected output from `git init`: ```text Initialized empty Git repository in /path/brca_rnaseq_2026/.git/ ``` Check the top level: ```bash ls -1 ``` Expected output: ```text 00_admin 01_raw_data 02_reference 03_metadata 04_scripts 05_work 06_results 07_reports 08_releases 09_archive ``` `mkdir -p` creates parent directories if needed and does not fail when a directory already exists. The brace expansion creates all ten directories in one command. `git init` creates the repository metadata. The `.gitignore` keeps `05_work/`, `06_results/`, and compressed release tarballs out of routine Git tracking. Common error: ```text mkdir: cannot create directory 'brca_rnaseq_2026': Permission denied ``` Fix: run the command in a directory where you have write permission, such as your home directory or a scratch project area assigned by your institution. ### Step 2: Write the README before the analysis ```bash cat > README.md <<'EOF' # brca_rnaseq_2026 ## Question Does condition X change gene expression in batch B1? ## Inputs - Raw FASTQ: 01_raw_data/fastq, read only - Sample sheet: 03_metadata/samples.tsv - Reference: 02_reference/genome/GRCh38.fa ## Run order 1. 04_scripts/01_qc.sh 2. 04_scripts/02_align.sh ## Outputs - QC report: 07_reports/multiqc/multiqc_report.html - Counts: 06_results/counts/gene_counts.tsv ## Environment - 00_admin/environment.yml - Tool versions: 00_admin/tool_versions.txt ## Release - 08_releases/brca_rnaseq_2026_results_2026-11-10.tar.gz EOF ``` The `<<'EOF'` heredoc writes the text exactly as shown and prevents the shell from expanding variables inside the block. The README must answer five questions: what the project is asking, what goes in, what order to run, what comes out, and which environment and release identify the run. ### Step 3: Intake raw data and create checksums ```bash mkdir -p 01_raw_data/fastq rsync -av /path/to/sequencer/run_2026_10_14/ 01_raw_data/fastq/ chmod -R a-w 01_raw_data find 01_raw_data -type f -name '*.fastq.gz' -exec sha256sum {} + | sort -k2 > 00_admin/raw_checksums.sha256 sha256sum -c 00_admin/raw_checksums.sha256 ``` Expected output from the checksum check: ```text 01_raw_data/fastq/S001_R1.fastq.gz: OK 01_raw_data/fastq/S001_R2.fastq.gz: OK 01_raw_data/fastq/S002_R1.fastq.gz: OK 01_raw_data/fastq/S002_R2.fastq.gz: OK ``` `rsync -av` copies in archive mode with verbose output and preserves timestamps where possible. `chmod -R a-w` removes write permission from all users, including you, which protects the raw files from accidental edits. `find -exec sha256sum {} +` passes many file names to one checksum process, which is faster than one process per file. `sort -k2` sorts by the file path field. `sha256sum -c` reads the checksum file and verifies each listed file. On macOS, use `shasum -a 256` and `shasum -a 256 -c` in place of `sha256sum`. The checksum file format is otherwise the same. Common error: ```text rsync: [sender] change_dir "/path/to/sequencer/run_2026_10_14/" failed: No such file or directory (2) ``` Fix: check the source path with `ls -ld /path/to/sequencer/run_2026_10_14`. Do not continue with an empty raw data directory. Common error: ```text sha256sum: WARNING: 1 computed checksum did NOT match ``` Fix: recopy the affected file from the source, regenerate the checksum file, and verify again. A mismatch means the file changed during transfer or storage. ### Step 4: Build and validate the sample sheet ```bash printf 'sample_id\tfastq_1\tfastq_2\tcondition\tbatch\n' > 03_metadata/samples.tsv printf 'S001\t01_raw_data/fastq/S001_R1.fastq.gz\t01_raw_data/fastq/S001_R2.fastq.gz\tcontrol\tB1\n' >> 03_metadata/samples.tsv printf 'S002\t01_raw_data/fastq/S002_R1.fastq.gz\t01_raw_data/fastq/S002_R2.fastq.gz\ttreated\tB1\n' >> 03_metadata/samples.tsv printf 'S003\t01_raw_data/fastq/S003_R1.fastq.gz\t01_raw_data/fastq/S003_R2.fastq.gz\tcontrol\tB2\n' >> 03_metadata/samples.tsv printf 'S004\t01_raw_data/fastq/S004_R1.fastq.gz\t01_raw_data/fastq/S004_R2.fastq.gz\ttreated\tB2\n' >> 03_metadata/samples.tsv ``` Check the number of fields in the header: ```bash awk -F'\t' 'NR==1{print NF}' 03_metadata/samples.tsv ``` Expected output: ```text 5 ``` Check every row for the same field count: ```bash awk -F'\t' 'NF!=5{print NR, NF}' 03_metadata/samples.tsv ``` Expected output: no output. `awk -F'\t'` sets the field separator to a tab. `NR` is the current line number. `NF` is the number of fields in the current line. A row with six fields or four fields is a metadata error that will break a pipeline later. Check that every FASTQ path exists: ```bash cut -f2,3 03_metadata/samples.tsv | tail -n +2 | tr '\t' '\n' | while read -r f do test -f "$f" || echo "MISSING $f" done ``` Expected output: no output. If a path is missing, the loop prints `MISSING` and the path. `cut -f2,3` extracts the two read columns. `tail -n +2` removes the header. `tr '\t' '\n'` puts one path per line. The `while read` loop tests each path. Tabular files used by bioinformatics tools have strict formatting requirements, and validation before analysis prevents a common bottleneck [3]. A sample sheet is not a casual spreadsheet. Treat it as a machine-readable input. Common error: ```text awk: warning: escape sequence '\t' treated as plain 't' ``` Fix: use single quotes around the awk program and write `-F'\t'`. Shell quoting changes how the backslash reaches awk. ### Step 5: Put every runnable step in a script ```bash mkdir -p 04_scripts cat > 04_scripts/01_qc.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail mkdir -p 06_results/qc 07_reports/multiqc fastqc -t 2 -o 06_results/qc 01_raw_data/fastq/*.fastq.gz multiqc 06_results/qc -o 07_reports/multiqc EOF chmod +x 04_scripts/01_qc.sh ``` `set -e` stops the script when a command fails. `set -u` stops when an unset variable is used. `set -o pipefail` makes a pipeline fail if any command in the pipe fails, not only the last one. `fastqc -t 2` uses two threads. `-o 06_results/qc` sets the output directory. `multiqc` reads the FastQC outputs and writes an aggregate report. Expected output after a successful run: ```bash ls 07_reports/multiqc ``` ```text multiqc_report.html ``` Common error: ```text bash: ./04_scripts/01_qc.sh: Permission denied ``` Fix: run `chmod +x 04_scripts/01_qc.sh`. If the file was created on Windows and has carriage returns, run `file 04_scripts/01_qc.sh` and convert with `dos2unix` if needed. Common error: ```text fastqc: command not found ``` Fix: load the environment module, activate the Conda environment, or install FastQC. The same pattern applies to `multiqc: command not found`. ### Step 6: Keep reference files and indexes together ```bash mkdir -p 02_reference/genome 02_reference/annotation rsync -av /path/to/reference/GRCh38.fa 02_reference/genome/ samtools faidx 02_reference/genome/GRCh38.fa ``` Expected output: ```text 02_reference/genome/GRCh38.fa.fai ``` `samtools faidx` creates a FASTA index beside the FASTA file. Many tools expect that location, so do not move the `.fai` file into a separate directory. If you build a BWA index or a STAR index, record the command and tool version in `00_admin/tool_versions.txt`. Indexes are derived data. They can be rebuilt, but the reference FASTA and its checksum must be preserved. Common error: ```text [E::fai_build3_core] Failed to open the file 02_reference/genome/GRCh38.fa ``` Fix: check the path with `ls -l 02_reference/genome`. A reference file is not optional, and a pipeline should stop when it is missing. ### Step 7: Separate scratch work from results ```bash mkdir -p 05_work/trimmed 05_work/tmp 06_results/alignment 06_results/counts export TMPDIR="$PWD/05_work/tmp" ``` `05_work/` is for files that can be regenerated. Put trimmed reads, temporary BAMs, and tool caches there. Set `TMPDIR` so tools that write temporary files keep them inside the project rather than in `/tmp`, where they may fill a small system partition. `06_results/` is for files you would defend in a review. Do not write final tables into `05_work/`, and do not use `06_results/` as scratch space. ### Step 8: Run with a log ```bash set -o pipefail ./04_scripts/01_qc.sh 2>&1 | tee 07_reports/01_qc_2026-11-10.log ``` `2>&1` sends standard error into standard output so both streams are captured. `tee` writes the combined output to the terminal and to the log file. With `pipefail`, a failure in the script makes the pipeline return a nonzero status even though `tee` succeeds. Expected output begins with FastQC progress lines for each FASTQ and ends with a MultiQC summary. The exact text depends on the number of files and tool version. Check the log size and contents: ```bash wc -l 07_reports/01_qc_2026-11-10.log head -n 20 07_reports/01_qc_2026-11-10.log ``` Common error: ```text tee: 07_reports/01_qc_2026-11-10.log: No such file or directory ``` Fix: create `07_reports/` first. A reproducible workflow should capture the commands, printed messages, and generated files from a run, because that record is what lets another person replay the analysis [4]. ### Step 9: Record the Git commit and tool versions ```bash git add README.md .gitignore 03_metadata 04_scripts git commit -m "Add project scaffold, sample sheet, and QC script" git rev-parse HEAD | cut -c1-7 > 00_admin/git_commit.txt cat 00_admin/git_commit.txt ``` Expected output from `cat` is a seven-character commit prefix such as `7f3a9c1`. The exact value differs for every repository. `git rev-parse HEAD` prints the full commit hash. `cut -c1-7` keeps the short form used in reports. Put the commit prefix in the final report so a reader can find the exact script version. Record tool versions: ```bash fastqc --version > 00_admin/tool_versions.txt samtools --version >> 00_admin/tool_versions.txt ``` The output begins with the tool name and version. If you use Conda, export the environment: ```bash conda env export > 00_admin/environment.yml ``` Containerized environments can make tool and dependency capture more reliable, especially when a project mixes Bioconductor, Conda, and command-line tools [1]. If you use Docker, record the image identifier in `00_admin/` as well. Common error: ```text fatal: pathspec '03_metadata/' did not match any files ``` Fix: confirm that `03_metadata/samples.tsv` exists and is not ignored by `.gitignore`. Git does not track empty directories. ### Step 10: Build a release with a manifest ```bash cat > 08_releases/MANIFEST.txt <<'EOF' Project: brca_rnaseq_2026 Release date: 2026-11-10 Reference: GRCh38 EOF git rev-parse HEAD >> 08_releases/MANIFEST.txt sha256sum 06_results/counts/gene_counts.tsv >> 08_releases/MANIFEST.txt printf '05_work\n01_raw_data\n' > 00_admin/release_exclude.txt tar -czf 08_releases/brca_rnaseq_2026_results_2026-11-10.tar.gz -X 00_admin/release_exclude.txt 03_metadata 04_scripts 06_results 07_reports 08_releases/MANIFEST.txt README.md sha256sum 08_releases/brca_rnaseq_2026_results_2026-11-10.tar.gz > 08_releases/brca_rnaseq_2026_results_2026-11-10.tar.gz.sha256 ls -lh 08_releases ``` `tar -czf` creates a gzip-compressed archive. `-X 00_admin/release_exclude.txt` reads exclude patterns from a file, which keeps raw data and scratch work out of the release. The manifest identifies the project, date, reference build, Git commit, and checksum of a key result. The final `sha256sum` gives the archive its own fingerprint. Expected output from `ls -lh` includes one tarball and one checksum file: ```text -rw-r--r-- 1 user staff 2.4M Nov 10 15:20 brca_rnaseq_2026_results_2026-11-10.tar.gz -rw-r--r-- 1 user staff 120B Nov 10 15:20 brca_rnaseq_2026_results_2026-11-10.tar.gz.sha256 ``` Archive formats that bundle a manifest and metadata make a project easier to exchange and reproduce [5]. A release tarball is not a substitute for a data repository when raw data must be shared, but it is a practical handoff for scripts, metadata, reports, and summary results. ## Bioinformatics file formats and where each one belongs A directory structure is only as good as the files it stores. The same extension can hide different content, and the same content can appear under several extensions. Validate the content, not only the name. | Format | What it holds | Correct home | Rule | |--|--|--|--| | FASTQ | Reads plus per-base quality scores | `01_raw_data/` or `05_work/` | Compress with gzip and never edit raw reads in place [6] | | FASTA | Sequences without per-base quality | `02_reference/` | One header per record and build name in the file name | | SAM, BAM, CRAM | Alignments with header and records | `05_work/` or `06_results/alignment/` | Keep the header and record the reference build [7] | | BED | Genomic intervals | `06_results/` | Validate columns and coordinate convention [2] | | VCF, BCF | Variant calls | `06_results/variants/` | Put the reference build in the path or file name | | GFF, GTF | Gene annotation | `02_reference/annotation/` | Keep the coordinate convention explicit [8] | | HDF5 | Hierarchical binary data | `06_results/` or `09_archive/` | Avoid routine Git tracking [9] | | K-mer set | Compact k-mer counts or indexes | `02_reference/` or `05_work/` | Use a specialized k-mer format when possible [10] | | MMTF | Macromolecular structure | `02_reference/structures/` | Use compact binary exchange formats for large structures [11] | BED is a good example of why format validation matters. The format is popular for genomic intervals, but software packages differ in how they handle edge cases. Formal specifications and verification tests improve interoperability, yet many tools are not tested against unusual inputs [2]. Keep a tiny known-good BED file in `03_metadata/test_files/` and use it to check a new tool before running a large job. HDF5 is a good example of why file format and version control do not always mix. HDF5 stores data and metadata in a complex binary hierarchy that is not human-readable and is not optimal for version control [9]. The Experimental Directory Structure alternative represents the hierarchy with directories and human-readable YAML metadata, which is easier to inspect and track [9]. If your project produces HDF5 files, keep them out of Git, checksum them, and document the tool that reads them. ### FASTQ vs FASTA FASTQ is the raw read format. A record has four lines: an `@` header, a sequence line, a `+` separator, and a quality line. The quality line has one character per base and encodes a Phred score. FASTQ has existed in Sanger, Solexa, and Illumina variants that are not interchangeable [6]. FASTA has a `>` header and sequence lines. It carries no per-base quality. Reference genomes, transcript sets, and protein sets are FASTA. A minimal FASTQ record looks like this: ```text @S001/1 ACGTACGT + IIIIIIII ``` A minimal FASTA record looks like this: ```text >chr1 ACGTACGT ``` Under the usual Phred+33 encoding, an `I` corresponds to a quality score of 40. The sequence is the same in both examples. The quality information is the difference. Check the first character of a gzipped file before trusting its extension: ```bash for f in 01_raw_data/fastq/*.fastq.gz do first=$(gzip -dc "$f" | head -n 1) if [[ "$first" == @* ]] then printf '%s\tFASTQ\n' "$f" elif [[ "$first" == \>* ]] then printf '%s\tFASTA\n' "$f" else printf '%s\tUNKNOWN\n' "$f" fi done ``` Expected output: ```text 01_raw_data/fastq/S001_R1.fastq.gz FASTQ 01_raw_data/fastq/S001_R2.fastq.gz FASTQ 01_raw_data/fastq/S002_R1.fastq.gz FASTQ 01_raw_data/fastq/S002_R2.fastq.gz FASTQ ``` `gzip -dc` decompresses to standard output. `head -n 1` reads the first line. The `[[ ... == @* ]]` test checks for a FASTQ header. The escaped `\>` checks for a FASTA header. The output is a simple audit table that catches a FASTA file renamed as `.fastq.gz` before it reaches an aligner. Common error: ```text gzip: 01_raw_data/fastq/S001_R1.fastq.gz: not in gzip format ``` Fix: run `file 01_raw_data/fastq/S001_R1.fastq.gz`. If it is plain text, do not rename it and hope. Either use the plain file or compress it properly with `gzip -c`. A `.gz` extension is not compression. ## Sensitive data, checksums, and backups Human genomic data needs stricter handling than a typical code repository. Keep identifiable or protected data in an access-controlled location. Set a restrictive `umask` before creating sensitive files, for example `umask 077`, so new files are not world-readable. Confirm group permissions on shared HPC storage before locking a directory down, because a blanket `chmod` can break a collaborator's access. Crypt4GH defines a file container that enables direct byte-level compatible random access to encrypted genetic data stored in SAM, BAM, CRAM, VCF, and BCF formats [12]. That means analysis tools can read encrypted data without creating a decrypted copy. If you use this approach, keep decryption keys outside the project directory and record the encryption method in `00_admin/`. Do not put keys in Git. Checksums are cheap provenance. Keep `00_admin/raw_checksums.sha256`, verify it after every transfer, and regenerate it only when the raw data legitimately changes. Keep at least one backup outside the project directory. A checksum on a corrupted file is still a checksum, so verify the backup as well. ## Common Mistakes and Limitations **Putting raw FASTQ files in Git.** Git stores every version of every byte. A few lanes of compressed FASTQ can make a repository slow and difficult to clone. Keep raw data in `01_raw_data/` and exclude it with `.gitignore`. Use a data repository or institutional storage for sharing. **Editing raw data in place.** Trimming, filtering, or renaming raw reads destroys the starting point. Write trimmed reads to `05_work/trimmed/` and leave `01_raw_data/` untouched. **Using spaces, absolute paths, or personal home directories in scripts.** A script with `/home/alice/project/...` breaks for Bob. Use paths relative to the project root and run scripts from the root. Use underscores in names. **Letting sample IDs drift.** `S1`, `sample_1`, and `Sample-1` are three different strings to a computer. Choose one ID and use it in the sample sheet, file names, and results. **Confusing FASTQ and FASTA.** A FASTQ file contains quality scores and a FASTA file does not [6]. A reference genome is FASTA. Raw reads are usually FASTQ. A tool that expects one will not silently do the right thing with the other. **Failing to validate tabular metadata.** A sample sheet with a missing tab, an extra column, or a trailing space can derail a pipeline. Validate field counts and file paths before the run [3]. **Losing tool versions.** A result without a tool version is hard to reproduce. Record versions in `00_admin/tool_versions.txt` and commit the environment file. Containerization helps, but the container identifier must be recorded too [1]. **Tracking HDF5 or other large binaries in Git.** HDF5 is not human-readable and is not optimal for version control [9]. Keep large binary outputs in `06_results/` or `09_archive/`, checksum them, and document their reader. **Using the results directory as scratch space.** A final results directory should contain files that can be explained. Temporary BAMs and intermediate caches belong in `05_work/`. **Skipping the manifest.** A tarball without a manifest is an anonymous box. Record the project name, date, reference build, Git commit, and key checksums [5]. **Over-nesting.** A path with eight levels of subdirectories is hard to type and hard to audit. Two or three levels below the project root is usually enough. **Assuming one layout fits every project.** Clinical genomics, [single-cell RNA sequencing](/knowledge/bioinformatics/single-cell-rna-sequencing-from-bulk-to-resolution), proteomics, and structural biology have different file types and retention rules. Keep the top-level roles, then adapt the subdirectories. Limitations exist. A directory structure cannot enforce good experimental design, and it cannot make a poorly documented tool reproducible. Institutional storage policies, backup schedules, and data-sharing agreements may override local preferences. The layout is a contract for your team. Write it down in the README and follow it. ## Checking the project before handoff Run a small set of checks before you send a project to a collaborator or attach it to a manuscript. ```bash for d in 00_admin 01_raw_data 02_reference 03_metadata 04_scripts 05_work 06_results 07_reports 08_releases 09_archive do test -d "$d" || echo "MISSING $d" done ``` Expected output: no output. A missing directory prints `MISSING` and its name. ```bash sha256sum -c 00_admin/raw_checksums.sha256 ``` Expected output: one `OK` line per raw file. ```bash git status -s ``` Expected output: no output if everything intended for Git is committed. Untracked files in `05_work/` and `06_results/` should not appear because they are ignored. ```bash du -sh 01_raw_data 05_work 06_results 08_releases ``` Use this to confirm that scratch space is not larger than the results, and that the release archive is present. ```bash tar -tzf 08_releases/brca_rnaseq_2026_results_2026-11-10.tar.gz | head -n 20 ``` `tar -tzf` lists the archive contents without extracting them. Check that raw data and scratch files are absent and that the README, sample sheet, scripts, reports, and manifest are present. ## Frequently Asked Questions ### Should raw data live inside the project directory? Yes, but as a read-only subdirectory that Git does not track. Keep it in `01_raw_data/`, checksum it, and back it up separately. ### How deep should the directory tree go? Two to three levels below the project root is usually enough. Deeper trees slow navigation and make scripts harder to read. ### Can I store FASTQ files in Git? No. Git is for text and small binaries, not multi-gigabyte FASTQ files. Use `01_raw_data/` and a data repository or institutional storage for sharing. ### What is the difference between FASTQ and FASTA? FASTQ stores sequence plus per-base quality scores. FASTA stores sequence without quality scores. Reference genomes are FASTA and raw reads are usually FASTQ [6]. ### Where should reference genome indices go? Beside the reference FASTA in `02_reference/genome/`. Many aligners expect the index next to the reference, so moving it can break the tool. ### Should final results be committed to Git? Small summary tables can be committed. Large BAM, VCF, matrix, or HDF5 files should go into a release archive or data repository and be checksummed [9]. ### How do I handle multiple sequencing batches? Give each batch its own subdirectory under `01_raw_data/` and record the batch in the sample sheet. Keep the batch column in every downstream results table. ### What belongs in the README? The question, inputs, run order, outputs, environment, and release identifier. A reader should be able to start from the README and know what to run next. ## Sources 1. [CREDO: a friendly Customizable, REproducible, DOcker file generator for bioinformatics applications](https://pubmed.ncbi.nlm.nih.gov/38475691/) 2. [Assessing and assuring interoperability of a genomics file format](https://pubmed.ncbi.nlm.nih.gov/35575355/) 3. [Keemei: cloud-based validation of tabular bioinformatics file formats in Google Sheets](https://pubmed.ncbi.nlm.nih.gov/27296526/) 4. [Bioinformatics recipes: creating, executing and distributing reproducible data analysis workflows](https://pubmed.ncbi.nlm.nih.gov/32640986/) 5. [COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project](https://pubmed.ncbi.nlm.nih.gov/25494900/) 6. [The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants](https://pubmed.ncbi.nlm.nih.gov/20015970/) 7. [SamQL: a structured query language and filtering tool for the SAM/BAM file format](https://pubmed.ncbi.nlm.nih.gov/34600480/) 8. [DeNoFo: a file format and toolkit for standardised, comparable de novo gene annotation](https://pubmed.ncbi.nlm.nih.gov/40236033/) 9. [Experimental Directory Structure (Exdir): An Alternative to HDF5 Without Introducing a New File Format](https://pubmed.ncbi.nlm.nih.gov/29706879/) 10. [The K-mer File Format: a standardized and compact disk representation of sets of k-mers](https://pubmed.ncbi.nlm.nih.gov/35904548/) 11. [MMTF-An efficient file format for the transmission, visualization, and analysis of macromolecular structures](https://pubmed.ncbi.nlm.nih.gov/28574982/) 12. [Crypt4GH: a file format standard enabling native access to encrypted data](https://pubmed.ncbi.nlm.nih.gov/33543751/)