# How to Extract Specific Columns from BED and GFF Files Using awk By the end of this guide you will be able to extract specific columns from BED and GFF files using awk, reorder fields into any layout, filter rows during extraction, compute interval lengths, parse the GFF attribute column, convert GFF coordinates to BED coordinates, and verify every result against a known answer. All commands run on a stock Linux install, in the macOS Terminal, or on Windows through WSL (Windows Subsystem for Linux, a compatibility layer that runs a Linux environment on Windows). You need awk (GNU awk, mawk, and the BSD awk shipped with macOS all work), the standard utilities cat, cut, diff, sort, and wc, and gzip for compressed files. No root access, no package installs, no Python environment. Column extraction sounds trivial until a BED file shows up with 12 optional columns, a GFF attribute field packs dozens of key-value pairs into one string, and one format counts bases from zero while the other counts from one. Those details are where silent errors come from. The walkthrough below uses two tiny practice files so you can check every command against a known answer, and the same patterns scale to genome-size files without modification. ## Why awk Is a Core Linux Command for Bioinformatics Most genomics data that moves between tools is tab-delimited text (plain text where columns are separated by tab characters): coordinate intervals, gene annotations, variant calls, alignment summaries. Linux for bioinformatics therefore starts with text-stream tools, and awk is the workhorse among them. awk (named for its three authors, Aho, Weinberger, and Kernighan, who built it in 1977) reads input one line at a time, splits each line into fields, and runs a small program against every line. Memory use stays flat no matter how large the file is, so a multi-gigabyte annotation file streams through in minutes on an ordinary laptop, while a spreadsheet application will either refuse the file or truncate it. Compared with cut, the other common extraction tool, awk does three things cut cannot do. It reorders columns (cut always prints fields in the order they appear in the input), it filters rows with arbitrary logic, and it computes new values such as interval lengths in the same pass. That combination of extraction, filtering, and arithmetic in a single read of the file is why awk is among the first Linux commands for bioinformatics that anyone learns, and why it survives in pipelines long after fancier tools arrive. awk is also a POSIX standard (POSIX is the family of Unix portability standards), so the same one-liner runs on GNU/Linux servers, macOS workstations, and inside Docker containers, with small dialect differences covered later in this guide. ## The Anatomy of BED and GFF Files BED (Browser Extensible Data) comes from the [UCSC Genome Browser](/knowledge/bioinformatics/ucsc-genome-browser-beginner-tutorial-and-tips) and is the standard format for coordinate intervals: ChIP peaks, alignments, exons, regions of interest. A BED file is tab-delimited with three required columns and up to nine optional ones. GFF (General Feature Format) is the standard format for annotated features on a sequence. The current version, GFF3, is specified by the Sequence Ontology project. GTF (Gene Transfer Format) is the variant that Ensembl and GENCODE use to distribute gene annotations. Both are always exactly nine tab-delimited columns, and both store coordinates as one-based and closed, meaning both the start base and the end base are included in the feature. The table maps every physical column to its awk field number in both formats. | awk field | BED column | GFF3 and GTF column | |--|--|--| | $1 | chrom (chromosome name) | seqid (sequence identifier) | | $2 | chromStart (0-based) | source (annotation provenance) | | $3 | chromEnd (exclusive) | type (feature type, for example gene) | | $4 | name | start (1-based) | | $5 | score (0 to 1000) | end | | $6 | strand | score (float or .) | | $7 | thickStart | strand (+, -, or .) | | $8 | thickEnd | phase (0, 1, 2 for CDS, otherwise .) | | $9 | itemRgb | attributes (key-value string) | | $10 | blockCount | not present | | $11 | blockSizes | not present | | $12 | blockStarts | not present | The coordinate systems differ, and this single fact causes most off-by-one errors. BED chromStart is zero-based (counting starts at 0) and chromEnd is exclusive (the feature stops one base before chromEnd), so the interval length is chromEnd minus chromStart. GFF start is one-based and end is inclusive, so the length is end minus start plus 1. The same gene covering the first 2,500 bases of a chromosome is stored as BED 10000 to 12500 and as GFF 10001 to 12500. Both describe 2,500 bases: 12500 - 10000 equals 2500, and 12500 - 10001 + 1 equals 2500. Note also what surrounds the data. BED files may contain track and browser display lines for the UCSC browser and comment lines starting with #. GFF3 files begin with the pragma ##gff-version 3 (a pragma is a metadata line, not data), may contain # comment lines, and may embed the reference sequence itself after a ##FASTA line. None of those lines are data rows, and every extraction command below guards against them. ## awk Fundamentals You Will Use in Every Command An awk program is a list of rules of the form pattern { action }. awk reads each input line (a record), splits it into fields named $1 through $NF, where NF is the number of fields, and evaluates each rule in order. If the pattern is true, or missing, the action runs. $0 is the whole line. BEGIN rules run before any input is read, END rules run after everything is read. The command-line options that matter for column work: - -F'\t' sets the field separator to a single tab. This preserves empty fields, which the default whitespace splitting does not. - -v OFS='\t' sets the output field separator (OFS), inserted between values when print receives them as a comma-separated list. - -v name=value passes a shell variable into the script. Patterns can compare strings ($3 == "gene"), compare numbers ($5 >= 500), match regular expressions ($9 ~ /ID=/, negated with !~), or combine conditions with && and ||. awk reads from files named on the command line or from standard input, so it drops into pipes naturally. NR is the record number across all input, FNR restarts at 1 for each file, and the test NR == FNR is true only while the first file is being read. One example previews the whole guide: ```bash awk -F'\t' '$6 == "+" {print $4}' genes.bed ``` This prints the name field of every plus-strand feature. Everything else in this guide is a variation on that shape. ## Step-by-Step: Extract Specific Columns from BED and GFF Files with awk Work through these steps in order. Each command is complete, and each output block shows exactly what you should see. ### Step 1: Create the practice files printf interprets \t as a tab and \n as a newline, so these commands produce real tab-delimited files no matter how your editor treats pasted text. Run them in an empty directory. ```bash printf 'chr1\t10000\t12500\tgeneA\t620\t+\n' > genes.bed printf 'chr1\t58000\t60250\tgeneB\t310\t-\n' >> genes.bed printf 'chr2\t9500\t11000\tgeneC\t880\t+\n' >> genes.bed printf 'chrX\t150200\t151000\tgeneD\t450\t-\n' >> genes.bed printf '##gff-version 3\n' > genes.gff3 printf 'chr1\tref\tgene\t10001\t12500\t.\t+\t.\tID=gene:geneA;Name=geneA\n' >> genes.gff3 printf 'chr1\tref\tmRNA\t10001\t12500\t.\t+\t.\tID=transcript:geneA.1;Parent=gene:geneA\n' >> genes.gff3 printf 'chr1\tref\tCDS\t10200\t10400\t.\t+\t0\tID=CDS:geneA.1;Parent=transcript:geneA.1\n' >> genes.gff3 printf 'chr1\tref\tgene\t58001\t60250\t.\t-\t.\tID=gene:geneB;Name=geneB\n' >> genes.gff3 printf 'chr2\tref\tgene\t9501\t11000\t.\t-\t.\tID=gene:geneC;Name=geneC\n' >> genes.gff3 printf 'chrX\tref\tgene\t150201\t151000\t.\t+\t.\tID=gene:geneD;Name=geneD\n' >> genes.gff3 printf 'geneA\t12.4\n' > annot.tsv printf 'geneB\t3.1\n' >> annot.tsv printf 'geneC\t0.0\n' >> annot.tsv printf 'geneD\t7.7\n' >> annot.tsv ``` The BED and GFF files describe the same four features, which makes the coordinate-conversion steps checkable. Confirm the tabs are real tabs with cat -A, which displays tabs as ^I and line ends as $: ```bash head -1 genes.bed | cat -A ``` ```text chr1^I10000^I12500^IgeneA^I620^I+$ ``` Six fields, five tabs, no stray spaces. If you see spaces instead of ^I, the file was created with spaces and every downstream split will misbehave. ### Step 2: Inspect the files before extracting Never extract columns blind. Check the field count of every line first: ```bash awk -F'\t' '{print NF}' genes.bed | sort | uniq -c ``` ```text 4 6 ``` Read that as four lines with six fields each. Do the same for the GFF file: ```bash awk -F'\t' '{print NF}' genes.gff3 | sort | uniq -c ``` ```text 1 1 6 9 ``` One line with a single field (the ##gff-version 3 pragma) and six data rows with nine fields. On a real file, unexpected counts here tell you the file is ragged before you extract anything, which is far cheaper than discovering it downstream. ### Step 3: Extract columns from a BED file The basic extraction, chromosome, start, end: ```bash awk -F'\t' 'BEGIN{OFS="\t"} {print $1, $2, $3}' genes.bed ``` ```text chr1 10000 12500 chr1 58000 60250 chr2 9500 11000 chrX 150200 151000 ``` Three pieces do the work. -F'\t' splits each line on single tabs, so empty fields survive. BEGIN{OFS="\t"} sets the output separator to a tab. The commas between $1, $2, and $3 insert that separator between values. Leave out the BEGIN block and the output comes back space-separated, which looks fine on screen and then breaks the next tab-aware tool in the pipeline. Reordering is the same command with the fields listed in a different order, here name first, then strand, then coordinates: ```bash awk -F'\t' 'BEGIN{OFS="\t"} {print $4, $6, $1, $2, $3}' genes.bed ``` ```text geneA + chr1 10000 12500 geneB - chr1 58000 60250 geneC + chr2 9500 11000 geneD - chrX 150200 151000 ``` Adding a computed column needs no extra pass. Interval length in BED is end minus start: ```bash awk -F'\t' 'BEGIN{OFS="\t"} {print $1, $2, $3, $4, $3 - $2}' genes.bed ``` ```text chr1 10000 12500 geneA 2500 chr1 58000 60250 geneB 2250 chr2 9500 11000 geneC 1500 chrX 150200 151000 geneD 800 ``` awk converts numeric strings to numbers automatically when you use them in arithmetic, so $3 - $2 just works. ### Step 4: Filter rows while extracting Put a pattern before the action and only matching rows are printed. High-scoring plus-strand features, with the name leading: ```bash awk -F'\t' '$5 >= 500 && $6 == "+" {print $4, $1, $2, $3, $5}' genes.bed ``` ```text geneA chr1 10000 12500 620 geneC chr2 9500 11000 880 ``` geneB and geneD drop out, one for its score of 310 and one for its minus strand. The same pattern syntax handles strings, for example $4 ~ /^gene[AB]/ selects two names by regular expression in one condition. ### Step 5: Extract columns from a GFF file First, see what happens without a guard: ```bash awk -F'\t' '{print $1, $4, $5}' genes.gff3 ``` ```text ##gff-version 3 chr1 10001 12500 chr1 10001 12500 chr1 10200 10400 chr1 58001 60250 chr2 9501 11000 chrX 150201 151000 ``` The pragma line leaks into the output with empty second and third columns. Add the guard $1 !~ /^#/ (the first field does not start with #) and select gene rows only: ```bash awk -F'\t' '$1 !~ /^#/ && $3 == "gene" {print $1, $4, $5, $7}' genes.gff3 ``` ```text chr1 10001 12500 + chr1 58001 60250 - chr2 9501 11000 - chrX 150201 151000 + ``` $7 is the strand. String comparison is case sensitive, so "gene" does not match "Gene" or "mRNA". To take every feature type except a few, negate with !=, for example $3 != "gene" && $3 != "mRNA". ### Step 6: Parse the attribute column Column 9 is one string, so it has no sub-fields until you create them. To pull the ID value from a GFF3 file, match the pattern ID= followed by every character up to the next semicolon, then slice with substr: ```bash awk -F'\t' '$3 == "mRNA" { match($9, /ID=[^;]+/) print $1, $4, $5, substr($9, RSTART + 3, RLENGTH - 3) }' genes.gff3 ``` ```text chr1 10001 12500 transcript:geneA.1 ``` match sets two variables: RSTART, the character position where the match begins, and RLENGTH, its length. The prefix ID= is three characters, so the value starts at RSTART + 3 and runs for RLENGTH - 3 characters. This works identically in gawk, mawk, and BSD awk. Real annotation files pack several attributes per row. The strict way to pull one is to split the string on semicolons, walk the pieces, and split each piece on the equals sign: ```bash awk -F'\t' '$3 == "mRNA" { n = split($9, attrs, ";") i = 1 while (i <= n) { if (attrs[i] ~ /^Parent=/) { split(attrs[i], kv, "=") print $1, $4, $5, kv[2] } i++ } }' genes.gff3 ``` ```text chr1 10001 12500 gene:geneA ``` split fills the array attrs with the semicolon-separated pieces and returns the count. Anchoring the key test with ^Parent= matters: in larger files a loose search for Parent could also match a key like ParentID, while the anchor makes the test exact. GTF stores attributes differently, with quoted values. Create a realistic sample line: ```bash printf 'chr1\tensembl_havana\tgene\t11869\t14409\t.\t+\t.\tgene_id "ENSG00000223972"; gene_name "DDX11L1"\n' > sample.gtf ``` Ensembl and GENCODE put gene_id first by convention, which makes a quick trick available. Split the attribute string on the double-quote character and take the second piece: ```bash awk -F'\t' '$3 == "gene" { split($9, a, "\"") print $1, $4, $5, a[2] }' sample.gtf ``` ```text chr1 11869 14409 ENSG00000223972 ``` The position-independent version, for files where gene_id is not first, matches the whole quoted expression and slices off the nine-character prefix (gene_id, a space, and an opening quote) plus the trailing quote: ```bash awk -F'\t' '$3 == "gene" { match($9, /gene_id "[^"]+"/) print substr($9, RSTART + 9, RLENGTH - 10) }' sample.gtf ``` ```text ENSG00000223972 ``` ### Step 7: Convert GFF to BED ![Worked example where GFF start 10001 and end 12500 become BED start 10000 and end 12500, both giving a length of 2500 bases.](/article-images/illustrations/awk-extract-columns-bed-gff-gff-to-bed-calc.webp) The conversion formula is one subtraction: BED start = GFF start - 1, and BED end = GFF end unchanged. Combine it with attribute parsing to turn gene rows into a usable BED6 file (BED with six columns): ```bash awk -F'\t' 'BEGIN{OFS="\t"} $1 !~ /^#/ && $3 == "gene" { match($9, /ID=[^;]+/) id = substr($9, RSTART + 3, RLENGTH - 3) sub(/^gene:/, "", id) print $1, $4 - 1, $5, id, ".", $7 }' genes.gff3 > genes_from_gff.bed cat genes_from_gff.bed ``` ```text chr1 10000 12500 geneA . + chr1 58000 60250 geneB . - chr2 9500 11000 geneC . - chrX 150200 151000 geneD . + ``` sub rewrites id in place, stripping the gene: prefix that many annotation pipelines attach to ID values. The score column is filled with . because BED scores are integers from 0 to 1000 and the GFF score was empty. Now prove the conversion is right. The original BED file and the converted file describe the same four features, so their shared columns must match exactly: ```bash diff <(cut -f1-4,6 genes.bed) <(cut -f1-4,6 genes_from_gff.bed) ``` diff prints nothing when the files are identical. Any output is an off-by-one or a parsing bug, and you want to find it here rather than downstream. Process substitution with <( ) requires bash or zsh. ### Step 8: Convert BED to GFF The reverse formula adds one to the start and leaves the end alone. Fill source, type, and phase with placeholder values: ```bash awk -F'\t' 'BEGIN{OFS="\t"} { print $1, "mybed", "feature", $2 + 1, $3, $5, $6, ".", "ID=" $4 }' genes.bed ``` ```text chr1 mybed feature 10001 12500 620 + . ID=geneA chr1 mybed feature 58001 60250 310 - . ID=geneB chr2 mybed feature 9501 11000 880 + . ID=geneC chrX mybed feature 150201 151000 450 - . ID=geneD ``` Note the deliberate concatenation in the last field: "ID=" $4 with no comma glues the prefix onto the value. The same operator, used by accident inside a print list, is the most common way people mash columns together, covered in the mistakes section below. ### Step 9: Join a second file onto the extraction Lookup tables are the other half of column work. The file annot.tsv maps gene names to expression values. Read it first, stash it in an array, then annotate the BED rows: ```bash awk -F'\t' 'NR == FNR { map[$1] = $2 next } { print $1, $2, $3, $4, map[$4] }' annot.tsv genes.bed ``` ```text chr1 10000 12500 geneA 12.4 chr1 58000 60250 geneB 3.1 chr2 9500 11000 geneC 0.0 chrX 150200 151000 geneD 7.7 ``` NR == FNR is true only while awk reads the first file, so the first rule builds the lookup array (an associative array, indexed by strings instead of numbers) and next jumps to the next input line. Every line of the second file falls through to the second rule. A name missing from the table yields an empty field, not an error. To mark misses explicitly, replace map[$4] with the conditional expression ($4 in map ? map[$4] : "."). ### Step 10: Verify the result Four checks catch almost everything. Line count against expectation: ```bash wc -l genes_from_gff.bed awk 'END{print NR}' genes_from_gff.bed ``` Both print 4. The two methods disagree only in edge cases such as a final line without a trailing newline, and a disagreement means a malformed tail. Field-count distribution, so ragged lines cannot hide: ```bash awk -F'\t' '{print NF}' genes_from_gff.bed | sort | uniq -c ``` Coordinate invariants. Starts must be non-negative and ends must exceed starts: ```bash awk -F'\t' '$2 < 0 || $3 <= $2 {print "bad line", NR}' genes_from_gff.bed ``` No output means the check passed. For files from unknown sources, add a numeric check: ```bash awk -F'\t' '$2 !~ /^[0-9]+$/ || $3 !~ /^[0-9]+$/ {print NR, $2, $3}' genes_from_gff.bed ``` Finally, spot-check one record against a source you trust: grep geneA in both the input and the output and compare the coordinates by eye. Automated checks plus one manual anchor is the right level of care for files that feed downstream analysis. ## awk Syntax Reference for Column Extraction Keep this table within reach. Every construct is POSIX awk and runs on gawk, mawk, and BSD awk. | Task | awk syntax | Notes | |--|--|--| | Set the input field separator | -F'\t' or BEGIN{FS="\t"} | Tab splitting preserves empty fields | | Set the output field separator | -v OFS='\t' | Rebuild the record with $1=$1 to apply it to existing data | | Print selected fields | print $1, $3, $9 | Commas insert OFS between values | | Whole line | $0 | Use for passthrough with filters | | Last field | $NF | Adapts to variable column counts | | Number of fields | NF | NF == 0 catches blank lines | | Record numbers | NR, FNR | NR == FNR is true only while reading the first file | | Conditional extraction | $3 == "gene" {print $1} | String comparison is case sensitive | | Regex on a field | $9 ~ /ID=/ | Negate with !~ | | Extract a substring | match($9, /re/) then substr($9, RSTART, RLENGTH) | Portable across awk implementations | | Split a string | split($9, a, ";") | a, a hold the pieces, return value is the count | | Replace text | sub(/re/, "x", var) or gsub(/re/, "x", var) | sub replaces the first match, gsub all matches | | Stop reading early | exit | Handy after the ##FASTA line | | Formatted output | printf "%s\t%d\n", $1, $3 | printf adds no newline unless the format asks for one | ## Practical awk Tips for BED and GFF Files These are the habits that separate one-off commands from reliable extraction code. **Set LC_ALL=C for large files.** LC_ALL=C awk -F'\t' '...' big.gff3 makes awk work bytewise instead of through locale rules, which is faster on multi-gigabyte files and avoids surprises with character ranges in regular expressions. Use the same setting for sort. **Parameterize with -v.** awk -v min=500 -F'\t' '$5 >= min {print $4}' genes.bed keeps thresholds on the command line, so the same script serves every project without edits. **Read compressed files through a pipe.** zcat genes.bed.gz | awk -F'\t' 'BEGIN{OFS="\t"} {print $1, $2, $3}'. On macOS, if zcat complains, use gzip -dc instead. The same pipe reads bgzip-compressed files (the blocked gzip format used with tabix), because bgzip output is gzip compatible. **Convert space-separated files to tabs with the rebuild trick.** awk -v OFS='\t' '{$1=$1} 1' spaced.bed > genes.bed. Assigning to $1 forces awk to rebuild the record with the current OFS, and the pattern 1 (always true) prints every line. **Use NF as a line filter.** awk -F'\t' 'NF == 9 {print $1, $4, $5}' genes.gff3 keeps only well-formed nine-column rows, which drops blank lines and most pragma lines in one move. Combine it with the # guard for files that contain tabbed comments. **Stop at embedded FASTA.** awk -F'\t' '/^##FASTA/ {exit} $1 !~ /^#/ {print $1, $4, $5}' genes.gff3. exit stops reading immediately, which saves real time on annotation files that ship with the genome sequence attached. **Stay portable.** match, substr, split, sub, gsub, and index behave identically in gawk, mawk, and BSD awk. gensub is a GNU awk extension and fails on the other two with an undefined-function error. Minimal Debian containers, common in bioinformatics images, often ship mawk rather than gawk, so portable constructs survive Docker unchanged. **Save reusable extractions as script files.** Write the rules into a file (comments start with #) and run awk -f extract_genes.awk genes.gff3. Version the file with your project and test it against a small fixture like the practice files in this guide. ```awk # extract_genes.awk: gene rows to BED BEGIN { OFS = "\t" } $1 !~ /^#/ && $3 == "gene" { match($9, /ID=[^;]+/) print $1, $4 - 1, $5, substr($9, RSTART + 3, RLENGTH - 3) } ``` **Format region strings with printf.** awk -F'\t' '{printf "%s:%d-%d\n", $1, $2, $3}' genes.bed prints chr1:10000-12500 style regions for primer orders, genome browser sessions, or batch job arrays. printf writes no newline unless the format asks for one, hence the \n. **Sort BED output properly.** awk ... | LC_ALL=C sort -k1,1 -k2,2n > sorted.bed. The -k1,1 sorts chromosome names lexicographically and -k2,2n sorts starts numerically, which is what interval tools expect as input. ## Common Mistakes and Limitations These are the errors that actually corrupt results, with their symptoms and fixes. 1. **Space-separated output.** You ran awk -F'\t' '{print $1, $2, $3}' genes.bed without setting OFS. The default output separator is a single space, so the result looks readable on screen and then fails the next tab-aware tool. Fix: BEGIN{OFS="\t"} or -v OFS='\t'. 2. **Mashed columns from a missing comma.** print $1$2 concatenates strings with no separator, producing values like chr110000. print $1, $2 inserts OFS. The confusion bites because "ID=" $4 is intentional concatenation while $4$5 is a typo, and both are legal awk. 3. **Whitespace splitting instead of tab splitting.** The default field separator treats each run of spaces and tabs as one separator and discards empty fields. Demonstrate on a line with an empty name field: ```bash printf 'chr1\t1000\t2000\t\t500\t+\n' > tricky.bed awk '{print NF, $4}' tricky.bed awk -F'\t' '{print NF, $4}' tricky.bed ``` The first command prints 5 and hands you the score (500) as if it were the name. The second prints 6 with an empty fourth field, which is the truth. Always use -F'\t' for these formats. 4. **Off-by-one from mixing coordinate systems.** BED is zero-based and half-open, GFF is one-based and closed. Pasting GFF columns next to BED columns without converting shifts every interval by one base at the start. The formulas: BED start = GFF start - 1, BED end = GFF end. The length checks must agree for the same feature: $3 - $2 in BED versus $5 - $4 + 1 in GFF. For geneA, 12500 - 10000 = 2500 and 12500 - 10001 + 1 = 2500. 5. **Headers, pragmas, and embedded FASTA.** Symptoms are junk rows in the output or a line count one or more too high. Guards: $1 !~ /^#/ for GFF3 pragmas and comments, $1 !~ /^(track|browser|#)/ for BED, and /^##FASTA/ {exit} to stop before embedded sequence. 6. **Carriage returns from Windows files.** A file saved on Windows ends lines with \r\n, and the \r stays attached to the last field. Comparisons such as $9 == "ID=gene:geneA" then never match, and output values carry an invisible character. Detect with cat -A (lines end in ^M$) or the file command. Fix inside awk with a first rule {sub(/\r$/, "")} or run dos2unix on the file once. 7. **Assuming a fixed BED column count.** BED ranges from 3 to 12 standard columns, and real files mix. $6 is the strand only in BED6 and above. In a BED3 file $6 does not exist, and awk returns an empty string with no error. Check the distribution with awk -F'\t' '{print NF}' file.bed | sort | uniq -c before trusting any field number. 8. **Naive attribute parsing.** Splitting $9 on "=" breaks when a value contains "=" or when you wanted the second attribute. GTF values are quoted and may contain spaces. GFF3 values may be percent-encoded (a space stored as %20, a comma as %2C), so a literal match on a human-readable value can fail. Anchor key tests with ^, split on semicolons first, and decode percent escapes only when needed. 9. **Double-quoted awk programs.** The shell expands $1 inside double quotes before awk ever runs. awk "print $1" genes.bed either prints the whole file unchanged (when the shell variable is empty, awk receives the valid program "print") or produces a syntax error. Representative messages: mawk: line 1: syntax error at or near print, the gawk message unexpected newline or end of string with a caret pointing at the break, or the BSD awk message awk: syntax error at source line 1 followed by a context snippet and a bailing-out line. Fix: single-quote the program and pass values in with -v. 10. **Treating awk as a validator.** awk never warns when you read a nonexistent column, when start is greater than end, or when IDs do not resolve. Empty output almost always means the pattern never matched, not that the file is empty, so head the file and check NF before debugging anything else. For interval algebra (intersect, merge, closest, coverage), use a tool built for it such as bedtools. Also remember that array-based lookups grow with the number of unique keys, so collecting millions of unique IDs from a whole-genome file holds real memory. A missing file produces its own clear error rather than silent corruption: gawk prints a fatal message of the form cannot open file `genes.bde' for reading (No such file or directory), and BSD awk prints awk: can't open file genes.bde. ## Where to Go Next ![Decision path asking if you need filtering, reordering or arithmetic, then if you need interval overlap work, ending at awk, cut or bedtools.](/article-images/illustrations/awk-extract-columns-bed-gff-awk-or-cut-or-bedtools.webp) Once extraction is solid, three directions pay off. First, sort your BED output with LC_ALL=C sort -k1,1 -k2,2n, because interval tools expect sorted input. Second, move interval questions to bedtools: intersect for overlaps, merge for collapsing adjacent intervals, getfasta for pulling sequences using a BED of coordinates. awk prepares the columns, bedtools does the genome arithmetic. Third, keep the awk scripts. A small library of tested .awk files plus tiny fixture files like the ones in this guide turns column extraction from an improvisation into a reusable step, and starting pipeline scripts with set -euo pipefail ensures a failed decompression step cannot silently feed an empty stream into awk. ## Frequently Asked Questions ### How do I extract the first three columns of a BED file with awk? Run awk -F'\t' 'BEGIN{OFS="\t"} {print $1, $2, $3}' genes.bed. The -F'\t' option splits each line on tabs, and the print statement outputs the chromosome, start, and end columns separated by tabs. ### How do I skip header or pragma lines while extracting columns? Add a pattern guard before the action. For GFF3 use awk -F'\t' '$1 !~ /^#/ {print $1, $4, $5}' genes.gff3. For BED, also exclude track and browser lines with the guard $1 !~ /^(track|browser|#)/. ### Why does the GFF attribute column not split the way I expect? Column 9 is a single string, not separate fields. Split it on the semicolon separators and then on the equals sign, or use match($9, /ID=[^;]+/) with substr to pull one value. GTF files use quoted values such as gene_id "ENSG00000223972", so split on the double quote character or match the full quoted pattern. ### How do I convert GFF coordinates to BED coordinates in awk? Subtract one from the GFF start and keep the end unchanged: awk -F'\t' '$3 == "gene" {print $1, $4 - 1, $5}' genes.gff3. GFF is one-based and closed while BED is zero-based and half-open, so GFF positions 10001 to 12500 become BED 10000 to 12500. ### How do I extract columns from a gzipped BED or GFF file? Stream the file through a decompressor: zcat genes.bed.gz | awk -F'\t' '{print $1, $2, $3}'. If zcat complains on macOS, use gzip -dc instead. The same pipe works on bgzip-compressed files because they are gzip compatible. ### What is the difference between NR and FNR in awk? NR counts every record across all input files, while FNR restarts at one for each file. The test NR == FNR is true only while awk reads the first file, which is the standard way to load a lookup table before processing the second file. ### How do I keep empty fields when extracting columns? Set both the input and output separators to a tab and rebuild the record: awk -F'\t' -v OFS='\t' '{$1=$1} 1' file.bed. The default separator treats runs of spaces and tabs as one delimiter, which makes empty fields disappear. ### When should I use awk instead of cut or bedtools? Use cut for a plain extraction, awk when you also need to filter rows, reorder columns, or compute values in the same pass, and bedtools for interval operations such as intersect and merge. cut always prints fields in input order, so reordering requires awk.