# Google Cloud Streaming Genomics Pipeline: Template and Examples By the end of this guide you will be able to build and run a streaming genomics pipeline on Google Cloud that pulls sequencing reads directly from a public archive, processes them in parallel with Apache Spark, and writes results to cloud storage. You will stream FASTQ reads, run a quality-control pass, run a k-mer profiling pass, validate the outputs with arithmetic you can check by hand, and debug the failures that actually happen on a first run. Everything is organized around a reusable template you can adapt to RNA-seq quality control, metagenomic profiling, or any read-level analysis. You need a Google Cloud project with billing enabled, the `gcloud` command-line tool installed and authenticated, Python 3 on the machine where you edit code, and basic familiarity with FASTQ files and the Spark model. A web browser is enough for the notebook-style pilot described in the LSTrAP-Cloud design. No specialized bioinformatics server is required, which is the entire point of cloud bioinformatics for labs with modest local infrastructure. ## What a Streaming Genomics Pipeline Actually Is There is no official Google Cloud product called a "streaming genomics pipeline." The phrase describes a design pattern that two published pipelines have already validated: sequencing reads are read over the network and processed record by record, instead of being downloaded in full to local disk first. In this pattern, the pipeline format at the input boundary is almost always FASTQ. FASTQ is a four-line text record holding an identifier, the [nucleotide sequence](/knowledge/molecular-biology/nucleotide-sequence), a separator line, and an ASCII-encoded quality string. Reads are usually served gzip-compressed by the archive, so the pipeline opens a network stream, wraps it in a gzip decoder, and parses records as they arrive. Nothing waits for a full file to land on disk. This differs from batch pipelines in three practical ways. First, you never pay for or manage a disk large enough to hold the raw reads. Second, a rerun re-fetches the data over the network rather than reading a cached copy. Third, any network interruption is a compute failure, so retry logic belongs inside the processing code, not in a separate download step. "Streaming" here means streaming ingest of reads into parallel workers. It does not mean Spark Structured Streaming or continuous event processing. Keeping that distinction straight prevents a lot of confusion when you read the Spark documentation. ## Two Proven Designs From the Literature The template in this guide combines ideas from two peer-reviewed pipelines, one built on Google infrastructure and one on AWS. Reading how each solved the core problems tells you what your template needs. ### LSTrAP-Cloud: Streaming RNA-seq Quality Control From ENA LSTrAP-Cloud (Large-Scale Transcriptomic Analysis Pipeline in Cloud) is a user-friendly pipeline built on Google Colaboratory, Google's hosted notebook environment. It performs quality control on [RNA sequencing](/blog/guides/rna-sequencing) data streamed from the European Nucleotide Archive (ENA), a public repository of sequencing reads, and then builds a gene coexpression network, a graph where nodes are genes and edges connect genes with correlated expression across samples [1]. The biological payoff is gene function prediction. A biologist picks one gene of interest as bait, and the network returns a shortlist of functionally related genes. The authors demonstrated this with the nicotine biosynthesis pathway in Nicotiana tabacum (tobacco), using the pipeline to identify the enzymes, transporters, and transcription factors involved in the synthesis, transport, and regulation of nicotine [1]. Because the pipeline works for any organism with a sequenced genome and publicly available RNA-seq data, it is a template rather than a single-purpose tool. The design lesson for your pipeline: if the analysis fits in one notebook session and one dataset at a time, you do not need a cluster. Stream the reads, run quality control, and keep the whole workflow in a browser. ### Flint: Streaming Metagenomic Profiling With Spark Flint attacks a harder scaling problem. Metagenomic profiling means identifying which bacteria are present in a mixed sample by aligning sequencing reads against a collection of reference genomes. Most profiling tools are built for small reference collections and do not scale to large ones, even though larger collections give a more complete and accurate picture of the bacterial population [2]. Flint is built on the Apache Spark framework and uses Spark's built-in parallelism and streaming engine architecture to map reads against a large reference collection: 170 GB covering 43,552 bacterial genomes from Ensembl. It runs on Amazon's Elastic MapReduce service, and it profiles 1 million Illumina paired-end reads against that collection on 64 machines in 67 seconds, an order of magnitude faster than the state of the art while using a much larger reference [2]. Streaming the reads into the cluster is what makes this speed possible, because mapping work starts before any full file has been staged. The design lesson for your pipeline: when the reference or the sample count outgrows a notebook, move the same streaming pattern onto a managed Spark cluster. Google Cloud's managed Spark service is Dataproc, and the Spark code is portable across providers. ## Prerequisites Before you touch the template, confirm the following: 1. A Google Cloud project with billing enabled. 2. The `gcloud` CLI installed and authenticated (`gcloud auth login`). 3. Python 3.9 or later for local editing and testing. 4. A browser for the Colaboratory-style pilot if you want to reproduce the LSTrAP-Cloud flow first. 5. An ENA study or run accession for publicly available reads. The European Nucleotide Archive assigns these identifiers, and they appear in every published dataset that deposited reads there. One networking fact worth knowing before you start: Dataproc clusters reach external hosts such as ENA by default when the nodes have external IP addresses. Clusters configured for internal-only networking need Cloud NAT configured, or every worker's outbound connection to the archive will time out. ## Template Walkthrough: A Streaming Pipeline on Google Cloud The walkthrough builds a minimal but complete pipeline. It resolves FASTQ URLs from ENA, launches a Dataproc Spark cluster, streams the reads into a distributed k-mer count and a quality-control pass, and writes results to a Cloud Storage bucket. Each step lists the command, the reasoning, and the expected output. ### Step 1: Configure the Project and Storage Enable the services the pipeline needs and create a bucket for code, URL lists, and results. ``` gcloud config set project YOUR_PROJECT_ID gcloud services enable dataproc.googleapis.com storage.googleapis.com gcloud storage buckets create gs://genomics-stream-demo \ --location=us-central1 \ --uniform-bucket-level-access ``` Expected output from the bucket creation: ``` Creating gs://genomics-stream-demo/... Completed 1/1 ``` The `-uniform-bucket-level-access` flag simplifies permissions by applying access control at the bucket level rather than per object. For a single-person pilot this is the least error-prone choice. The `-location` flag pins the bucket to a region, and keeping the bucket in the same region as the cluster avoids cross-region transfer latency. ### Step 2: Resolve Streaming Read URLs From ENA ENA exposes a REST endpoint that reports, for any accession, the FTP and HTTPS locations of the submitted FASTQ files. Query it with `curl`: ``` curl "https://www.ebi.ac.uk/ena/portal/api/filereport?accession=YOUR_ENA_ACCESSION&result=read_run&fields=run_accession,read_count,fastq_ftp&format=tsv" \ -o runs.tsv ``` The output is tab-separated text. The shape looks like this, with your accessions in place of the examples: ``` run_accession read_count fastq_ftp ERR4840308 21234567 ftp.sra.ebi.ac.uk/vol1/fastq/ERR484/008/ERR4840308/ERR4840308_1.fastq.gz,ftp.sra.ebi.ac.uk/vol1/fastq/ERR484/008/ERR4840308/ERR4840308_2.fastq.gz ``` The `read_count` field is worth keeping. You will use it later to verify that the stream delivered every read. The `fastq_ftp` field lists the read 1 and read 2 files as a comma-separated pair for paired-end data. Convert the FTP paths to HTTPS URLs and split the paired files onto separate lines: ``` tail -n +2 runs.tsv | cut -f3 | tr ',' '\n' | sed 's|^|https://|' > run_urls.txt head -n 2 run_urls.txt ``` Expected output: ``` https://ftp.sra.ebi.ac.uk/vol1/fastq/ERR484/008/ERR4840308/ERR4840308_1.fastq.gz https://ftp.sra.ebi.ac.uk/vol1/fastq/ERR484/008/ERR4840308/ERR4840308_2.fastq.gz ``` Copy the URL list to your bucket so the Spark cluster can read it: ``` gcloud storage cp run_urls.txt gs://genomics-stream-demo/inputs/run_urls.txt ``` ### Step 3: Launch a Dataproc Spark Cluster ``` gcloud dataproc clusters create stream-demo \ --region=us-central1 \ --master-machine-type=n1-standard-4 \ --worker-machine-type=n1-standard-8 \ --num-workers=4 ``` Expected output ends with: ``` Wait for [operation operation-...] to complete...done. clusterName: stream-demo ``` Four `n1-standard-8` workers give you 32 worker cores. That is a pilot scale, deliberately far below the 64 machines Flint used for its benchmark, and you should expect correspondingly longer runtimes. The machine types are placeholders for the sizing decision described in the parameters table below. ### Step 4: The Streaming Worker Code Save this as `stream_kmers.py`. Each Spark task opens one FASTQ URL, decodes gzip on the fly, and counts k-mers, which are all substrings of length K in each read. K-mer counting is the standard minimal demonstration that reads are flowing and being parsed correctly. ```python import gzip import sys import urllib.request from pyspark.sql import SparkSession K = 4 BUCKET = sys.argv[1] URL_FILE = sys.argv[2] spark = SparkSession.builder.appName("streaming-kmers").getOrCreate() sc = spark.sparkContext urls = sc.textFile(f"gs://{BUCKET}/{URL_FILE}", minPartitions=8) def counts_for_url(url): counts = {} attempts = 0 while True: try: with urllib.request.urlopen(url, timeout=60) as raw: stream = gzip.GzipFile(fileobj=raw) line_no = 0 for line in stream: line_no += 1 if line_no % 4 != 2: continue seq = line.strip().decode("ascii") for pos in range(0, len(seq) - K + 1): kmer = seq[pos:pos + K] counts[kmer] = counts.get(kmer, 0) + 1 return [(url, kmer, n) for kmer, n in counts.items()] except Exception as err: attempts += 1 if attempts >= 3: return [(url, "FAILED", str(err))] counts = {} ``` Two details in this function carry most of the correctness weight. The test `line_no % 4 != 2` selects the second line of every four-line FASTQ record, which is the sequence. The `urllib.request.urlopen` call with `timeout=60` combined with the retry loop turns transient network failures into retries instead of lost tasks. The rest of the script aggregates across tasks and writes the top k-mers: ```python rdd = urls.map(counts_for_url).flatMap(lambda rows: rows) totals = rdd.filter(lambda row: row[1] != "FAILED") \ .map(lambda row: (row[1], row[2])) \ .reduceByKey(lambda a, b: a + b) top = totals.takeOrdered(20, key=lambda pair: -pair[1]) out = sc.parallelize(top) out.saveAsTextFile(f"gs://{BUCKET}/results/top_kmers") failures = rdd.filter(lambda row: row[1] == "FAILED") failures.saveAsTextFile(f"gs://{BUCKET}/results/failures") ``` The `saveAsTextFile` calls write Spark's native format, one result per line, which is why the next step reads back with `cat`. The separate failures directory is deliberate. A silent partial failure is the worst outcome in a streaming pipeline, and writing failures to their own location makes them impossible to miss. ### Step 5: Submit and Watch the Job ``` gcloud dataproc jobs submit pyspark stream_kmers.py \ --cluster=stream-demo \ --region=us-central1 \ -- gs://genomics-stream-demo inputs/run_urls.txt ``` Expected output, abbreviated: ``` Job [b1a2c3d4e5f6] submitted. Waiting for job output... ... Job [b1a2c3d4e5f6] finished successfully. ``` Read the results back to your local machine: ``` gcloud storage cat "gs://genomics-stream-demo/results/top_kmers/part-*" ``` Expected output, one `(kmer, count)` pair per line: ``` ('AAAA', 483211) ('ATAT', 401557) ('ACGT', 398210) ('TTTT', 470115) ``` The exact counts depend entirely on your reads. What matters is that counts appear, that they are in the hundreds of thousands or millions for a real run, and that no entry in `results/failures` exists. ### Step 6: Stream a Quality-Control Pass First Before trusting any downstream result, reproduce the LSTrAP-Cloud philosophy and run quality control on the same stream. The most compact QC summary is the per-cycle mean quality, where a cycle is one position along the read. Quality characters in FASTQ are Phred+33 encoded, meaning the numeric quality score is the ASCII code of the character minus 33. Add this function and a driver block to a second script, `stream_qc.py`: ```python def qc_for_url(url): sums = None n = 0 with urllib.request.urlopen(url, timeout=60) as raw: stream = gzip.GzipFile(fileobj=raw) line_no = 0 for line in stream: line_no += 1 if line_no % 4 != 0: continue qual = line.strip().decode("ascii") vals = [ord(c) - 33 for c in qual] if sums is None: sums = [0.0] * len(vals) for i, v in enumerate(vals): sums[i] += v n += 1 if n == 0: return [(url, "EMPTY")] means = [round(s / n, 2) for s in sums] return [(url, i, m) for i, m in enumerate(means)] ``` The selector `line_no % 4 != 0` picks the fourth line of each record, the quality string. Run it with the same submit pattern as Step 5 and read the output: ``` gcloud storage cat "gs://genomics-stream-demo/results/qc/part-*" | head -n 8 ``` Expected output: ``` (0, 34.61) (1, 35.02) (2, 35.14) (3, 35.09) (4, 34.98) (5, 34.87) (6, 34.71) (7, 34.55) ``` Cycle 0 is the first base of the read. A healthy profile starts high and declines gently toward the read end. A steep drop in the first few cycles, or a collapse anywhere along the run, flags the sample for the trimming and filtering steps that LSTrAP-Cloud applies before its downstream analysis [1]. ### Step 7: Parameterize the Template Turn the constants into a config block so the same code serves different analyses. A minimal shell wrapper: ``` #!/bin/bash set -euo pipefail PROJECT=YOUR_PROJECT_ID REGION=us-central1 BUCKET=gs://genomics-stream-demo CLUSTER=stream-demo SCRIPT=stream_kmers.py URL_FILE=inputs/run_urls.txt gcloud config set project "$PROJECT" gcloud storage cp "$URL_FILE" "$BUCKET/$URL_FILE" gcloud dataproc jobs submit pyspark "$SCRIPT" \ --cluster="$CLUSTER" \ --region="$REGION" \ -- "$BUCKET" "$URL_FILE" ``` For larger production workflows, move this orchestration into an workflow scheduler such as Apache Airflow, which Google Cloud offers as the managed Cloud Composer service. The pipeline logic stays identical. The scheduler adds retries, dependencies between steps, and alerting when a run fails, which matters once the pipeline runs unattended. ## Key Parameters at a Glance | Parameter | Template default | What it controls | Practical guidance | |--|--|--|--| | K (k-mer length) | 4 | Substring length counted per read | Larger K gives more specific features and more memory per count | | `minPartitions` | 8 | Parallelism when reading the URL list | Set to at least 2 times the worker core count | | `-num-workers` | 4 | Dataproc worker node count | Scale up for large references, following the Flint model of many parallel mappers | | `-worker-machine-type` | n1-standard-8 | Cores and memory per worker | More memory per core helps reference-heavy workloads | | `timeout` in `urlopen` | 60 seconds | Network stall tolerance before a retry | Raise it for large files on slow links | | Retry count | 3 | Attempts per URL before marking FAILED | Log the final exception so failures are diagnosable | | Phred offset | 33 | ASCII-to-quality conversion in QC | 33 is the standard for modern FASTQ, 64 appears only in legacy data | | Region and bucket location | us-central1 | Where compute and storage live | Keep them matched to avoid cross-region transfers | ## Understanding the Streaming FASTQ Format The input format for this pipeline is gzip-compressed FASTQ served over HTTPS. A single uncompressed record looks like this: ``` @ERR4840308.1 1 length=150 GATCGGAAGAGCACACGTCTGAACTCCAGTCAC + FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF ``` Four facts about this format determine whether a streaming parser works: 1. Record boundaries are positional, not tagged. Every record is exactly four lines, so line number modulo 4 tells you which field you are holding. Any file that violates this invariant, for example a multi-line sequence, will silently corrupt a positional parser. 2. The quality line is Phred+33. The character `F` has ASCII code 70, so its quality score is 70 minus 33, which is 37. Scores of this magnitude indicate a very confident base call. 3. The stream is gzip-compressed end to end. Decoding on the fly means the first record is parseable before the last byte arrives, which is the property that lets Spark tasks start working immediately. 4. Paired-end reads arrive as two separate files from ENA, not as one interleaved file. Process each file independently unless your analysis requires mate pairing, and keep read 1 and read 2 counts separate in your QC output so a mismatch is visible. The output format depends on your analysis. LSTrAP-Cloud's terminal output is a gene coexpression network usable for function prediction from a single bait gene [1]. Flint's terminal output is a per-sample abundance profile across its 43,552-genome reference [2]. Both are downstream of the same streaming read boundary you built above. ## Checking That the Output Is Right ![Worked calculation showing 147 k-mers per 150-base read with K of 4, giving 294,000,000 for 2,000,000 reads.](/article-images/illustrations/google-cloud-streaming-genomics-pipeline-kmer-arithmetic.webp) A streaming pipeline can fail in ways that produce plausible-looking wrong numbers, so verification is not optional. Three checks catch most problems. First, verify the read count. The ENA filereport you ran in Step 2 returned a `read_count` per run. Count the records your QC pass actually saw and compare. A mismatch means the stream truncated and your retry logic lost data. Second, verify the k-mer arithmetic. Each read of length L contributes exactly L minus K plus 1 k-mers of length K. For a paired-end sample with 1 million read pairs of 150 bases each and K of 4, the expected total is: ``` kmers per read = 150 - 4 + 1 = 147 reads per sample = 2,000,000 (both mates) expected total = 2,000,000 * 147 = 294,000,000 ``` Sum your `reduceByKey` counts and compare against this total. Equality confirms the parser handled every line. A deficit points at skipped or malformed records. Third, verify the QC profile shape. Per-cycle mean quality should start in the mid-30s for modern Illumina data and decline gradually. Compare across samples streamed from the same study. An outlier that sits tens of points below its siblings is usually a sequencing or deposition problem, not a pipeline bug, and LSTrAP-Cloud's quality-control stage exists precisely to surface that before it contaminates a coexpression network [1]. ## Common Errors and Fixes ![Decision path from the job error message to the fix: rate limit, bad gzip, memory, permissions or network.](/article-images/illustrations/google-cloud-streaming-genomics-pipeline-error-triage.webp) These are the failures you will actually hit, with their exact messages. ### HTTPError: HTTP Error 429: Too Many Requests ENA rate-limits aggressive clients. If a large URL list triggers this, add a per-task delay, batch accessions into smaller runs, and keep the retry loop with increasing wait times between attempts. The retry structure in Step 4 is the right place to implement backoff. ### gzip.BadGzipFile: Not a gzipped file This means the URL returned something other than a FASTQ gzip file, most often an HTML error page. Verify the URL in a browser, confirm the ENA accession has submitted FASTQ files, and confirm you converted the FTP path to HTTPS correctly. Some archived runs store only BAM or CRAM format, and those URLs will never parse as FASTQ. ### org.apache.spark.SparkException: Job aborted due to stage failure This generic message almost always hides a specific cause one screen up in the log. The most common underlying cause in this pipeline is executor memory exhaustion, reported as `Container killed by YARN for exceeding memory limits`. Fix it by raising worker memory through a larger machine type, reducing per-task memory pressure, or shrinking K. Flint's benchmark ran on 64 machines precisely because mapping against a 170 GB reference collection demands it [2], and pilot-scale clusters should not be expected to reproduce that on a handful of workers. ### AccessDeniedException: 403 ... on gs:// The cluster's service account lacks permission on your bucket. Grant the service account the Storage Object Admin role on the bucket, or the read-only role if workers only consume inputs. This error appears identically in the driver log and in `gcloud storage` commands run under the same account. ### ImportError: No module named requests Third-party Python packages are not installed on Dataproc workers by default. The template avoids this by using `urllib.request`, which is part of the Python standard library. If you prefer `requests` for its retry handling, install it cluster-wide using a Dataproc initialization action, which is a script the cluster runs on every node at startup. ### Workers cannot reach ENA and all tasks time out On clusters created without external IP addresses, outbound internet access requires Cloud NAT. The symptom is every task failing with a connection timeout while the driver is healthy. Either recreate the cluster with external IPs for a pilot or attach a Cloud NAT configuration to the cluster's subnet. ## Common Mistakes and Limitations Treat the template as a starting point and correct these predictable errors before scaling up. 1. Expecting benchmark numbers at pilot scale. Flint profiled 1 million paired-end reads in 67 seconds on 64 machines [2]. A four-worker pilot cluster will not match that, and comparing your timing to that figure leads people to conclude the code is broken when it is simply smaller. 2. Confusing streaming ingest with stream processing. This template streams reads into batch-style Spark jobs. It does not process a live feed of data, and forcing it into Spark Structured Streaming adds complexity with no benefit for file-based archive data. 3. Re-fetching everything on every rerun. Streaming pipelines have no local cache by design. For data you analyze repeatedly, stage one copy into your own bucket and stream from there, accepting that you then own storage costs and access control. 4. Ignoring paired-end bookkeeping. Counting read 1 and read 2 URLs together hides the failure of one mate file. Track them separately from the URL list onward. 5. Treating a public-archive stream as private data. Reads streamed from ENA are public. Human data with privacy obligations requires an entirely different access and control regime, and individual deployments handling sensitive data need review from someone with both genomics and cloud security experience before any production use. 6. Skipping the QC pass because the pipeline "ran." A completed job says nothing about data quality. LSTrAP-Cloud places quality control before every downstream step for that reason [1], and so should you. 7. Hardcoding one region, one bucket, and one accession. Every value in this template is a placeholder for a reason. Parameterize before your second run, because the second run is when the hardcoded copy diverges from reality. ## How This Compares With AWS Genomics The same pattern runs on AWS, and one of the two source pipelines proves it. Flint was built on Amazon's Elastic MapReduce service and used Spark's streaming engine architecture for exactly the ingest pattern shown above [2]. Because the heavy lifting lives in PySpark, portability is high. The changes needed to run this template on AWS are the storage URI scheme, where `s3://` replaces `gs://`, and the cluster launch command, where an `emr` command replaces `gcloud dataproc`. The analysis code, the ENA streaming logic, and the validation arithmetic transfer unchanged. This matters for planning. The right provider choice for most labs is whichever managed Spark service their team already operates, because the pipeline itself is provider-neutral. The literature supports both: LSTrAP-Cloud demonstrates the Google-native notebook path [1], and Flint demonstrates the AWS cluster path [2]. Google Cloud's advantage in this pairing is the Colaboratory entry point for biologists who need results from a browser, and Dataproc for the same team when the workload grows into Flint's territory. ## Frequently Asked Questions ### Is there an official Google Cloud product called a streaming genomics pipeline? No. The phrase describes a design pattern, streaming FASTQ reads over the network into cloud-hosted compute, rather than a named Google Cloud service. On Google Cloud the pattern is implemented with Cloud Storage, Dataproc, and code you write, and both published source pipelines assembled it from general-purpose parts. ### What does the streaming genomics pipeline format mean in practice? It means gzip-compressed FASTQ delivered over HTTPS and parsed record by record, four lines at a time. That is the input format both source pipelines consumed, one streaming RNA-seq reads from the European Nucleotide Archive and the other streaming reads into Spark mappers for metagenomic profiling. ### Can I run the same template on AWS instead of Google Cloud? Yes. The analysis code is Apache Spark and transfers unchanged. Flint, the published metagenomic profiler, already ran this exact streaming pattern on Amazon Elastic MapReduce, so switching providers means changing storage paths and cluster launch commands, not rewriting the pipeline. ### Do I always need a Spark cluster for a streaming pipeline? No. LSTrAP-Cloud runs entirely in Google Colaboratory, a hosted notebook, and performs quality control on ENA-streamed RNA-seq reads plus coexpression network construction there. A cluster earns its cost only when the reference, sample count, or analysis outgrows a single session. ### How do I get sequencing data to stream from public archives? Query the European Nucleotide Archive's REST endpoint with your study or run accession to retrieve the FASTQ file locations and read counts, then hand the HTTPS URLs to your workers. The `curl` command in Step 2 of the walkthrough does this and returns a tab-separated list you can convert to URLs with one pipeline of standard text tools. ### How large a reference collection can this streaming approach handle? Large enough that reference size stopped being the bottleneck in the published benchmark, which mapped reads against a 170 GB collection of 43,552 bacterial genomes. The practical limit is cluster size and memory configuration, and the correct response to a growing reference is more workers and more memory per worker rather than a different architecture. ### How do I check that my streaming pipeline produced correct results? Cross-check the number of records streamed against the read count published by the archive, sum your k-mer counts against the arithmetic expectation of reads times read length minus k-mer length plus one, and inspect the per-cycle quality profile for anomalies. All three checks are described with worked calculations in the validation section above. ### What does running this pipeline cost? Cost is driven by machine hours on the cluster, storage class and volume in your bucket, and any data transfer. There is no fixed price for the pattern, and the responsible method is to size the cluster from your data volume, run a one-hour pilot, and read the actual machine hours from the job logs before projecting a full campaign. Use the Google Cloud pricing calculator with your chosen machine types and region for current rates. ## Sources 1. [LSTrAP-Cloud: A User-Friendly Cloud Computing Pipeline to Infer Coexpression Networks.](https://pubmed.ncbi.nlm.nih.gov/32316247/) 2. [Large scale microbiome profiling in the cloud.](https://pubmed.ncbi.nlm.nih.gov/31510682/)