Using Google Cloud Pubsub for Batch Pipelines in Apache Beam

  Posted on November 23, 2022   ·   3 min read   ·   # comments   ·   #today-i-learned  #programming 

Google Cloud’s Pub/Sub is a useful service that provides an asynchronous and scalable messaging platform that decouples services producing messages from those that receive and process those messages. When combined with Apache Beam (and/or Dataflow, Google’s managed version of it), you can quickly develop powerful batch and streaming pipelines for data-parallel processing. However, I recently ran into one slight hiccup - although Apache Beam has a built-in IO connector for pubsub, it only supported streaming pipelines (at the time of development).

Tips and Tricks with Terraform's null_resource

  Posted on May 17, 2022   ·   2 min read   ·   # comments   ·   #today-i-learned  #programming 

Terraform’s null_resource resource can be useful when there aren’t any existing modules to satisfy your needs (with some caveats). Hashicorp’s documentation for it is a bit lacking, but fortunately there’s more information about the provisioners in their other docs here. After using these resources in a handful of places across our infrastructure deployments, I’ve developed a small collection of tips I picked up over the past few months that I thought I’d share.

Deploying Google Cloud Functions with Terraform

  Posted on March 8, 2022   ·   2 min read   ·   # comments   ·   #today-i-learned  #programming 

Cloud Functions are an easy, performant, and potentially inexpensive way to build serverless backends. I recently went down the route of setting up continuous deployments for them, and thought I’d share my learnings with it.