Posted on November 23, 2022 · 3 min read · # comments · #today-i-learned #programming
Google Cloud’s Pub/Sub is a useful service that provides an asynchronous and scalable messaging platform that decouples services producing messages from those that receive and process those messages. When combined with Apache Beam (and/or Dataflow, Google’s managed version of it), you can quickly develop powerful batch and streaming pipelines for data-parallel processing. However, I recently ran into one slight hiccup - although Apache Beam has a built-in IO connector for pubsub, it only supported streaming pipelines (at the time of development).
Posted on May 17, 2022 · 2 min read · # comments · #today-i-learned #programming
Terraform’s null_resource resource can be useful when there aren’t any existing modules to satisfy your needs (with some caveats). Hashicorp’s documentation for it is a bit lacking, but fortunately there’s more information about the provisioners in their other docs here. After using these resources in a handful of places across our infrastructure deployments, I’ve developed a small collection of tips I picked up over the past few months that I thought I’d share.
Posted on March 8, 2022 · 2 min read · # comments · #today-i-learned #programming
Cloud Functions are an easy, performant, and potentially inexpensive way to build serverless backends. I recently went down the route of setting up continuous deployments for them, and thought I’d share my learnings with it.