Using Google Cloud Pubsub for Batch Pipelines in Apache Beam

  Posted on November 23, 2022   ·   3 min read   ·   # comments   ·   #today-i-learned  #programming 

Google Cloud’s Pub/Sub is a useful service that provides an asynchronous and scalable messaging platform that decouples services producing messages from those that receive and process those messages. When combined with Apache Beam (and/or Dataflow, Google’s managed version of it), you can quickly develop powerful batch and streaming pipelines for data-parallel processing. However, I recently ran into one slight hiccup - although Apache Beam has a built-in IO connector for pubsub, it only supported streaming pipelines (at the time of development).

Tips and Tricks with Terraform's null_resource

  Posted on May 17, 2022   ·   2 min read   ·   # comments   ·   #today-i-learned  #programming 

Terraform’s null_resource resource can be useful when there aren’t any existing modules to satisfy your needs (with some caveats). Hashicorp’s documentation for it is a bit lacking, but fortunately there’s more information about the provisioners in their other docs here. After using these resources in a handful of places across our infrastructure deployments, I’ve developed a small collection of tips I picked up over the past few months that I thought I’d share.

Deploying Google Cloud Functions with Terraform

  Posted on March 8, 2022   ·   2 min read   ·   # comments   ·   #today-i-learned  #programming 

Cloud Functions are an easy, performant, and potentially inexpensive way to build serverless backends. I recently went down the route of setting up continuous deployments for them, and thought I’d share my learnings with it.

Precision Sourdough: A Smart Lid for Your Starter

  Posted on February 22, 2021   ·   12 min read   ·   # comments   ·   #3d-printing  #design  #electrical  #programming  #embedded 

A few years ago, I had the idea to track my sourdough starter using computer vision. It was neat to monitor it this way, but it was fairly impractical to do for each feeding since it required setting up a camera, downloading the images, and doing some manual image cropping before running it through my analysis script. The analysis was also only done after the fact, and what I really wanted was something that could tell me when the starter was ready to be used (or fed), or, if I missed the window of opportunity, how long ago it peaked.

Building a Backyard Chicken Coop in the City

  Posted on September 9, 2020   ·   6 min read   ·   # comments   ·   #woodworking 

Back in 2018, we decided to raise chickens in our urban backyard, all in the name of being able to have fresh eggs every morning. Our city had bylaws on having chickens, which included: A maximum of 4 hens (no roosters), 4 months or older Ducks, turkeys, or other livestock are not allowed Eggs, meat, and manure cannot be used for commercial purposes Backyard slaughtering is not allowed Since we were fortunate enough to have access to a backyard at the time, we decided to build a coop and start the lifestyle of having easy access to fresh eggs.