Using Google Cloud Pubsub for Batch Pipelines in Apache Beam

  Posted on November 23, 2022   ·   3 min read   ·   # comments   ·   #today-i-learned  #programming 

Google Cloud’s Pub/Sub is a useful service that provides an asynchronous and scalable messaging platform that decouples services producing messages from those that receive and process those messages. When combined with Apache Beam (and/or Dataflow, Google’s managed version of it), you can quickly develop powerful batch and streaming pipelines for data-parallel processing. However, I recently ran into one slight hiccup - although Apache Beam has a built-in IO connector for pubsub, it only supported streaming pipelines (at the time of development).

Tips and Tricks with Terraform's null_resource

  Posted on May 17, 2022   ·   2 min read   ·   # comments   ·   #today-i-learned  #programming 

Terraform’s null_resource resource can be useful when there aren’t any existing modules to satisfy your needs (with some caveats). Hashicorp’s documentation for it is a bit lacking, but fortunately there’s more information about the provisioners in their other docs here. After using these resources in a handful of places across our infrastructure deployments, I’ve developed a small collection of tips I picked up over the past few months that I thought I’d share.

Deploying Google Cloud Functions with Terraform

  Posted on March 8, 2022   ·   2 min read   ·   # comments   ·   #today-i-learned  #programming 

Cloud Functions are an easy, performant, and potentially inexpensive way to build serverless backends. I recently went down the route of setting up continuous deployments for them, and thought I’d share my learnings with it.

Precision Sourdough: A Smart Lid for Your Starter

  Posted on February 22, 2021   ·   12 min read   ·   # comments   ·   #3d-printing  #design  #electrical  #programming  #embedded 

A few years ago, I had the idea to track my sourdough starter using computer vision. It was neat to monitor it this way, but it was fairly impractical to do for each feeding since it required setting up a camera, downloading the images, and doing some manual image cropping before running it through my analysis script. The analysis was also only done after the fact, and what I really wanted was something that could tell me when the starter was ready to be used (or fed), or, if I missed the window of opportunity, how long ago it peaked.

Nordic nRF52 Development with Visual Studio Code

  Posted on April 25, 2019   ·   5 min read   ·   # comments   ·   #programming  #embedded 

A few years ago, I created a tutorial on setting up Visual Studio Code for development with the STM32. Since I’ve also been developing on the Nordic nRF52, I thought I’d share another tutorial to show how a project can be set up, flashed, and debugged using Visual Studio Code. The template project discussed in this post can be found on Github. Instructions The Nordic toolchain is cross-platform, but the instructions below are specifically for Linux.

Monitoring the Fermentation of Sourdough Starter with Computer Vision

  Posted on June 24, 2018   ·   15 min read   ·   # comments   ·   #programming 

Bread, the quintessence of life. People have survived for centuries off this staple consisting only of flour, water, salt, and yeast. Try consuming all these ingredients separately, and you’ll be in for a digestive surprise. However, mix them together and let time do its thing, and the result is the release of profound flavour, texture, and nutrients that were previously locked away. Despite it being relatively easy to turn dough into something that looks and feels like bread, the challenge is in squeezing every possible ounce of flavour and texture (using only those four ingredients) to achieve the embodiment of a true loaf of bread.

From Prototype to Cloud: A Python Recipe Converter

  Posted on April 6, 2018   ·   13 min read   ·   # comments   ·   #programming 

In case the title wasn’t clear, this blog post is about developing a web application using the Python programming language using Jupyter Lab, Flask, and the Heroku platform. If you were looking for an article on python recipes, you can start off with this one on making a poached Burmese python curry. The Backstory The problem with online baking recipes is that the majority of them use volumetric units. As any civilized baker would know, Patricia’s 1 cup of flour may very well be different than Patrick’s 1 cup of flour.

Synchronous vs Asynchronous Ping Sweep in C# Windows Form

  Posted on February 9, 2018   ·   8 min read   ·   # comments   ·   #programming 

As a mechatronics engineer (in training), sometimes I like to pretend that I also know how to program. In my most recent adventures to software land at MistyWest, I needed to write an application in C# that involved doing a ping sweep to find devices that were physically connected through ethernet. Since Google and Stack Overflow are my two best friends, I was able to find (what seemed to be) an off-the-net solution quite quickly.

Debugger Setup with GDB + OpenOCD in Visual Studio Code

  Posted on October 29, 2017   ·   2 min read   ·   # comments   ·   #programming  #embedded 

Visual Studio Code’s combination of functionality, customizability, and aesthetics makes it one of my favourite code editors. As such, I was set on making it work with embedded development since I was getting started with the STM32 line of microcontrollers. I was following the steps outlined in Mastering STM32 by Carmine Noviello (which is an excellent resource) until it said to use Eclipse, because life’s too short to use software with unnecessary bloat.

Overhead Robotic Gantry for Tethered VR Headsets

  Posted on April 4, 2017   ·   4 min read   ·   # comments   ·   #work  #programming  #design  #electrical 

Project Overview Objective: Create an autonomous gantry to follow the HTC Vive headset around, keeping its cable behind the user at all times. Motivation: An extravagant party prop for an evening at CES 2017, hosted by MistyWest. Features: CoreXY planar gantry design System built with 8020 aluminum extrusions and laser cut acrylic components Stepper motor control through Teensy 3.2 HTC Vive pose tracking through C++ Patent pending Skills:

ESP8266 Sous Vide Controller

  Posted on February 5, 2017   ·   6 min read   ·   # comments   ·   #electrical  #programming  #embedded 

Project Summary Objective: Create a small, modular controller to regulate the temperature of a water bath. Motivation: To get in on this cooking fad without dropping fat stacks of cash on an immersion circulator. Features: Crisp 0.96" OLED display Pushbutton rotary encoder provides simple user interaction Removable temperature plug through standard 3-pos audio connector Temperature controlled outlet to be used with any heating element (ie. rice cooker, slow cooker, etc.

Engineer's Diary

  Posted on October 1, 2016   ·   2 min read   ·   # comments   ·   #programming 

Background: Between paper notebooks, post-it notes, OneNote, Evernote, and so many more, there is no shortage of ways to write things down. Each has its strengths and weaknesses, but none satisfied my requirements to act as a daily work log to record key events, thoughts, and milestones during my work day. My paper notebook is excellent for free-form thoughts, sketches, and calculations, but I would want to keep a separate notebook to keep track of these sequential events.

The Making of Project Haikuza: Part 2

  Posted on August 6, 2015   ·   6 min read   ·   # comments   ·   #programming 

The format of this series is an outline of my thought process during the development of @thehaikuza. Poetry is hard To write when algorithms Are extremely dumb. I dont want my haiku generator to be a vegetarian chef. There’s nothing wrong with always making word salad, but eventually it’ll have to learn to make fancier things. A poetic risotto would be nice from time to time. Leaving @thehaikuza to make complete gibberish wasnt what I had intended.

The Making of Project Haikuza: Part 1

  Posted on July 12, 2015   ·   7 min read   ·   # comments   ·   #programming 

The format of this series is an outline of my thought process during the development of @thehaikuza. Haikus are simple Even children can write them maybe programs too? Nothing is cooler than algorithmic poetry. Except for maybe Carl Sagan. I heard he was a pretty cool guy. I was listening to the radio while driving home one Sunday evening, and an ad came up for a university that was submitting computer-generated poetry to a literature competition.

Project Haikuza

  Posted on July 10, 2015   ·   1 min read   ·   # comments   ·   #programming 

Objective: Develop an algorithm to generate haikus using song lyrics. Motivation: Because computational linguistics are cool. Project: twitter.com/thehaikuza Features: Scrapes Virgin Radio’s broadcast history to find recently played songs Creates a song-based haiku queue in Google Sheets Generates a haiku using the queue as a reference and posts it on Twitter Checks for new tweets every 5 minutes and generates a relevant haiku, if requested Finds all song lyrics from Lyrics Wikia Runs on a Raspberry Pi Challenges:

External Ballistics Simulation

  Posted on February 19, 2015   ·   1 min read   ·   # comments   ·   #programming 

Background: including (but not limited to) drag, gravity, air density, altitude, rotation of the bullet, and rotation of the Earth. One solution is to connect a computer to a manual targeting system (ie. a scope) and estimate the corrected target location by accounting for these external factors. The user may then line the manual targeting system up with the corrected target location, hoping to the high heavens that the target will be hit upon releasing the projectile.

MATLAB Photo Editing Script

  Posted on January 12, 2014   ·   1 min read   ·   # comments   ·   #programming 

Objective: Develop a script to find and remove any differences in a series of photos. Motivation: How to Remove People From Your Travel Using Photoshop Framework: MATLAB Script takes any number of images (preferably taken on a tripod) and combines them into one cleaned-up image. Flowchart of the implemented algorithm.