November 5, 2020

Developers Bin

I'm fortunate to work for a team that gives its developers time to play -- a day, every other week, that we can use to learn about or try something new. Over the years, a number of useful stuff has come out of those days. Many of them are small utilities that developers have created for themselves -- for example, to automate a task. We've gathered quite an eclectic collection of these, written in all sorts of languages and frameworks.

Unfortunately, for the most part, these were just being used by the programmer who wrote them, squirrelled away in some directory in their path (~/bin). For example, I've written some command-line utilities in Clojure that I think others would find useful, but we're a Scala and NodeJS shop. I don't want them to have to install Clojure build tools to try them out.

How do we make it easy for team members to share these utilities with their colleagues?

Private Homebrew Taps

The solution was to create our own, private Homebrew repository that I've dubbed our "Developers Bin." I chose Homebrew because most of us use Macs and already brew install applications. And, although I'm on Ubuntu most of the time, brew is on Linux, as well.

Homebrew "formulas", if written well, automatically install and configure an application, and ensure that dependencies are installed as well. My colleagues can just brew install <my application> and not know, nor care, that it was written in Go, Elm, Haskell or Clojure. It just runs.

One of the first hurdles you'll run into, though, is how to create a private repository. Creating a Homebrew repository that others can "tap" is simple -- just create a repo on Github of the form <user>/homebrew-<repo-name>. Users can just brew tap <user>/<repo-name> and then be able to brew install anything specified in a Formulas directory. But if that repo must be private, as in our case, that's a little trickier.

Fortunately, someone's already written up instructions on how to do that, including the crucial custom_download_strategy.rb that's needed. This bit of Ruby code used to be part of Homebrew, but the maintainers (probably sensibly) didn't want to have to maintain code that they don't use themselves, so it's been ripped out.

Generating and Updating Formulas

The next hurdle was to make it easy for my colleagues to add formulas to our repository and keep them updated. The strategy I've settled on is to create a "template" for each formula that then gets turned into a formula (or updated formula) by a Jenkins job configured to run when code is merged into the main branch.

Here's an example template (somewhat redacted) that I used for one of my utilities in Clojure. Notice the placeholders for _RELEASE_VERSION_, _RELEASE_URL_ and _RELEASE_SHA__.

require_relative "../custom_download_strategy.rb"

class QuickLogin < Formula
  desc "Simple command-line interface for logging into one of our services."
  homepage "https://github.com/..."
  version "_RELEASE_VERSION_"
  url "_RELEASE_URL_", :using => CustomGitHubPrivateRepositoryReleaseDownloadStrategy
  sha256 "_RELEASE_SHA_"

  bottle :unneeded

  depends_on "openjdk"

  uses_from_macos "ruby" => :build

  def install
    system "./install.sh", prefix
    bin.env_script_all_files libexec/"bin", :JAVA_HOME => "${JAVA_HOME:-#{Formula["openjdk"].opt_prefix}}"
  end

  test do
    # cosmos-login --help
  end
end

It runs my install.sh script, pulls in Openjdk as a dependency and installs the code on the user's path. I've written a few variations of this template, depending on the language or runtime environment of the application. Our developers can just copy one that depends on "node" or depends on "python" or whatever and modify it.

I then had to write a script (run by Jenkins) to do the following:

  1. Create a release in Github.
  2. Build the code -- generating a JAR file in this case.
  3. Upload the JAR, and anything else needed, such as the install.sh script, to Github as a tar.gz file, tagged with the release.
  4. Finally, the script checks out our homebrew repo, uses the template to create (or update an existing) formula and checks the changes in again.

That sounds like a lot, and it is the first time you have to create one, but I've written one for Clojure, Python, Node and Bash scripts so far. Once I've written one, it's very easy to modify slightly for the next.

When a user runs brew install, for example, or brew upgrade (if they already have the application installed), Homebrew uses the instructions in the formula to download the artifact (using the SHA to do an integrity check) and install or update the application.

For the first few utilities, I had to do some of the heavy lifting -- create the scripts and the Jenkins jobs. But once my colleagues started to see our "Developers Bin" as a valuable shared resource, they've started adding formulas and templates themselves -- one or two a month.

We now have a valuable toolbox of utilities that are easy to install, easy to use and make our day-to-day work easier.

Tags: packaging homebrew