The one thing necessary

Putting email in its place with Emacs and Mu4e

Wed, 03 Dec 2025 00:00:00 +0000

I had a few free hours recently and decided to have a go at a long-simmering idea: To improve how I manage multiple email accounts. As someone who struggles mightily to cope with distractions (I have the attention span of a gnat), I was hoping to consolidate several separate interfaces into one, simpler, less distracting, more focused client.

Sadly, I wasn’t quite successful – my work emails remain out of reach. But I learned a lot and ended up in at least a better place.

There are already many guides on the Internet around what I’ve done here, but all of the information I needed was a bit scattered, especially around using OAuth 2.0. I thought it was worth writing up for posterity, even if just for me.

Why?

I manage my schedule, notes and to-dos in Emacs, via Org Mode and Org Roam. I get some ribbing from (much younger) VS Code-using colleagues for that – using Emacs in 2025 is a bit like wearing a bow tie – but I’ve whittled and shaped Emacs into a sharp, well-honed tool over the last 30 years or so. It fits my brain nicely. Having my email directly available – instead of jumping to a completely different interface or, worse, the Distracto Land™ of the browser – when I’m organising things or taking notes just makes sense for me.

I also wanted this to specifically work offline. I’m writing this right now on the train from London to Manchester and, despite the spotty-at-best connection, I have access to all my emails.

Why Mu4e? There are other email clients for Emacs, including Gnus, which is built-in, and multiple back ends and indexers like Notmuch. I settled on Mu4e (a combination of an indexer call mu and an Emacs package), because I liked its nicely compact and customisable interface, where I can easily set up quick (two character) shortcuts to my most common email searches.

However, most of what I’m going to go through here would be applicable to many other offline email clients (Emacs or not), so you might find it useful, even if you don’t agree with me on Mu4e.

The main Mu4e page, where you can set up bookmarks to quickly check multiple accounts..

The message view.

Gathering the Emails

I started this whole process with my personal Gmail account, and then moved on to the emails I help manage as a volunteer for a local charity (a chapter of the Saint Vincent de Paul society). I mostly work in its food bank, but I also act as an unofficial IT guy and provide a couple of client-facing emails (info@ and foodbank@), which are currently implemented with Amazon’s Simple Email Service and proxied through another Gmail account.

I’m considering another email provider, Fastmail, for managing those charity emails. I’m currently in the 30-day free trial, so I threw in that one as well.

For all of the emails, I’ll use the Internet Message Access Protocol (IMAP) to download the messages and save them locally in a standard format called Maildir for searching and viewing.

There are a couple of options for doing this. I tried mbsync first, but ran into issues with Gmail (the downloads kept stopping early with an error). OfflineIMAP was slower, but worked better for me.

On the Mac, you can install OfflineIMAP via brew: brew install offlineimap. On Linux, you can also use brew, but probably best to use something like pipx.

Setting up OAuth 2.0 for Google

OfflineIMAP needs to know how to log in to your provider and retrieve the emails. OfflineIMAP’s deep integration with Python (which you can call directly from the configuration file) and packages such as keyring and google-auth make this almost straightforward. (I’ll mention some issues later). It allows me to keep any login details out of my config files and just call functions to retrieve them at runtime, which is better for security.

Most of the guides on the Web steer you toward “app passwords”, which was the previous way of securely providing access to your Gmail account from third party tools. Fastmail, as we’ll see, still uses these. Google, however, has deprecated them. They seemed to work at first, but they fail (get rejected) fairly randomly. I’m just guessing, but I think this is Google’s way of gently nudging us on to more modern authentication.

I put most of the code I wrote for this step in a separate repo – eamonnsullivan/oauth2-tools. The code is very heavily inspired by Kuibin Lin’s blog post for setting up a Raspberry Pi to use Gmail as a relay (which I also needed to do). The main changes I did was to use uv for the package manager and some refactoring to handle multiple gmail accounts.

But, first, I needed to create an OAuth client in Google Cloud:

Go to Google’s cloud console, and create a new project.
Navigate to API & Services, then to OAuth consent screen, click Get Started.
Fill out the form. The important bit is to choose ‘External’ for User Type
Create a new OAuth client (API & Services -> Credentials -> Create Credentials -> OAuth client ID).
For Application Type choose ‘Desktop App.’
When created, download the client secret (JSON) file.

I needed to do this for both Gmail accounts.

Using that file, run the authorise.py script to create a credentials.json file for each account. The next couple of scripts, get_token.py and get_auth_creds.py just use that credentials.json file to get the fields I’ll need, updating any expired token automatically. I’ve put these in my local bin directory (~/bin/).

It’s important to protect these JSON files like passwords, so at a bare minimum, make sure only you have access to the file:

chmod 0600 *.json

If they are lost, stolen or you accidentally check them into Github (don’t laugh), you can disable the client in the Google Cloud console and create a new one.

Creating an app password for Fastmail

Fastmail still uses app passwords. They also provide OAuth2, but you need to contact them to become a “partner” or something or other (at the time of writing). I’ll just use the app password during my free trial period.

In Fastmail’s security settings, click on “Manage app passwords and access” to create a new one. I then saved it to Keychain on my Mac.

security add-generic-password -s offlineimap -a me@mydomain.com -w

Or, on Linux, something like:

secret-tool store --label="offlineimap" password value

In both cases, I was prompted to enter the password.

Once it’s stored in the local secrets system, I can retrieve the passwords with a script. I use something like this, using uv’s support for standalone scripts, and saved in my local ~/bin directory:

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.14"
# dependencies = [
# "keyring",
# ]
# ///

import keyring

def get_password_from_keychain(account: str) -> str:
 """
 Retrieve the password for the given account from the macOS Keychain or Gnome Keyring.

 Args:
 account (str): The name of the account for which to retrieve the password.

 Returns:
 str: The retrieved password.
 """
 return keyring.get_password("offlineimap", account)

My offlineimap_helper.py

I put my OfflineIMAP Python “helper” code in ~/bin/offlineimap_helper.py. One issue I ran into is that OfflineIMAP can’t seem to handle any dependencies outside of the standard library. If I tried to use the function above in my helper library, I’d get an error:

ERROR: No module named 'keyring'

To be honest, that could be because I’m a very rusty Python programmer and I’m missing some trick with virtual environments or something, but I couldn’t figure it out. Instead, my ~/bin/offlineimap_helper.py file actually runs these scripts and gets the values it needs from standard out. I did it this way, rather than just opening the credentials file directly (it’s just JSON) because I want to keep a layer of abstraction in place, for when I (probably) add other OAuth providers. That’s probably overthinking it!

import os
from subprocess import check_output

def get_password_from_keychain(account: str) -> str:
 """
 Retrieve the password for the given account from macOS Keychain.
 """
 return check_output(
 [
 "security",
 "find-generic-password",
 "-s",
 "offlineimap",
 "-a", account,
 "-w"],
 env=os.environ
 ).strip()

def get_client_id(credentials: str) -> str:
 """
 Retrieve the OAuth2 client id for the given credentials file.
 """
 return check_output(
 [
 "get_auth_creds",
 credentials,
 "client_id"
 ],
 encoding="utf-8",
 env=os.environ
 ).strip()

def get_client_secret(credentials: str) -> str:
 """Retrieve the OAuth2 client secret from the given credentials.
 """
 return check_output(
 [
 "get_auth_creds",
 credentials,
 "client_secret"
 ],
 encoding="utf-8",
 env=os.environ
 ).strip()

def get_refresh_token(credentials: str) -> str:
 """
 Retrieve the OAuth2 refresh token from the given credentials.
 """
 return check_output(
 [
 "get_auth_creds",
 credentials,
 "refresh_token"
 ],
 encoding="utf-8",
 env=os.environ
 ).strip()

Saving the emails

We now have all the tools we need to retrieve the emails with OfflineIMAP.

I created the directories where I wanted my two Gmail accounts and one Fastmail account to go to.

mkdir -p ~/.maildir/gmail1 ~/.maildir/gmail2 ~/.maildir/fastmail

OfflineIMAP also needs to be pointed toward the root certificates to use secure connections. I found it easier to just copy the root certificates to a file in .maildir. I followed this guide to export them from Keychain on MacOS and save them as a pem file in ~/.maildir/certificates.

My current configuration for OfflineIMAP (which I put in ~/.offlineimaprc) looks something like this (I’ve anonymised it), for the three accounts:

[general]
accounts = Gmail1,Gmail2,Fastmail
# The implementation code for get_client_id,
# get_client_secret,
# get_password_from_keychain and
# get_refresh_token used below.
pythonfile = ~/bin/offlineimap_helper.py
[Account Gmail1]
localrepository = Local1
remoterepository = Remote1
synclabels = yes
[Repository Local1]
type = GmailMaildir
localfolders = ~/.maildir/gmail1/
[Repository Remote1]
type = Gmail
auth_mechanisms = XOAUTH2
oauth2_client_id_eval = get_client_id("/path/to/gmail1/credentials.json")
oauth2_client_secret_eval = get_client_secret("/path/to/gmail1/credentials.json")
oauth2_request_url = https://oauth2.googleapis.com/token
oauth2_refresh_token_eval = get_refresh_token("/path/to/gmail1/credentials.json")
remotehost = imap.gmail.com
remoteuser = user.name1@gmail.com
ssl = yes
sslcacertfile = ~/.maildir/certificates/root-certificates.pem
[Account Gmail2]
localrepository = Local2
remoterepository = Remote2
synclabels = yes
# I don't need every email for this one, just the last couple of months
maxage = 60
[Repository Local2]
type = GmailMaildir
localfolders = ~/.maildir/gmail2/
[Repository Remote2]
type = Gmail
auth_mechanisms = XOAUTH2
oauth2_client_id_eval = get_client_id("/path/to/gmail2/credentials.json")
oauth2_client_secret_eval = get_client_secret("/path/to/gmail2/credentials.json")
oauth2_request_url = https://oauth2.googleapis.com/token
oauth2_refresh_token_eval = get_refresh_token("/path/to/gmail2/credentials.json")
remotehost = imap.gmail.com
remoteuser = user.name2@gmail.com
ssl = yes
sslcacertfile = ~/.maildir/certificates/root-certificates.pem
[Account Fastmail]
localrepository = LocalFastmail
remoterepository = RemoteFastmail
synclabels = yes
[Repository LocalFastmail]
type = Maildir
localfolders = ~/.maildir/fastmail/
[Repository RemoteFastmail]
type = IMAP
# I'm not sure which auth mechanism Fastmail is using,
# so here are some options. It'll try each until
# one works.
auth_mechanisms = CRAM-MD5, PLAIN, LOGIN
remotehost = imap.fastmail.com
remoteport = 993
remoteuser = me@mydomain.com
remotepasseval = get_password_from_keychain("me@mydomain.com")
ssl = yes
sslcacertfile = ~/.maildir/certificates/root-certificates.pem

I then just ran offlineimap to start downloading my email. For my primary personal Gmail account, this took days (many, many gigabytes of emails dating back to 2004), but for my secondary one (where I’m only downloading a couple of months) and new Fastmail account, it was quick.

Sending email

To send, most providers (usually) provide an SMTP service for this. I’m using msmtp, which is among the simplest mail transport agents to set up.

On the mac, you can brew install msmtp. On Linux, it’s usually available in the standard repositories. On Debian-flavoured distributions (e.g., Ubuntu) run this:

sudo apt-get install msmtp msmtp-mta

Again, we don’t want any plain text credentials in the config, so we’ll use the scripts to either retrieve a valid OAuth token or the app password. This is my ~/.msmtprc (anonymised):

defaults
logfile ~/.maildir/msmtp.log
tls_trust_file ~/.maildir/certificates/root-certificates.pem
account gmail1
auth oauthbearer
host smtp.gmail.com
port 587
protocol smtp
from user.name1@gmail.com
user user.name1@gmail.com
passwordeval "get_token.py -c /path/to/gmail1/credentials.json"
tls on
tls_starttls on
account gmail2
auth oauthbearer
host smtp.gmail.com
port 587
protocol smtp
from user.name2@gmail.com
user user.name2@gmail.com
passwordeval "get_token.py -c /path/to/gmail2/credentials.json"
tls on
tls_starttls on
account fastmail
auth on
host smtp.fastmail.com
port 465
protocol smtp
from me@mydomain.com
user me@mydomain.com
passwordeval "security find-generic-password -s offlineimap -a me@mydomain.com -w"
tls on
# This needs to be off for Fastmail, for some reason
tls_starttls off
account default : gmail1

The get_token.py script returns the actual login token directly and refreshes it automatically if the token has expired.

Setting up Mu4e

The final piece of the puzzle is to use mu to index all the messages and set up Mu4e to search that index and send email. The creator of Mu4e has written some excellent documentation and you can see my complete Emacs config of Mu4e in Github, so I’ll just call out a few bits.

To install mu, I used brew install mu on the Mac. I use straight.el to manage my Emacs configuration, so this got a little complicated (Mu4e needs some special tools to compile), but I could use the brew installed version of Mu4e directly:

(use-package mu4e
 :straight
 (:local-repo "/opt/homebrew/share/emacs/site-lisp/mu/mu4e"
 :type built-in)
 [...]

Once I downloaded my email, I needed to initialise the index. I did something like this:

mu init --personal-address user.name1@gmail.com \
 --personal-address user.name2@gmail.com \
 --personal-address me@mydomain.com \
 --maildir ~/.maildir

For a complete example, take a look at my config. I have three different “contexts” for the different accounts, each with their own settings. I’ve also started (barely) customising the experiencing, adding my own actions. This is one that saves a link to the current message and creates a node in Org Roam:

(defun eds/get-org-directory ()
 "The location of my org directory varies by computer."
 (file-truename "~/Dropbox/org"))

(defun eds/strip-invalid-chars (title)
 "Strip characters that don't work in filenames or github branches"
 (replace-regexp-in-string "[\\?\\>\\<\\|\\:\\&]" "" title))

(defun eds/process-title (title)
 "Downcase and hyphenate a title case string. Remove characters
that don't work in a filename."
 (eds/strip-invalid-chars (mapconcat 'identity (split-string (downcase title)) "-")))

(defun eds/orgify-msg (msg)
 "Create a new org-roam node from an email message."
 (let* ((link (org-store-link msg nil))
 (subject (mu4e-message-field msg :subject))
 (time-string (format-time-string "%Y%m%dT%H%M%S"))
 (clean-subject (eds/strip-invalid-chars subject))
 (extension "org")
 (slug (eds/process-title clean-subject))
 (file-base-name (concat time-string "-" slug "." extension))
 (org-roam-directory (eds/get-org-directory))
 (file-name (expand-file-name file-base-name org-roam-directory))
 (title-line (format "#+title: %s\n" subject))
 (startup-line "#+startup: content\n")
 (first-heading (format "* %s\n" link)))
 (find-file file-name)
 (insert (concat title-line startup-line first-heading))
 (org-id-get-create)
 (end-of-buffer)
 (save-buffer)
 (org-roam-db-sync)))

(add-to-list 'mu4e-view-actions
 '("Org" . eds/orgify-msg) t)

The message "action" menu, with my custom Org action.

Tripped up at the last hurdle

Alas, the email I really struggle with is for work. My personal and charity emails are important, but low volume. My work emails are important and high volume. We use Outlook and Microsoft 365, which do provide hooks I need, but my employer has disabled IMAP, SMTP and support for most third-party clients (except Apple Mail and I think whatever the equivalent is on Android). I can’t, for example, create the client id and secret I need to use OAuth 2. My employer has done this for undoubtedly good security reason (smaller surface area to worry about, for example), but it’s not great for me.

Still, I’ve learned quite a bit, and I’ve brought at least my personal and charity emails into the Org system. I can create a link to an email and make a to-do item out of it in a couple of keystrokes. That’s progress, and worth doing.

Local TLS the easy way, with Let's Encrypt and Certbot

Mon, 01 Apr 2024 00:00:00 +0000

One huge and welcome change in the last decade or so for Web development is that secure connections are ubiquitous. Free certificates are available and most hosting providers make it very easy to obtain, use and renew the certificates automatically. This site, for example, runs on Amazon Web Services and enabling https took not much more effort than ticking a few checkboxes.

Browsers, like Chrome, Firefox and Safari, have also played their part, by gently steering users toward secure connections, warning about sending sensitive information over plain text and switching to https automatically, when available.

The last bastion for plain old http connections are web apps running on your local network, like a home wiki. But, so what? This is my local network, right? Yes, it is, but using SSL to encrypt the traffic between your browser and the application is a good, belt-and-braces, defence-in-depth security step. It can protect you from a dodgy device or compromised application, inside of your network, from listening in on your traffic.

Enabling local certs

There are a few ways to enable SSL connections to local servers. The easiest way is just use self-signed certificates. The disadvantage is that it requires you to accept these certificates on each browser you want to use.

Another option, which many larger companies use, is to run your own Certificate Authority. That isn’t easy, and you’ll still need to configure all of your browsers to accept your local root certificate. (This can be done with configuration scripting like Puppet or something. But see the isn’t easy part…)

Either of those methods would be fine if I were the only user, but that isn’t always the case. We have a local, household wiki, which acts as this house’s README. I want to make that available to any guests, where they can find info like where the cat food is, when the rubbish is picked up and where the water shut off valve is. Once connected to my WiFi, guests can open the wiki using QR codes posted around the house. I don’t want them to have to click through any warnings about insecure connections or have to accept any self-signed certificates.

A portion of the local wiki page for my house.

Local domain

The option I went with was to use a real, Let’s Encrypt certificate for a domain I have registered but not used publicly. In my case, I had sullivanhome.org.uk available.

The first step was to tell my router that this domain is the “local” one, which is easy enough to do on most routers, as long as you can get to the admin pages.

Configuring my router (Asus ZenWiFi AX) to use my registered domain as the "local" domain.

Once this is set up, local computers resolve on that domain.

My router acts as the local DNS, serving computers from the domain you specify.

Certbot

The next step is to obtain a wildcard certificate (e.g., one for “*.sullivanhome.org.uk”). My applications run across two Raspberry Pis, one called Pineapple and the other called Blueberry. I want one cert that I can apply on both, so that either pineapple.sullivanhome.org.uk or blueberry.sullivanhome.org.uk present as valid (with a lock icon and no errors) in the browser.

Fortunately, this is made extremely easy with certbot. This little tool from the Electronic Frontier Foundation automates the process of obtaining and periodically renewing the certificate almost entirely.

On Pineapple (which runs Ubuntu), I installed certbot from the snap store. I also had to change the snap store permissions and install a plugin for the registration provider I’m using (Amazon’s Route53). There are plugins for a wide variety of registrars.

sudo snap install --classic certbot
sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-route53

To get our certificate, I just ran the following.

certbot certonly --dns-route53 -d sullivanhome.org.uk -d *.sullivanhome.org.uk

That command needs a bit of explanation: The certonly argument tells certbot to just download the certs. It can install the certs directly in many applications and web servers (e.g., nginx or apache), but I have several different applications using the same certs, so I’ll do that later. The -dns-route53 option tells certbot to use the plugin to temporarily add entries in the DNS to prove that we own the domain. Finally, the -d option tells certbot what certificates to register.

Certbot then prompted me for various bits it needs (like my email address), registered me with Let’s Encrypt and downloaded the certificates to /etc/letencrypt. The /etc/letsencrypt/live directory looks like:

lrwxrwxrwx 1 root root 43 Jan 4 15:54 cert.pem -> ../../archive/sullivanhome.org.uk/cert1.pem
lrwxrwxrwx 1 root root 44 Jan 4 15:54 chain.pem -> ../../archive/sullivanhome.org.uk/chain1.pem
lrwxrwxrwx 1 root root 48 Jan 4 15:54 fullchain.pem -> ../../archive/sullivanhome.org.uk/fullchain1.pem
lrwxrwxrwx 1 root root 46 Jan 4 15:54 privkey.pem -> ../../archive/sullivanhome.org.uk/privkey1.pem

Automatic renewals

Notice that the live certs (the currently active certs) are symlinks. The software also sets up a systemctl timer service that will renew the certificate before it expires and update those symlinks. The renewal feature allows me to provide a script that gets called on a successful renewal (in /etc/letsencrypt/renewal-hooks/post/post-renew-certs.sh). I use this script to copy the file (via scp) from pineapple to blueberry. All of my local web apps and services run in docker, so I also restart the containers on both Raspberry Pis.

#!/bin/bash

# Make a backup copy on blueberry
scp /etc/letsencrypt/archive/sullivanhome.org.uk/* blueberry.local:/etc/letsencrypt/archive/sullivanhome.org.uk
# This will follow the symlinks and create non-symlinked files
ssh blueberry.local rm /etc/letsencrypt/live/sullivanhome.org.uk/*
scp /etc/letsencrypt/live/sullivanhome.org.uk/* blueberry.local:/etc/letsencrypt/live/sullivanhome.org.uk
# restart the services on blueberry -- this script just navigates to the appropriate directory
# runs docker compose restart (like below)
ssh blueberry.local /etc/letsencrypt/renewal-hooks/post/post-renew-certs.sh

cd /media/homeassistant/home_assistant/
docker compose restart

Using the certs

Changing the applications so that they present themselves using these certificates varies a lot by application. Node RED, for example, provides a function in its configuration to retrieve the certs, force https and periodically pick up any changes to the certs. I make the certs available in the docker container in a volume and then:

https: function() {
 // This function should return the options object, or a Promise
 // that resolves to the options object
 return {
 key: require("fs").readFileSync('/certs/live/sullivanhome.org.uk/privkey.pem'),
 cert: require("fs").readFileSync('/certs/live/sullivanhome.org.uk/fullchain.pem')
 }
},

/** If the `https` setting is a function, the following setting can be used
 * to set how often, in hours, the function will be called. That can be used
 * to refresh any certificates.
 */
httpsRefreshInterval : 12,

/** The following property can be used to cause insecure HTTP connections to
 * be redirected to HTTPS.
 */
requireHttps: true,

Wiki

For the home wiki, it’s a bit more complicated. It has a database (PostGres) behind it and a front end that does support Let’s Encrypt certificates, but in a way that doesn’t work well with my setup. So, instead, I installed nginx (sudo apt install nginx) and used what’s called a reverse proxy. This sits in front of the non-SSL enabled site and just proxies it, serving it over SSL.

Install and configure nginx

On blueberry, I created a file in /etc/nginx/sites-available called blueberry.sullivanhome.org.uk. The contents of that file is as follows:

server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
ssl_certificate /etc/letsencrypt/live/sullivanhome.org.uk/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sullivanhome.org.uk/privkey.pem;
server_name blueberry.sullivanhome.org.uk;
location / {
proxy_pass http://localhost:6830;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}

Then make that site “available” with a symlink and start it:

sudo ln -s /etc/nginx/sites-available/blueberry.sullivanhome.org.uk /etc/nginx/sites-enabled/blueberry.sullivanhome.org.uk
sudo systemctl reload nginx
# or restart
sudo systemctl start nginx

That’s it. When you go to https://blueberry.sullivanhome.org.uk in your browser, nginx forwards your request to localhost:6830, which is where I set the wiki to serve from.

And, that’s it

This set up has been working well for me now for three auto-renewals of the 90-day cert and happens without any intervention on my part. But if something does go wrong, I have all I need (backups of the certs, etc) to fix things.

The caveat is that I’m not a security expert and this is unlikely to be the best approach for a public app, or for use in a public place (a restaurant’s WiFi, maybe) but it works for my purposes.

The extra stuff about Home Assistant

Fri, 16 Feb 2024 00:00:00 +0000

My last post was my first in a long time and I’d forgotten how hard they are to write. After a couple of days, I got sick of looking at it and just chopped out a whole bunch of detail and posted it. Here’s a quick follow-up with some bits I left out.

Zigbee vs Z-Wave

Visualisations of my Zigbee (left) and Z-Wave (right) networks.

Zigbee and Z-Wave are low-power, mesh networking protocols aimed at home automation. I lean more toward Zigbee. As mentioned in the article, I started out with several Philips Hue devices (bulbs and motion sensors), which use Zigbee under the hood, and that’s probably the main reason why I have more Zigbee devices (38) than Z-Wave (seven), but not the only reason.

Z-Wave has some advantages over Zigbee. It doesn’t use the same radio spectrum as WiFi, so it’s less susceptible to interference. The certification process is stricter: Z-Wave devices are much more likely to Just Work™ in my experience than the average Zigbee gadget.

But on the downside, Z-Wave devices are lot more expensive and there are fewer options to choose from. Also, the Z-Wave protocol is more power-hungry than Zigbee. My one Z-Wave motion sensor in the kitchen goes through its odd-ball CR123A battery every few months, while my Zigbee devices can last two years on their plain double-As.

So, I voted with my wallet. Yes, the average Aliexpress impulse buy is a gamble, but I can afford to try out more devices. I was considering getting rid of Z-Wave, but its been so reliable for what I use it for (mostly smart plugs with energy monitoring) that I’ll probably keep it. I’m encouraged that Nabu Casa, the company behind Home Assistant, has joined the Z-Wave Alliance.

Home Assistant Community Store

Even though I use docker to run Home Assistant, I can still install the (very unofficial, unsupported) Home Assistant Community Store (HACS) and add third-party plugins. I use a handful of these. I mentioned Node-RED in the last post and that requires a plugin, but I also use a few others that have made my life simpler.

One that I added last summer is Adaptive Lighting. This adjusts the brightness and temperature of my lights based on the elevation of the sun. My automations used to have lots of code for adjusting the lights depending on the time of day. With Adaptive Lighting, those automations got a lot simpler. I just turn on the light and let the plugin figure out how brightly.

Another very useful addition is Tuya Local. The ceiling fan in our bedroom uses the Tuya App, which is a popular cloud-based App for many Chinese devices, but I’ve found its cloud services to be very unreliable.

I used to have a bunch of Tuya-controlled smart plugs, for example, that were extremely flaky, mostly because Tuya insists on serving me from data centres in Poland, about a thousand miles from London. But the fan was exactly what we wanted, so we gave Tuya another shot.

It was still terrible. The fan frequently wouldn’t turn on or wouldn’t turn off. The Tuya Local code talks directly to the fan, bypassing the long trip to Poland. We’ve had a few niggles at first, but it’s been trouble-free for almost a year now.

Philips Hue

I mentioned in the previous post that, if I had to do it over again, I might have tried smart switches for the lighting, instead of smart bulbs. But I still really like my Philips Hue bulbs. I like being able to control the colour, temperature and brightness (now via Adaptive Lighting) and the quality is quite good: the light looks nice and all of them are still working.

The Hue bulbs, especially the full-colour ones, cost the earth, though. I can’t afford them. Only some of my bulbs are full-colour ones, and the rest are what are called “White ambiance”. I’ve tried various other Zigbee bulbs, such as those from Ikea, but they just aren’t as good.

If I could get over my fear of fiddling with light switches (gangs and such!), I might try a hybrid solution. The switches would turn on/off the smart bulbs, but I could still control the brightness.

WLED

I'm getting the hang of using strip lighting, such as these on my living room shelves.

Another technology I’ve incorporated into the home is WLED lighting. These use WiFi, but locally only, and are supported quite well by Home Assistant. My first attempts at creating using strips were quite rudimentary – bare strips with ESP32 chips hanging off them. But I’m getting the hang of it now. These ones in my living room have proper rails, but they’re still using developer board ESP32 chips with clip-on wiring and they’re a bit flaky. My next challenge is to get the hang of soldering, which I hope will make them more reliable.

All for now. I’m sure there are bits I’m forgetting, but those are the main extra bits I wanted to mention.

Home Assistant: Three years later

Sun, 11 Feb 2024 00:00:00 +0000

Almost three years ago, deep in the pandemic, my kids got me my first Raspberry Pi. That started me on an adventure with Home Assistant, mostly because it was one of the first search results for “what to do with a Raspberry Pi.”

I’m still going, and learned much since then, so it is time for a quick update.

Snippets from my Home Assistant dashboard. From lighting, through heating and even some infrastructure monitoring (Pineapple, Blueberry and Watermelon are the Pis in my house).

What hasn’t changed

At the start, the foray into Home Assistant was tentative, but what attracted me to it then only got more compelling over time. Being able to use devices from many different manufacturers, without getting stuck in any walled gardens, is still hugely important. I don’t have to decide between an Apple home, an Amazon home, a Philips home or a Google home. With Home Assistant, I can make my home out of the best bits from (almost) anyone, using Home Assistant’s integrations.

The software continues to run on the exact same Raspberry Pi 4 the kids gave me, with a few tweaks (see below).

I’m also still using Node-RED for most of my automations. I remain more comfortable in Javascript than YAML (which is what Home Assistant’s native automations are written in), even though Home Assistant has made many improvements to its built-in automation editor since I started.

The Node-RED flow for responding to motion in the master bedroom, taking into account the time of day and the day of the week.

I’m continuing to use the Home Assistant Cloud service, which is provided by the company behind Home Assistant and helps fund its development. It allows me to connect to my server securely while out of the house. It’s possible (easy, even) to set that up yourself, but I want to contribute in some way.

What has changed

The main thing that has changed is what I consider a good automation.

Three years ago, I was excited about things like flashing and colourful lights, getting notifications and making my speakers say things. I was aiming for something more like HAL 9000. My automations were heavy-handed and opinionated, and often annoyed my ever-patient wife and amused the kids, especially when they went wrong.

Nowadays, I’m using a lighter touch. I want the house to do the right thing, the expected thing, at the appropriate time. A good automation should be barely noticeable, almost invisible: The lights should turn on when you enter the room, if it’s dark, and turn off again when they are not needed. The heating should come on when we wake up and turn off when we’re out of the house or go to bed. Deviations, like keeping a light off or on, should be as simple as pressing a conveniently located button.

Most of my automations are not huge improvements over the simple light switch on the wall, and could hardly be described as life changing, but a small number are more substantive: We can tell the house “good night” (via a cheap, home-made voice control gadget) and lots of things happen – some lights turn on, some off, the heating shuts down and the TV turns off. The house then waits for a quiet period (by checking motion sensors) before shutting all the lights and settling into “sleep” mode. The house does a similar “shut down” when it notices that no one is home, and “wakes” up when we return.

The Node-RED flow for shutting down the house at the end of the day.

Local focus

My taste in devices has also become more discerning. At the beginning, I tried loads of things, from many manufacturers, most of which relied on cloud services. I quickly discovered how unreliable that was. I want to turn my lights off, even when my Internet connection is down.

In general, if we’ve learned anything over the years, it’s that relying on the good will of any company is a really bad idea. If you buy an Alexa, a Google Nest, Hive thermostat or some other Internet-connect device, you are essentially renting a future paperweight – features are withdrawn, companies go out of business and cloud services get shut down.

Today I buy and use devices that work locally, without an Internet connection. That means using local protocols, like Zigbee or my WiFi. I avoid anything that requires the cloud. It’s OK to have a proprietary app, but I shouldn’t be required to use it for important features. I haven’t thrown out some of my past mistakes, like my Ring doorbell, but I won’t buy more. I’m gradually transitioning, as I can afford to.

Unfortunately, this approach is antithetical to the interests of most companies, no matter how “not evil” they seem, so it is a bit of a struggle and will probably remain that way. I have hopes about Matter (a very new, local-focused standard for home automation), and will try it when supporting devices become more widely available, but I’m only cautiously optimistic. If it’s possible to get an edge and keep the customer tethered, Google, Apple, Amazon, etc., will find a way.

Reliability

One important shift in my attitude is that I now consider Home Assistant to be very important, even mission-critical, rather than just an experiment or a hobby. If that Pi gives up the ghost, it would be a real pain. I’ve beefed it up a bit: The Home Assistant software now runs on a hard disk (using an internal SSD like this and a new case), but the operating system (Ubuntu) is still running on an SD card that will crap out at the most inconvenient time.

I use the very slightly “advanced” container-based installation option for Home Assistant, which might help a bit. Originally, I used this method to learn docker and docker compose, but I’ve stuck with it because I like the flexibility. I could move services from one Pi to another, swiftly revert to a previous version or (assuming I’ve backed up the data volume recently) recover quickly from a hardware failure.

Still, I’ve only started to put some thought into what will replace this server hardware.

Zigbee

At the start, the main way I managed all of my lights and motion sensors was via Home Assistant’s Philips Hue integration. Under the hood, Philips Hue uses a light-weight, mesh protocol call Zigbee, but with some proprietary features accessible from an app. Almost everything I owned was through this integration. So much so that, other than a couple of buttons and speakers, I could have just used the Hue app to control things. This was Plan B, if Home Assistant didn’t work out.

I’ve now switched to a more generic Zigbee system, using the SkyConnect dongle, and Home Assistant’s own Zigbee Home Automation, because I wanted to start using devices that don’t fit into the Philips Hue worldview, such as thermostats and radiator valves.

The transition wasn’t easy. The SkyConnect was a lot fiddlier to get working and was more susceptible to network interference than the Hue hub (which had its own power supply and a bigger antenna).

I fixed the interference problems by buying a long (five-metre), shielded USB extension and moving the SkyConnect far away from the Raspberry Pi (and its USB3 connected hard drive), the WiFi router and other sources of interference. I had to monitor the airwaves in the house and switch Zigbee channels until I found a quiet place in the radio spectrum. Finally, I bought some Zigbee network extenders (like this one) to increase coverage and reliability.

What I don’t like

Knock on wood, everything in the house seems to be working now, but every big change seems to involve a bit of swearing and moving things around (or channel-changing) until things are stable again. Because of this, I’m becoming reluctant to fiddle – or at least not until I have some time on my hands. I still enjoy the tinkering, but it has to be more carefully considered.

While Home Assistant is well documented compared with most open-source software, if something goes wrong, it can be difficult to debug. There’s a lot of moving parts! The log messages are hard to decipher, and must be inscrutable to a non-programmer. The forums and Discord chats are friendly, but busy. Unless you catch the attention of someone who knows, your pleas for help quickly sink into the ether. Home Assistant users are a small percentage of most manufacturers customer base, so help is spotty as well from official tech support. I’m usually reduced to searching for the error messages in Github issues, hoping for a workaround.

Compared with my day job as a software engineer, I find automating my house a bit like tightrope walking without a net. It’s basically impossible to be test-driven, or at least automated tests. Typically, after a big refactor, it’ll take me a few days to find and fix simple bugs like copy and paste errors.

But, overall, my experience has been very positive. Home Assistant is such a large and diverse project that I’m still discovering new features almost every day.

Regrets

When I look back, there are a few things that I would do differently. I might have leaned more toward smart switches and integrated wall sockets than smart bulbs and dongle-like plugs. And I could have saved myself a lot of heartache by avoiding cloud services. But the flexibility of Home Assistant and its thousands of integrations have meant that I can work through those issues and make do with almost anything – there aren’t many dead ends.

Would I do it all over again? That’s a more complicated question. Nothing quite disturbs your peace as sitting in a dark room, or having a light turn on in the middle of the night, because an automation has gone wrong. Hardware, in general, is also crap and I’ve spent too many hours trying to fix that faulty light strip, dodgy motion sensor or misbehaving sensor.

But I’ve also quite enjoyed learning about it all. It’s been a great hobby, particularly over the pandemic, when I had a lot of time on my hands. I’d do it again, and I’m looking forward to the next few years.

Home automation adventures

Mon, 21 Jun 2021 00:00:00 +0000

For the past six weeks or so I’ve been playing around with home automation. It’s a rather large and complex subject and I’ve only just started, but thought it might be worth writing up what I’ve found so far.

The adventure started when the kids gave me a new Raspberry Pi for my birthday near the beginning of May. I installed Home Assistant on it and, a bit later, Node Red. Then, over the next few weeks, much to my wife’s dismay, I purchased 10 smart light bulbs, a few motion sensors and half a dozen smart plugs – gradually changing the house into a version of HAL. Each time she would come home from work, some wall switch or another would stop working as it did… She’s been very patient!

Home Assistant

The first piece of this, and the cornerstone, is Home Assistant. It is a very active open source project, compatible with (at the time of writing) more than 1,800 devices. Using “integrations”, Home Assistant allows you to tie together incompatible products, from manufacturers who all want to be your sole home automation provider. Rather than going all Apple, or all Google, or all Amazon/Alexa, or having six different apps on your phone, you can you use the best bits and pieces from several different ecosystems. You want to trigger music on two different brands of Wifi speaker, turn on a smart switch in one room when a bulb toggles in another, or tell a Google Home speaker to activate an Apple device? All of this and more are possible.

And it’s not just devices – I can also tie together APIs, such as Google’s Text-to-Speech or The Met Office’s weather API.

Can you do this easily? Well… It’s not too bad, but expect a learning curve. Freedom comes at a cost.

To install Home Assistant on the Pi, I used the docker container installation option. This installs the core of the software in a container, but you miss out on some one-click integration with add-ons. You can still use add-ons, but you have to install them manually, using some docker tricks. This wasn’t an issue for me, since one of my motivations was to learn those docker tricks. Other options include installing a Home Assistant Operating System or even buying a fully configured box.

The software exposes a web dashboard, which you can configure with various buttons and switches, to run your house.

To help fund the open source project, there is also a mobile app, letting you connect to that local dashboard to control Home Assistant while you are out of the house. It’s very optional, but I found it useful enough to subscribe (£5 a month).

Automations

You can get quite far with just the Home Assistant software, turning on and off lights and switches using various events and inputs. These so-called automations are put together using a visual editor and stored using YAML. For example, I bought a starter set for Philips Hue and configured the lights to turn on when the sun is setting. (To be clear, you can do this with the Philips Hue app, but I don’t want several different apps to control my house.)

The wizard-like visual editor steps you though creating a response to some event. The ultimate outcome is a piece of YAML, looking like this, but many users could probably avoid ever seeing the YAML:

As a software engineer, I think that’s pretty readable and would often drop down into Home Assistant’s built-in YAML editor to write this stuff directly. You can get more complicated, with nested OR and AND conditions. However, at some point, YAML starts to get hard to read and maintain, or the wizard interface just hasn’t thought of some wrinkle that you need. At that point, I turned to Node Red.

Node Red

Node Red and Home Assistant are a popular combination. The two together let you write more complex automations, called flows. The flows are assembled using widgets from a palette, connected with lines indicating sequences and branches, in a visual way.

But underneath it all, it’s just NodeJS. The widgets come from NPM packages, one of which lets you write flows that can be triggered from Home Assistant buttons, or respond to its events. It also gives me the ability to drop down to writing code, or write my own widgets, making it nearly infinitely extensible.

Here’s an example: My simple sunset routine (above) worked pretty well, but it has some compromises. I had set the trigger elevation a bit higher than it needs to be (the sun sets behind our neighbouring houses at an elevation of about 5 percent) because it gets very dark in the house on cloudy days well before that trigger point. So, instead, I use a flow that can trigger on any one of several events – the (location-adjusted) sun level or the level of light in various rooms, using sensors to measure lights levels.

This is what the flow looks like in Node Red:

Docker Compose

Both Home Assistant and Node Red run on my Raspberry Pi in docker containers. I bring them both up using a single docker-compose.yml file. This ensures that they are started/restarted/stopped/etc. together and can communicate with each other. Here is my current docker-compose configuration:

version: '3'
services:
 homeassistant:
 container_name: homeassistant
 image: homeassistant/raspberrypi4-homeassistant:stable
 volumes:
 - /home/eamonn/home_assistant:/config
 - /etc/localtime:/etc/localtime:ro
 restart: unless-stopped
 network_mode: host
 nodered:
 container_name: nodered
 image: nodered/node-red
 ports:
 - "1880:1880"
 volumes:
 # Local path where all Node-RED config will be stored.
 - /home/eamonn/home-node-red:/data
 depends_on:
 - homeassistant
 environment:
 TZ: "Europe/London"
 restart: unless-stopped

The /config (for Home Assistant) and /data (for Node Red) point to local directories on the Pi, both of which are backed up to repositories in Github. I can rollback my configuration and flows to a known good state if I do something drastically wrong.

The restart setting (unless-stopped) ensures that the two of them restart when the Pi is rebooted.

Lights

The first thing I tackled was the lights. As mentioned, I started with a Philips Hue hub and two bulbs, but the Philips bulbs are really expensive. They work great, and look good, but I can’t afford to light my whole house with these things.

Fortunately, the Philips Hue is based mostly on a standard protocol and I was able to buy eight more from several different (mostly Chinese) manufacturers for less than half the price. I was able to integrate those into the Philips Hue hub, usually with a click or two in the Philips app. Here’s one example of a compatible Chinese-manufactured bulb.

The knock-offs lack one important feature – the ability to configure what happens when the power goes off and then back on again (like, when the better half tries to turn the wall switch off and on). I guess this isn’t part of the standard. This is a useful feature, but not worth the extra money. I just put the Philips ones where the switch-on behaviour is most desirable.

The bulbs, even the Philips ones, come in several flavours, only some of which support changing colours. I mostly don’t use the colours and just need control over the “temperature” and brightness. I do have a couple of colour-capable bulbs, which are mostly useful for just showing off.

Smart plugs

The next bit was to integrate some smart wall plugs. Our sun room (an extension to the back of the house) has no wall switch – just an overhead light, connected to a mains plug high on one wall, and a floor lamp with four bulbs in the corner. For years, we used a pool cue to turn on the power for the main ceiling light and then had to turn on the lamp’s four bulbs. Every. Time. I wanted to be able to turn on and off the overhead light and the floor lamp at the same time, as if connected to the same switch.

I put a smart bulb in the overhead light, leaving the power on, and then used Node Red to respond to a “turn on” or “turn off” event on this bulb and automatically toggle the floor lamp, which I connected to a smart plug. These are two entirely different products, with different APIs and protocols, but Home Assistant and Node Red are able to hide this complexity.

Another good use of the smart switches was to control the clothes dryer. Our ageing dryer has a dodgy timer, but otherwise works fine. The timer would, very occasionally, fail to turn off the dryer, which is a dangerous fire hazard. What we’ve been doing is to put the clothes in and set a timer on our phones. When the alarm goes off, we’d have to go outside to the shed to check that the dryer stopped.

So, what I did was plug the dryer into one of the smart plugs, create a button in Home Assistant, tied to a flow in Node Red, that ensures the power shuts off again after an hour.

Notice that last bit about “announcing” when the dryer is finished – that’s just a little showing off. I use a Home Assistant integration to connect to two speakers in the house, then use Google Text-to-Speech to tell us when the clothes are dry… Yes, completely unnecessary, but still fun.

I found these plugs work well, using the Tuya integration in Home Assistant.

Motion, time and presence

Another piece of the puzzle was to turn lights and power on and off in response to events. For example, I want the hallway lights to turn on when my wife comes home from a late shift. I want the bathroom light to turn on when we enter, but to control the brightness depending on the time of day (e.g., very dimly in the middle of the night). And I wanted to turn the power off to certain appliances overnight, such as the laptop chargers.

For this, I’ve tried a couple of different approaches. One was to listen for when our phones connect to the home Wifi. This is good enough to turn the lights off if neither of us are home, but it’s slow to react and not good enough to, say, trigger the lights when we enter.

For that, I needed motion sensors. I’ve tried two so far. One is the Shelly Motion. This uses a rechargeable battery and connects to your Wifi directly. It also exposes other sensors – the light and battery level and “tampering” (i.e., shaking the device). I don’t really need the tampering and I only get a light-level event when motion is detected. I’m not that happy with it – it was expensive, a bit fiddly to set up and the events I get from it are intermittent sometimes.

The second I tried is the Philips Hue motion sensor, which despite being the “brand-name” product was actually cheaper and connected to the Hue hub instead of directly to Wifi. The batteries (two AAAs) aren’t rechargeable, but reportedly work for at least a year (and up to two). In addition to motion, these expose light levels (like the Shelly), but also temperature. That’s much more useful than the tampering sensor and means I might be able to hook these up to the heat (or a fan or two) later.

By strategically placing these around the house, and checking the time of day, I can create some sophisticated flows that intelligently turn on lights (at the appropriate brightness level) and turn them off again when no motion is detected for a while.

Music

The music integration was another response to a household frustration (like the broken dryer switch and pool cue light). Several years ago, I bought a couple of Bose SoundTouch speakers. They sound great and I can play the same music, synchronised, though both of them. But the software to control them is truly awful.

With Home Assistant, and the Bose SoundTouch integration, I can just create a couple of buttons that do exactly what I want without having to go anywhere near Bose’s software.

Scenes and routines

Another very nice feature of Home Assistant are “scenes”, where you can set light levels and toggle smart plugs in several rooms to pre-set levels. For example, we have a “watching TV” scene that turns off several lights, and then dims others.

But you can’t run scripts from Home Assistant scenes. This is where Node Red comes in. We have an empty (does nothing) scene in Home Assistant called “Good night”. Node Red is configured to listen to the “scene activation” event from Home Assistant and do various things, such as shut off most lights, turn the TV off, turn on the bathroom lights, etc. Then, dim the lights in stages until they shutoff altogether in 15 minutes.

The pink boxes in this flow are examples of calling saved, reusable routines. It’s an easy way to reduce the complexity of flows, the same way you’d write your own modules or functions in node.

Tying it all together

At the end of the day, however, it’s hard to beat the simplicity of a light switch. Finding your phone or navigating to a web page to turn on the dryer or the lights isn’t very convenient. To address that aspect, I got (yet another) Raspberry Pi, and a compatible touch screen, to create something that hopefully bridges some of that gap.

It’s been a fun adventure, and I’ll probably continue building it out for a while.

A year on

Tue, 16 Mar 2021 00:00:00 +0000

A year ago, I was told to vacate the office and start working from home. I expected it to last several weeks or a few months, at most. I also expected it to be fun. Working from home is fun. Conducting every meeting, every discussion, every personal interaction of any kind, for months on end, through the virtual keyhole that is Zoom is a bit less so.

I’ve adapted surprisingly well, in some respects. I talk to my children more frequently now than I did before the pandemic, for example, and I’ve slowly assembled a very comfortable working environment in one of our spare bedrooms. But I can also count quite a few costs: I’ve lost a year in my seven-year-old granddaughter’s life. I miss being able to hug my kids, something I now realise I did far too infrequently before the pandemic. (Let that be a lesson: Hug while you can.)

And the cost has been too high for society. The number of visitors to the food bank I volunteer at has grown more than five fold in the past year, with long queues of desperate people pushed over the edge, forced to choose between eating and keeping the lights on or paying the rent.

Beyond the obvious changes, though, there has been a more subtle shift, so gradual that I’ve almost missed it. My days flow through my various “roles” more fluidly than before.

I wake up, go for a jog around the neighbourhood, hop on a stand-up meeting on zoom, then pop a load of laundry in. I work for a bit, then have a tea with my wife, maybe in the garden if the weather’s nice. Over lunch, I pop down to the shops and catch up with the neighbour. In the afternoon, I might have to sort something for the food bank, in between bouts of work or meetings. After dinner, I might take another stab at some of the home-automation software I’ve been working on, or have a zoom with the family, or answer that work email.

I am almost certainly more productive at work, and put in similar (or more) hours, but my various roles are braided through my day.

I imagine this is how it’s like to be a farmer. One minute, you are milking the cows, the next you’re fixing the roof on the house, or helping the kids with schoolwork. Life is one big lump, with no sharp lines.

To be sure, that can be a bad thing: no boundaries between life and work can lead to burnout. But I think that’s a danger only if you mix up the priorities. You can see the old assumptions in the post I made on the day (linked above). I took as given the rigid separation between my work-me and me-me, and vowing to keep the boxes separate. But, really, all my roles are important and have their place and time: husband, father, son, software engineer, neighbour, volunteer.

We just have to learn to relax into that. We are a complicated whole, made up of many personal and economic connections. Without that sharp change of location, with the commute between, that web of roles and links has become more obvious. Maybe the stress we feel is that we fail to embrace all those parts of us.

Life would be boring, if it was easy. I’m glad that it’s a year on and I’m still trying to get the hang of this. I’m still learning.

Walking along with suffering

Sun, 06 Dec 2020 00:00:00 +0000

I’ve been volunteering for a couple of years with the Jesuit Refugee Service, which aims to support refugees, asylum seekers and migrants. I’m a volunteer “social visitor.” Before the lockdown, that meant regularly travelling out to the Immigration Detention Centres near Heathrow, Europe’s largest. (There are two centres, next to each other, called Harmondsworth and Colnbrook.)

Several hundred young men, and a very small number of women, are held here indefinitely, often for several weeks or months, while the courts consider their cases. They are usually alone, with no family or friends to visit them. My job is mostly to just talk with them and treat them as people, not cases to be adjudicated.

During the pandemic, these visits continue, but by phone.

The U.K. is a particularly inhospitable place for refugees, sadly. We’re in a period of time where attitudes to outsiders of any kind have hardened and nationalism and xenophobia are on the rise. I feel the work is as important as ever.

This month, I was asked to do the “reflection” for the JRS’s monthly newsletter, on the themes of the Advent season and on its mission of accompanying people through this brutal, heartless process. I’ll share the thought here, as well.

When I first started visiting detainees in Harmondsworth and Colnbrook about two years ago, my aim was "do something" for those at the bottom of our society -- the people jailed indefinitely by an inhumane, degrading immigration and asylum system. I quickly learned how wrong-headed I was.
The people I met needed solicitors who returned their calls, immigration officials who cared, edible food, a safe environment, the company of their loved ones and medical care. I could provide them with none of these. I met them, after being searched and led through several layers of locked doors and guards, with empty hands and pockets. Even the room we shared (back when we could share a room) offered about as much comfort or warmth as a GP’s waiting room. I was not a doer of “something.” I was helpless.

Instead, in this helplessness, I learned what the Jesuit Refugee Service meant by accompaniment. To walk along with people who have suffered, are suffering and still have much more suffering ahead of them is not something you do. It’s something you are. It’s about being, not doing. Just being present. Listening to, and seeing them. Recognising them as fellow human beings. It feels like nothing, because it is. I’m not doing anything. God is. My job is to just be with this person in front of me, who has been through more than I can imagine. God works through that, somehow, through nothing I do or say.

Often, the benefit of visits isn’t evident. I leave someone who appears to be just as depressed as when I arrived. But, sometimes, God does give me a glimpse of the fruits. Farhad (1), who I call regularly at the moment, refers to our conversations (which are almost always about nothing) as “talking therapy”, while another I met says he was able to sleep best after our visits. Usually when I least expect it, God shows me how He is able to use what I could consider to be nothing. Truly, the yoke is easy.

In this season of Advent, we are asked to think about what we bring to Christ in our midst. The asylum seeker, reviled by so many, blamed for so much, is among today’s “least of my brothers”, very much like the homeless, refugee family living in a barn at the centre of the Advent season. I have no gold, frankincense or myrrh, but I have something even more. If you doubt the power of just being present to each other – even just being merely respectful and decent – imagine what the world would look like if we all did that? It would be heaven on earth, with no more individual effort than a smile, attention and love. No amount of gold could come close.

At the start of this journey, I was driven more by anger at the injustice of it all, and the need to fight back. I’m still angered by the hostile environment and culture of disbelief that makes so many lives miserable. But I’m slowly learning that fighting isn’t the answer. Loving is the only response.

[1] not his real name.

Developers Bin

Thu, 05 Nov 2020 00:00:00 +0000

I’m fortunate to work for a team that gives its developers time to play – a day, every other week, that we can use to learn about or try something new. Over the years, a number of useful stuff has come out of those days. Many of them are small utilities that developers have created for themselves – for example, to automate a task. We’ve gathered quite an eclectic collection of these, written in all sorts of languages and frameworks.

Unfortunately, for the most part, these were just being used by the programmer who wrote them, squirrelled away in some directory in their path (~/bin). For example, I’ve written some command-line utilities in Clojure that I think others would find useful, but we’re a Scala and NodeJS shop. I don’t want them to have to install Clojure build tools to try them out.

How do we make it easy for team members to share these utilities with their colleagues?

Private Homebrew Taps

The solution was to create our own, private Homebrew repository that I’ve dubbed our “Developers Bin.” I chose Homebrew because most of us use Macs and already brew install applications. And, although I’m on Ubuntu most of the time, brew is on Linux, as well.

Homebrew “formulas”, if written well, automatically install and configure an application, and ensure that dependencies are installed as well. My colleagues can just brew install <my application> and not know, nor care, that it was written in Go, Elm, Haskell or Clojure. It just runs.

One of the first hurdles you’ll run into, though, is how to create a private repository. Creating a Homebrew repository that others can “tap” is simple – just create a repo on Github of the form <user>/homebrew-<repo-name>. Users can just brew tap <user>/<repo-name> and then be able to brew install anything specified in a Formulas directory. But if that repo must be private, as in our case, that’s a little trickier.

Fortunately, someone’s already written up instructions on how to do that, including the crucial custom_download_strategy.rb that’s needed. This bit of Ruby code used to be part of Homebrew, but the maintainers (probably sensibly) didn’t want to have to maintain code that they don’t use themselves, so it’s been ripped out.

Generating and Updating Formulas

The next hurdle was to make it easy for my colleagues to add formulas to our repository and keep them updated. The strategy I’ve settled on is to create a “template” for each formula that then gets turned into a formula (or updated formula) by a Jenkins job configured to run when code is merged into the main branch.

Here’s an example template (somewhat redacted) that I used for one of my utilities in Clojure. Notice the placeholders for _RELEASE_VERSION_, _RELEASE_URL_ and _RELEASE_SHA__.

require_relative "../custom_download_strategy.rb"

class QuickLogin < Formula
 desc "Simple command-line interface for logging into one of our services."
 homepage "https://github.com/..."
 version "_RELEASE_VERSION_"
 url "_RELEASE_URL_", :using => CustomGitHubPrivateRepositoryReleaseDownloadStrategy
 sha256 "_RELEASE_SHA_"

 bottle :unneeded

 depends_on "openjdk"

 uses_from_macos "ruby" => :build

 def install
 system "./install.sh", prefix
 bin.env_script_all_files libexec/"bin", :JAVA_HOME => "${JAVA_HOME:-#{Formula["openjdk"].opt_prefix}}"
 end

 test do
 # cosmos-login --help
 end
end

It runs my install.sh script, pulls in Openjdk as a dependency and installs the code on the user’s path. I’ve written a few variations of this template, depending on the language or runtime environment of the application. Our developers can just copy one that depends on "node" or depends on "python" or whatever and modify it.

I then had to write a script (run by Jenkins) to do the following:

Create a release in Github.
Build the code – generating a JAR file in this case.
Upload the JAR, and anything else needed, such as the install.sh script, to Github as a tar.gz file, tagged with the release.
Finally, the script checks out our homebrew repo, uses the template to create (or update an existing) formula and checks the changes in again.

That sounds like a lot, and it is the first time you have to create one, but I’ve written one for Clojure, Python, Node and Bash scripts so far. Once I’ve written one, it’s very easy to modify slightly for the next.

When a user runs brew install, for example, or brew upgrade (if they already have the application installed), Homebrew uses the instructions in the formula to download the artifact (using the SHA to do an integrity check) and install or update the application.

For the first few utilities, I had to do some of the heavy lifting – create the scripts and the Jenkins jobs. But once my colleagues started to see our “Developers Bin” as a valuable shared resource, they’ve started adding formulas and templates themselves – one or two a month.

We now have a valuable toolbox of utilities that are easy to install, easy to use and make our day-to-day work easier.

Who Am I?

Sun, 24 May 2020 00:00:00 +0000

The extended family members are getting together again tonight, on the usual unwieldy, international Zoom conference with too many people on it. In an effort to make it less cacophonous, we’re each being asked to come up with something to share – a piece of music, a poem, or whatever – that we liked. We’ll each get three minutes.

I like the idea, so I thought I’d write my thoughts on my “thing” beforehand, so that I can make the best of my three minutes.

My show-and-tell – Richard Dawkins’s 2005 Ted Talk on “Why the universe seems so strange.” – is a bit odd. I have, in the past, really disliked Dawkins – a dogmatic, arrogant man who has lifted the straw man argument to a new art form.

But in this talk, those flaws are less evident. It’s a humble exploration of the limits of knowledge, and our ability to understand a universe that we can only glimpse in a narrow, distorted way. He starts by quoting biologist J.B.S. Haldane:

“My own suspicion is that the universe is not only queerer than we suppose, but queerer than we can suppose.”

The whole thing is about 20 minutes and very well worth a listen. It goes into why we have such a hard time understanding the concepts in the very small (e.g., quantum effects) and very large and fast (the strange quirks of the Theory of Relativity). I’ve linked to it, below.

The part that stuck with me for years, however, was when he described how we are like a wave, comparing us to dunes that retain their shape as the winds march them across the desert. He summarises Steve Grand:

Think of an experience from your childhood, something you remember clearly, something you can see, feel, maybe even smell, as if you were really there. After all, you really were there at the time, weren’t you? How else would you remember it? But here is the bombshell: You weren’t there. Not a single atom that is in your body today was there when that event took place. Matter flows from place to place and momentarily comes together to be you. Whatever you are, therefore, you are not the stuff of which you are made.

The image is extraordinary: atoms, forged in exploding stars over billions of years, temporarily come together to form you, in this moment in time, and then move on. It has stayed with me ever since. I think about it every time I walk along a beach. We’re a wave of organised information, moving through time and space, until in the end we crest and crash into some distant shore.

And that image has led me to explore what it means to be me. What is the mind, what makes me unique and how am I connected to everyone else? It’s led me back to exploring science and reading about physics, something I failed at emphatically in school. I’ve been fascinated by the study of our sense of self, the way we think, the limits of our free will and our sense of time.

Perhaps it would irk Dawkins to learn that his talk also led me back to religion after I had mostly fallen away. All major world religions are remarkable similar on this point – the miraculous nature of us and the present moment. The way we are all connected, literally moving through, breathing and having our being in the fabric of the universe, which they call God.

In the 15 years since I first heard this, I’ve also reevaluated my thinking on Dawkins. The straw man he knocks down is the concept of a magical being with a flowing beard, up in the sky somewhere. He’s smart enough to know that religions really teach nothing of the sort, which has always irked me, but what he’s really critiquing is the small-minded concepts that we’ve made into religions.

If God exists, he is queerer than we suppose, and most definitely queerer than we can suppose. So, in a sense, Dawkins is more truly in touch with that concept than most religious people, and most religions. A modern prophet. Who would have guessed?

Remote First, Pair Programming and Emacs

Sat, 25 Apr 2020 00:00:00 +0000

At my work, we are advocates of pair programming. From an organisational perspective, the practice has several advantages. We’re often more productive working with someone else – we keep each other on task and get stuck less often and for shorter periods of time. More importantly, the resulting code is generally better and more robust, in my experience, especially when combined with test- or behaviour-driven development.

Businesses often object to the practice based on cost. Why pay two programmers to do the work of one? But at the end of the day, you’ll have two developers intimately familiar with the code for future modifications and maintenance – i.e., the bulk of the software’s lifespan. Yes, you spend more in the short term, but for a better payoff in the future. You’re investing in the team, spreading the knowledge more widely.

We’ve had to make some modifications to the practice, because our team hasn’t been co-located for some time. The lockdown has just reinforced those practices. Rather than sitting at the same workstation, passing the keyboard back and forth, we pair remotely in various different ways.

Git Ping Pong

My favourite approach, even when sat next to a colleague, is to ping-pong the work through a shared repository (such as Github). One of us writes a test, which fails, and commits the code. The other, checks out the code and makes the test pass. For a bigger refactoring, I’ll try to use the pomodoro technique. Each of us types for 25 minutes and then commits the work-in-progress. We take a break between sessions.

One big advantage of the ping-pong approach is that both of you must be set up to work on the code – at a minimum, be able to compile and run the tests. Getting to that point is often a big hurdle with complex software. If you can get two programmers to that stage, you’ve again increased the robustness and resilience of the team as a whole. Also, at any given time, our work-in-progress is committed and available, making it easier to do things like mix up pairs, do interim code reviews or pursue experimental paths with the confidence of having a known good state behind us.

Generally, I try to make sure that the person who is “driving” the work isn’t the same as the person typing. So, for example, I’ll be telling the person what test to write, and they will be telling me how to get the test to pass. This keeps us both fully engaged. But usually, once we really get going, the roles blur and we mind-meld into a single programmer with twice the brain power. It’s quite magical when you hit that stride!

Editor Wars

As an Emacs user – and the sole Emacs user on my team – I face a couple of unique challenges. My pair for the day will often be intimidated by the spartan interface and unfamiliar keyboard shortcuts. There is one Vim user among my colleagues, but most use either IntelliJ IDEA or VS Code. The VS Code users will also sometimes use the Live Share feature, which I agree is a nice way to share a session.

This can be a source of friction. Two programmers each using their own editor doesn’t fit the traditional definition of pair programming, and people do get quite heated on this point of doctrine. Why bother if you don’t do it “right”, they ask? Some programmers also can’t understand why I won’t just use IntelliJ or VS Code. To them, I’m being pig-headed or stubborn, but I feel as crippled in those editors as they probably do in Emacs or Vim.

Emacs has a couple of other disadvantages for pairing remotely, as well. It doesn’t show line numbers by default, for example. You can’t easily point to a section of code verbally (“Now, change line 23 so that it uses the variable on line 6”). I also like to have as much of my code on the screen at once. I turn off all the menus and other distractions and set the font size a bit low. That’s not as nice an experience for my remote partner.

Visual Niceties

To make the experience of working with me as pleasant as possible, I’ve recently tweaked my configuration. I have added a couple of items to my top-level menu (bound to Ctrl-z) to let me turn line numbers on and off and adjust the font size globally. Neither involved writing any new code – the functionality was already in Emacs. Here’s how I set up my main menu, for which I use the hydra package:

(global-set-key
 (kbd "C-z")
 (defhydra hydra-global-menu (:color red :hint nil)
 "
^Display^ ^Buffers^ ^Actions^
^^^^^^^^^-----------------------------------------------------
_g_: zoom-in _d_: close all buffers _u_: update all packages
_s_: zoom-out _o_: open buffer on desktop _l_: display line numbers

_q_: quit this menu _r_: restart emacs
"
 ("g" default-text-scale-increase)
 ("s" default-text-scale-decrease)
 ("d" kill-all-buffers)
 ("l" global-display-line-numbers-mode)
 ("r" stop-and-restart-emacs)
 ("u" eds-straight-pull-or-prune)
 ("o" eds/open-buffer-on-desktop)
 ("q" nil)))

The above makes a colour-coded menu appear at the bottom of my screen whenever I type C-z. The first two items (zoom-in and zoom-out) and the one to display line numbers were added specifically for remote working. I can quickly set up a font size that works for both of us and turn on line numbers with just a couple of keystrokes. I’ve also been experimenting with different themes, to make the contrast better and easier to read on a small screen over a remote link. My current is Leuven. For the pomodoro technique, I use pomidor, which displays a nice, big colour coded history of the time we’ve spent in each session.

If you’re interested in some of the other items in that menu, my whole configuration is on Github.

Reducing Friction

I’m not going to convince any of my colleagues to pick up Emacs – I’ve long ago given up on religious wars – but we can meet in the middle, both of us using the environment and editor that makes us as productive as possible.

In the end, the aim is to get to that mind-meld stage – getting into the flow. Anything that reduces the friction helps. If we’re both comfortable in our working environment, can see and talk to each other easily and can view each other’s screens, we’re well on our way.

A Sustainable Pace

Fri, 10 Apr 2020 00:00:00 +0000

It’s now been a few weeks since many of us in the U.K. have gone into lockdown. We’re spending a lot more time at home and our social calendar has suddenly freed up, completely, for the foreseeable future. So why are we so tired?

When we first moved to the U.K. in 1998 from the Boston, Massachusetts area, my very smart Aunt Kathleen gave my wife and I an abbreviated version of the course she gave to corporate managers who were about to go on overseas assignments. We found this training to be extremely helpful in the first few months in London. It was a guide for handling life-upturning change, precisely what we’re going through again right now, and is one of the reasons I was expecting the tiredness.

So it’s a useful exercise to revisit some of that wisdom and see what it could teach us about this period in our lives.

Mood swings

One unfortunate side-effect of our radical change in location in 1998 was rapid swings of mood. We’d go from, “This is great – it’ll be like vacationing all the time!” to “What the hell have we done with our life?” from one day to the next.

Aunt Kathleen showed us a graph similar to the one above. She said to expect times where things looked unrealistically wonderful, followed by bleak periods. These swings will become less sharp and more gradual over time, she said.

I like to think of it as a stone thrown into a pond. The waves are high and frequent closer to the event, but become less pronounced the further you move away (in time and space) from the disruption.

I can’t say I had many periods of euphoria about lockdown, but I’d also be lying if I said I didn’t look forward to working from home for an extended period. Quickly, however, the reality of using my home as my place of work sunk in. The constant video conferencing, the poorer office equipment and less downtime and separation from work got old quickly. So has separation from my family – all five of my children and my sister in London used to get together very frequently. Now we see each other only on video. My enthusiasm for it all as waned.

As the weeks drag on, though, I’m getting used to things. I still have swings in mood from day to day, but there becoming less sharp, more just the normal ups and downs. I’m beginning to appreciate the upsides – I can wander out to my garden for breaks, and my wife and I can spend more time together. It’ll be OK. The waters are calming.

Why are we tired

Another aspect of radical change that my aunt warned us of was exhaustion. You would think moving from New England to Old England wouldn’t be that big of a jump – and it wasn’t. We spoke more or less the same language, for example.

But there was a constant underlying current of effort – things were slightly different everywhere we went. The foods we were used to buying in the supermarket weren’t there, so we had to find substitutes. The schools were organised completely differently, and we ended up with our children split between two primary schools. Driving on the left, figuring out the Tube, understanding some of the accents, and on and on.

None of the individual changes were overwhelming. They were just minor adjustments, for the most part. But a whole day of learning new things took its toll. We were exhausted by the time we went to bed each night.

Similarly, the cognitive load of constant remote working and socialising is draining. We don’t notice each adjustment, each individual tweak to our life, yet they suck the energy out of us.

We have to learn to take it easy on ourselves. We simply cannot keep going at the same pace that we were in the before-time (or BC, Before Coronavirus, as it coming to be known). And that’s OK. We’re not just working from home, we’re trying to keep going through some truly historic upheavals. We need to be kind to ourselves – go to bed earlier, rest up. Just because we’re not going out, for example, doesn’t mean we have to fill all that time with learning a new language or new instrument.

Things that help

The final part of my aunt’s training was to suggest things that can help get over the worst effects of change. For example, she suggested drawing a rough map of the neighbourhood (the location of the local shops, schools and church) and sticking it on the fridge, to help us make the mental adjustment to our new environment. A lot of what she suggested, however, was specific to relocating and not as relevant to what we’re going through now. So what would help?

One of the big surprises about the lockdown is what I’m missing. It’s not what I would have expected. Take commuting – why would I miss that? I ride a motorcycle to work. Operating 250 kg (550 pounds) of metal, on two wheels, in London traffic – especially the way we do it around here, using the space between lanes of traffic – is an activity that fully engages the mind and body. If I want to survive the experience, I have to be awake and aware. I don’t have the mental space to be thinking too much about what I want to accomplish today (on the way to work) or regretting what I didn’t (on the way home). That commute was a very useful boundary period – a total separating wall – between work time and home time.

It also was a very useful activity for my mental health. I practice mindfulness meditation, have for years and still do it every morning. Motorcycle riding is very different, but it has similar benefits. The first chapter of Zen and the Art of Motorcycle Maintenance describes it well. It’s nothing like driving a car. You are in the environment, the weather literally (and often, here) streams off you and drips down your back. You’re not just watching the world go by like it’s a TV channel.

I miss those two periods of my day when I was brought out of my thoughts, into the stark reality of how I’m going to safely thread my way through that black cab to my left, and the lorry to my right. I’m finding that I need substitutes during this period. They don’t have to be death-defying, but they need to be engaging in a way that is completely different than my work.

Writing this blog is one such activity. It wouldn’t have helped at all for when I was a reporter or editor, but it is quite a break from working on software. Another activity is gardening and general DIY around the house. I’m absolutely rubbish at that kind of work, so it does occupy me more than most people. I’m also find that I need more play. My wife and I have started playing pool more frequently, for example.

I’m sure as this lockdown drags on that we’ll get better and better at coping, and even thriving. I’m actually looking forward to the adventure, good and bad. That’s what got us out of that transition period when we moved here. We started looking forward more than backwards. Instead of comparing our new life with the old, we just started noticing and appreciating our life, as it is.

Then and now, we’re taking things one day at a time, at a sustainable pace.

A well-honed tool

Sun, 05 Apr 2020 00:00:00 +0000

I took a few minutes last night to add a small feature to Emacs, saving me from some repetitive steps when I’m starting a blog post (like this one). It was a tiny bit of code and nothing exceptional in itself, but it reminded me of why I choose to use an editor that’s nearly as old as I am. It is designed for this kind of organic, incremental growth.

I often find it difficult to explain to my younger colleagues why you would use Emacs or Vim (another vintage editor) over modern tools such as VS Code or Intellij. The old editors are harder to learn. You can’t just sit down at one and be productive in minutes. It takes some effort, and usually at least a several weeks of it, to get the hang of them.

But Emacs and Vim have been shaped, balanced, sharpened and smoothed over by decades of usage by hundreds of thousands of programmers, each trying to get through their day as efficiently and fuss-free as possible. In the right hands, they move lines, shift paragraphs and fling code better and faster than anything out there.

They’ve evolved so much because it is trivial to extend them, to change even the most fundamental features. You just do it as a matter of course, to save a few keystrokes here and there. And many of these incremental modifications have been shared, one programmer to another, almost as long as there have been programmers.

Today’s bit of magic took a lot less time to write than this blog post, but it turns a half-dozen steps into one. It opens this site’s Github project, creates a new branch (using a plugin called Magit), creates a new file and inserts some boilerplate configuration. With one keystroke, my cursor is blinking, in the right file, in a new branch, configured and ready for me to start typing the post.

(defun eds/insert-skeleton-blog-post ()
 "Insert a basic skeleton for a blog post."
 (goto-char (point-min))
 (insert
 "{:title \"new post\"\n :layout :post\n :tags []}\n\n"))

(require 'init-git) ;; for magit
(defun eds/start-blog-post ()
 "Create a new post with today's date and a new branch"
 (interactive)
 (let* ((branch (format-time-string "%Y-%m-%d"))
 (filename (concat "~/git/eamonnsullivan.co.uk/content/md/posts/" branch "-new-post.md")))
 (find-file filename)
 (eds/insert-skeleton-blog-post)
 (save-buffer)
 (magit-branch-create branch "master")
 (magit-checkout branch)))

It’s not great code. There’s no error handling (it’ll just blow up if there’s already a branch with the same name) and the file names, title and configuration are hard-coded, but it’s good enough to start with. It’ll continue to evolved, as my needs change. I might prompt for the title or tags, for example.

Of course, you can extend Intellij and VS Code, too (VS Code has a good API, for example). Users don’t focus on tinkering as much, mostly because these tools probably already have the features they want. I tried Intellij for a long time when I first started at my current job a few years ago, because I was new to the Java Virtual Machine world (this was for Scala) and absolutely everyone else used it. It has just about every feature you would want, and it mostly just works.

What drew me back and motivated me to put effort into adding Scala features to Emacs is that I didn’t like all the black-box magic in Intellij. By adding the features I needed to Emacs, I was learning how the language and the build tools work, from the ground up. By shaping Emacs to fit my workflow, I believe I was becoming a better Scala programmer.

My Emacs configuration is nearing its third decade. I started using it to write HTML and, once I got the hang of it, started using it for my day job as a journalist. I found myself enjoying its sparse, no-nonsense, no-distractions interface. And piece by piece, one tiny snippet at a time, it has become almost an extension of my brain. I wouldn’t trade it for anything.

I love Zoom, but...

Tue, 31 Mar 2020 00:00:00 +0000

I’m trying to get one of the charities I work with on Zoom before our next scheduled meeting on Monday. This is at least the third time now that I’ve gone through the pain of getting a new group of people on the app, and it’s a very painful process. Every. Single. Time.

What kind of problems, you ask? Let me count the ways.

I can’t see (or hear) you..

One of the first problems new users run into is how to turn on the camera and microphone. For many good reasons, an application must be given permission to access your camera and microphone – you don’t want apps to be able to surreptitiously film you or eavesdrop. But is it really necessary to hide these controls the very first time a user runs your application?

First of all, we have to ask if the user connected via a laptop or a phone, and because they are muted and have their camera off, they can’t tell you. If it’s the former, they have to move their mouse to the bottom of the screen to reveal these controls. If the latter, they have to touch the screen. Neither action is obvious.

And the first questions to the new user are very confusing:

What does “Call using Internet Audio” mean, exactly? I dunno either, and I make a living at this kind of thing. (This is the one you want to pick, by the way…). Even people who consider themselves technically savvy will be tempted by the other options (such as to “Dial in”), but those are almost always wrong.

Where are you?

In most apps made in the last several years, if you want to share something that you’re looking at (like a Web address or photo), you click on a button usually labelled “share”. Often it has an arrow pointing up. In fact, just like this one:

You would expect this button to provide some way for you to share this meeting with someone else, so that they can join. But you would be wrong. This button is only used to share your desktop with the people who have already found your meeting.

How you actually share the address of this meeting varies (as always) between the desktop and the phone apps. On my desktop, I have to find this tiny little ‘i’ in a circle on the top left of my screen, like this one:

It’s really, really tiny and took me forever to find it. But when I did and decided to try clicking on it (not immediately obvious, in the least), I finally found what I was looking for:

That’s just the beginning of the issues. Most people will also have to wrestle with their device settings (completely different on the Mac and Windows), for example.

Practice makes perfect

It’s so painful getting started on Zoom, it’s a wonder anyone uses it. We persevere because it is, by far, the best solution out there right now, with the best video quality and with the least restrictions on the free version.

The only solution I’ve found for Zoom’s innumerable UX issues is practice, and lots of it. Our charity has several days until our first remote-first meeting, so I’m holding a surgery each night. I’m coaxing our members, one-by-one, around all of the obstacles and over all of the hurdles.

By the time Monday comes around, we’ll still probably spend the first 20-30 minutes getting everyone connected. Once we connect, however, we’ll be able to carry on our work, which is needed more than ever now. And that’s what matters.

What will change?

Sun, 29 Mar 2020 00:00:00 +0000

Here’s a game to while away some of those long evenings in lockdown: Predict how the pandemic will change the world. What do you hope, or fear, will be different about our everyday life in a year or two?

It’s likely that very little will change. We’re pretty good at getting used to anything and carrying on as before. Eventually, a vaccine, wide-spread immunity or an effective treatment will relegate COVID-19 to just another hazard out there that might kill us, but probably won’t. The roads will fill with traffic again, commuters will pack the Tube and we’ll all just go back to business as usual. It’s hard to knock us off track, especially well-grooved ones.

But this pandemic is big enough that there will be changes in the way we see the world, some of them long lasting.

Cottage Industry

Let’s take an easy one to start: Many businesses will be discovering that they can operate from their employees’ kitchen tables or spare bedrooms. It isn’t easy at first, but as the weeks drag on, we will start getting the hang of this. Stuff will get done.

It’ll be hard to put that genie back in the bottle. The staff will grow to like spending time with their families instead of commuting. The companies will find it hard to justify vast floor space at prestigious addresses.

Cottage industry has some downsides, of course. The biggest danger with working from home is over-work (not slacking off as most managers imagine), so employees will need to learn to keep a strict separation between home and work life, without the benefit of time and distance.

Also, ergonomic chairs that you can sit in for five or six hours at a time without killing your back cost a freaking fortune.

A slightly less obvious side effect of this will, I hope, be a resurgence of local high streets. All these people working at home will eventually be allowed outside again. They will be looking for things to do with that not-commuting time. They’ll want to work and hang out in coffee shops, for a change of pace from the months of solitary confinement. They’ll want to eat or buy lunch, and take yoga classes. They’ll want a drink with friends who they haven’t seen in person since the before-time. And they’ll want to walk to these places, like they used to do at the office. Our local high streets, which have been dying for years, will get a new boost from a more dispersed workforce – knowledge workers on break.

Priorities

Finally, how about a long overdue big shake-up of what patriotism means? To be patriotic for the last century or two has often been tied up with a near worship of the armed forces – all those marches, drumming and wreath laying. But if this pandemic has shown us anything it’s that our health services are as important to our security – and, really, much more so – than even the military. We’re seeing men and women putting their lives on the line in an entirely different way. Risking death to hold someone’s hand so that they won’t die alone. They are the new heroes.

I don’t necessarily think we need a medical Cenotaph (although maybe we should), just recognition and financial support. How many ICU beds can we build and staff if we build one less aircraft carrier? How many medical school placements equal one Trident nuclear submarine?

That’s my go. Your turn.

A useful tweak

Mon, 23 Mar 2020 00:00:00 +0000

On this site, I want URLs like https://my-site.com/my-page/ instead of https://my-site.com/my-page.html. It’s a subtle difference, but the addresses just look a bit neater, a bit more professional, less late-1990s without a filename at the end.

Cryogen, which I use to generate this site, can do this and does this by default. I had been overriding this behaviour because it wasn’t working with Amazon’s S3 service, but apparently I was just doing it wrong and Amazon’s S3 should be able to handle URLs like this.

My issue now was that I had already deployed the site and had to fix it on production. Here’s how I did it, albeit with a few minutes of downtime:

Get the correct endpoint

I copied the entire endpoint (including the “http://”) from the properties of the bucket in S3 where my content is stored.

Edit the origin

In CloudFront, find the distribution for the static website, select it and click “Distribution Settings”, then “Origins and Origin Groups” and paste in the URL you copied from the endpoint into “Origin Domain Name”, then save it.

Change the site’s configuration and publishing script

The next couple of steps are a bit tricky, because we’re going to change the way the site is generated pretty radically, push those changes and then invalidate the CloudFront distribution (removing all of the files and directories from the global cache). This will mean that the site is broken for a few minutes (between the push and invalidate steps). But, in my case, that is fine. I think I’m the only reader of this site! There probably is a better way.

The configuration change in Cryogen was to change the :clean-urls setting to :trailing-slash from :dirty, which was the default that I overrode at the beginning.

Also, in my .github/workflows/publish.yml file, I needed to add an argument to the S3 sync job: --acl public-read. This is actually a recommended setting, but I didn’t need it previously. With this new configuration, where it creates a sub-directory for each post and page, it became necessary.

When I commit these two changes, and push to Github, the site is generated automatically and synced to S3.

And, at this point, the site was badly broken to anyone looking at it. The front page was OK, but none of the links worked. The cache didn’t match up at all with reality.

Invalidate the CloudFront distribution

To fix that, I needed to clear the cache in CloudFront.

That was it. After a few minutes (this is still a small site), the distribution was cleared and the site began to work with better, cleaner URLs.

Reasons to be cheerful

Sun, 22 Mar 2020 00:00:00 +0000

Reason #1: My immediate family (my kids, my sisters and my parents) have been using videoconferencing to keep in contact regularly. I think I’ve seen some of my kids more often because of this. A new tool we’re using is Zoom. It allows several of us to have a virtual get-together. I use Zoom at work, which is where I first heard of it, but hadn’t realised how useful the free version is. The free version limits meetings to 40 minutes, after which you just need to create another meeting. But if you know someone (i.e., me) with a paid account, you can have them host the meetings and go over that limit. It’s easy enough to set up that I might even introduce my parents to the idea, gently.

Reason #2: The food bank I belong to has had to curtail a lot of its activities, such as serving tea and coffee, but we’re still open. We were quite worried about yesterday’s session – the first we’ve held since the near lock-down conditions have been imposed on London. We expected large, possibly aggressive crowds because the local supermarket shelves are empty. But that didn’t happen at all. We were busy, unsurprisingly, but everyone was lovely and helpful. I’m going to miss sitting down and having good conversations with many of our visitors, but we can still exchange a few kind words and we’re staying open.

Reason #3: I thought this tweet thread was uplifting. It turns out business at supermarkets are only up about 10% or so over the normal. The businesses are so efficient in just-in-time delivery and stock, that this is causing quite a bit of mayhem. People really aren’t hoarding. They’re just working from home and going out less, buying a bit more each week. It isn’t just mobs of idiots. The supermarkets will adjust. It will be OK.

Reason #4: I’m getting stuff done. Last week was busy at work, but we managed to shift a lot of code. The long evenings at home is also helping finally dig into things I’ve been meaning to do, like creating this web site, and the charity one I’ve promised for years. Yeah, it’s a bit harder doing everything over videoconferencing, but it’s not that much harder. We’re getting good at this.

Reason #5: The weather is improving. It’s beautiful out today and it’s forecast to get even better next week. I even mowed the lawn today, for the first time this year. The apocalypse will be a lot more enjoyable if the sun is shining.

It's getting real

Thu, 19 Mar 2020 00:00:00 +0000

If I had to put a word on what I’m feeling about all of this, it would be “grief.” In a matter of a few short weeks, just about everything we were looking forward to is gone. Holidays, family get-togethers, celebrations – all put on hold, as everything we once thought stable becomes uncertain. So many people will lose their jobs, their businesses, and even their lives.

I see people going through the stages: The roads in my neighbourhood have been full of the hard-core deniers, off to work as normal. When the schools close next week, even those last hold-outs won’t be able to ignore the pandemic any longer. The next stage is anger. We’re beginning to see that already as people queue for hours for a bag of dried pasta. Bargaining? What is meant by “isolation”, exactly? How far can I push that?

Depression isn’t far behind. The enthusiasm among my colleagues is waning. Day after day of constant video conferences is getting old. Even introverted software developers realise they can’t live on Internet alone.

What will acceptance look like? I don’t doubt that we will get to that last stage. We will learn to cope, but how? What will we learn during this period of forced separation? How will this change our societies, economies or neighbourhoods? We will soon see.

Home Sweet Work

Mon, 16 Mar 2020 00:00:00 +0000

As of today, I’ve joined the hordes told to work from home, for an indefinite and probably lengthy period. It felt weird walking out of the office this evening. I may not see my colleagues in person again for weeks or months, people I’ve spent the bulk of my waking hours with for the last four years.

We are living through an extraordinary, historic time. This compares with 9/11 in momentousness, but is unfolding in slow motion, as farce. Imagine if Coronavirus was as deadly as Ebola? We’ve proven ourselves unready. I hope we learn something.

So, a few resolutions to keep me sane over the next several weeks:

My home office is “work”. Outside of my home office is “home”. The two shall be kept separate.
I’ll get up, shower, run and otherwise prepare for work like I’m going to the office. No working in my PJs.
Get as much social time as feasible, in whatever way possible. We’re going to make liberal use of Facetime, WhatsApp, Slack and Zoom. This is what social media is really made for, not just something to stare mindlessly at while standing in queues.
I’m not going to pull up the drawbridge. I’ll be at the (already very busy) food bank each Saturday, and checking in on the neighbours, as long as I’m healthy.

Keep calm. Carry on.

Well, that was almost too easy.

Thu, 12 Mar 2020 00:00:00 +0000

As described in my previous post, I wanted to automate publishing to this site via Github Actions. As a Jenkins user at work, I expected this to take some time, requiring reading reams of documentation and writing fiddly scripts that don’t work the first 56 times you try them.

In the end it took me about 30 minutes, and hardly any RTFMing.

Publishing workflow

The process I wanted to automate is basically two steps, but it requires a bit of setup. I have to have a Java environment and a Clojure build tool called Leiningen. I also need the Amazon Web Service command-line interface and some environmental variables (the S3 bucket to use, the AWS access key and secret, etc.).

Setup secrets

The first step I did was create a new AWS user, with limited access, to do the uploading. I create a user that is for script-use only (it can’t log into the console). It also needed access to my S3 bucket, but nothing else.

I then generated a key and secret for this user, and stored that on my repository using the Secrets part of the repository’s settings.

Investigate existing workflows

Github has a marketplace of Actions. These are actions that people have put together and shared publicly. In addition to the standard Github ones (for checking out master and setting up Java) needed one for installing and configuring Leiningen and another for syncing my generated pages with the S3 bucket.

For syncing with S3, S3 Sync comes highly rated. For setting up my Clojure tools, I used Setup Clojure.

Combine everything in a workflow

The final step is to create a new directory and file in my repository: .github/workflows/publish.yml.

The first couple of attempts had mistakes in it, but fortunately, I did the work on a branch and then created a pull request. Github will automatically syntax-check any workflows in the pull request. The first time, I mixed the uses and runs keys in the same steps, which isn’t allowed. The second time I tried it, it caught a simple syntax error (a missing full colon).

Once the check passed, I merged and then fixed a typo to check if it worked. And it did. The first time. I’m still catching my breath!

Here’s the final script:

name: Upload Website

on:
 push:
 branches:
 - master

jobs:
 deploy:
 runs-on: ubuntu-latest
 steps:
 - name: Checkout Master
 uses: actions/checkout@master
 - name: Prepare Java
 uses: actions/setup-java@v1
 with:
 java-version: 1.8
 - name: Prepare Clojure
 uses: DeLaGuardo/setup-clojure@master
 with:
 lein: 2.9.2
 - name: Build
 run: lein run
 - name: publish
 uses: jakejarvis/s3-sync-action@master
 with:
 args: --follow-symlinks --delete
 env:
 AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 AWS_REGION: 'eu-west-1'
 SOURCE_DIR: 'public'

Notice that the only part that needed to be explicitly run was the lein build command (lein run), everything else just happens. And the steps are nice and clear. The labels I gave to the steps (name:...) show up in the actions, and I can drill down into each step and see the logs.

Overall, this was very easy and impressive. The site-generation and the continuous integration pipeline is beginning to look like a usable MVP (minimum viable product) for the charity site.

Creating this site

Tue, 10 Mar 2020 00:00:00 +0000

I’ve had this domain for a few years, letting Amazon’s Route 53 auto-renew it each year without actually doing anything with it. I’ve made a few starts, but hubris would always get the better of me. I’d start writing something overly complicated like, say, a single-page app in React (or whatever the framework du jour was at the time). I kept rediscovering the obvious: I’m too busy, and I’m rubbish at estimating what I can accomplish in my free time. I’d get 80 percent there, life would crowd in, the work would languish until some other shiny new technology caught my eye and I’d start again. A cycle of failure.

When a mostly free Sunday afternoon presented itself last week, I remembered with some chagrin that I had promised one of the charities I work with a web site. I promised this back in 2017. You already know what happened with that (see “cycle of failure”) and no web site has materialised. So, I really do need to get going on this, and I figured the best way to get started on the charity web site is to practice on one of my own.

Requirements

That leads me to my first requirement: It has to be simple. I just don’t have the time for complicated. Yes, static web sites are very late 1990s/early 2000s, but I don’t really have the need yet for anything more fancy.

Other requirements: It has to be cheap. I don’t want to be spending even tens of pounds a month for something I’m likely to stop updating after a while. (Maybe not, we’ll see.) I also want the site served from HTTPS with a globally valid SSL certificate because, well, it’s the 21st century and browsers are starting to get bitchy about “not secure” web sites. Finally, I’d like the site to be responsive and be usable from a small iPhone screen – because, again, 21st century and mobile-first is becoming mobile-only.

Note a missing requirement: It doesn’t have to look super spiffy. If I get the urge, I might tinker with the CSS, but if the pages are legible, and the design doesn’t make my eyes bleed, then that’s good enough for now.

All of these requirements apply equally well to the charity site. If this works, I’ll have a pattern to follow.

Cryogen

I had recently bookmarked Cryogen, a static website generator, as something to investigate. It creates a very simple blog site with posts (like this one) and pages (like About Me). By default, pages and posts are written in markdown and then compiled into HTML that can just be uploaded directly. It’s also written in Clojure, a language that I’ve been focusing on lately, so that was another plus.

The Readme explains how to get up and running. Basically, all I did was delete the sample pages and posts and write a couple of my own.

However, as I discovered a bit later, I also needed to make some configuration adjustments due to my chosen deployment platform: Amazon Web Services. The configuration you need still isn’t documented, so I really should put in a pull request. I’ll add that to the to-do pile.

Steps

Create a repository on Github for the site.
Use the leiningen Cryogen template to generate the site skeleton (lein new cryogen eamonnsullivan.co.uk)
Clear out the sample pages and write an “about” page, an initial post and edit the configuration.
Run the site locally (lein ring server) and verify that it all works.

Deployment

Amazon Web Services has a good guide for getting a static web site installed in S3. I followed that and ended up (once I had configured Cryogen to generate all of its files at the top level) with a public bucket on S3 that’s accessible via HTTP.

I then followed this article on Medium to configure Cloudfront (a global caching layer) and an SSL certificate. The final bit of this document describes how to configure the bucket in S3 to only allow access via Cloudfront.

Once that was all configured, I can publish changes to the web site using the AWS command-line interface, like so (assuming I’ve nicknamed my account as “home” in AWS’s configuration):

lein run
aws s3 cp ./public/ s3://eamonnsullivan.co.uk/ --recursive --profile home

Next steps

My next challenge is to automate updates as much as possible. I’ve been meaning to learn Github Actions and this is a good opportunity. Ultimately, I’d like for the web site to automatically publish each time I merge to master.

Wish me luck!

Introduction

Sun, 08 Mar 2020 00:00:00 +0000

This is my own web site. There’s not much here yet. I’m hoping to document how I get this online.

The title comes from a prayer I read once. In any given moment, there is one thing necessary for us to do that will brighten the world. We don’t need to solve the bigger picture, we just need leave the world and the people around us a little better off, more compassionate, kinder.

What’s that one necessary thing right now?