In the talk and writeup I describe how we can change the Active Record error message format from the default of "%{attribute} %{message}" and drop the attribute prefix, giving us more flexibility in how we phrase our error messages. The downside to this is that it results in the default Rails error messages being output as incomplete sentences. It’s since occurred to me that it’s possible to avoid this by replicating the default validation error messages in our app’s locale file with the attribute prefix as part of the message, rather than the format!

So we end up with a locale file that looks something like this:

# app/config/locales/en.yml
en:
  errors:
    # Our overridden message format without the attribute prefix
    format: "%{message}"

    # The Rails default messages, but with the attribute prefix included
    messages:
      inclusion: "%{attribute} is not included in the list"
      exclusion: "%{attribute} is reserved"
      invalid: "%{attribute} is invalid"
      confirmation: "%{attribute} doesn't match %{attribute}"
      accepted: "%{attribute} must be accepted"
      empty: "%{attribute} can't be empty"
      blank: "%{attribute} can't be blank"
      present: "%{attribute} must be blank"
      too_long:
        one: "%{attribute} is too long (maximum is 1 character)"
        other: "%{attribute} is too long (maximum is %{count} characters)"
      password_too_long: "%{attribute} is too long"
      too_short:
        one: "%{attribute} is too short (minimum is 1 character)"
        other: "%{attribute} is too short (minimum is %{count} characters)"
      wrong_length:
        one: "%{attribute} is the wrong length (should be 1 character)"
        other: "%{attribute} is the wrong length (should be %{count} characters)"
      not_a_number: "%{attribute} is not a number"
      not_an_integer: "%{attribute} must be an integer"
      greater_than: "%{attribute} must be greater than %{count}"
      greater_than_or_equal_to: "%{attribute} must be greater than or equal to %{count}"
      equal_to: "%{attribute} must be equal to %{count}"
      less_than: "%{attribute} must be less than %{count}"
      less_than_or_equal_to: "%{attribute} must be less than or equal to %{count}"
      other_than: "%{attribute} must be other than %{count}"
      in: "%{attribute} must be in %{count}"
      odd: "%{attribute} must be odd"
      even: "%{attribute} must be even"

      # Our bespoke error message definitions
      models:
        album:
          title:
            blank: "Enter a title for the album"

With that in place, we’re free to define bespoke validation errors for specific attributes and models without the attribute prefix, whilst also preserving the default error messages as a fallback.

I’m going to share ten things I think are interesting in i18n, and explain how we can take advantage of them in our single language app code. But first a brief primer on i18n…

What actually is i18n?

i18n is shorthand for Internationalisation When we’re talking about i18n, it’s useful to understand the difference between Internationalisation and Localisation:

• Internationalisation — the process of abstracting content and other locale-specific things away from application code itself
• Localisation — the process of adapting software to support different languages and regions (maybe using i18n)

The take away here is that although i18n is primarily a framework for adapting applications to support other languages, the core idea of separating content from applications code has useful properties that we can take advantage of it to improve our Rails applications, even if they only support a single language.

i18n in Rails

Rails ships with the i18n gem, and there are two main components:

• locale files where textual content is organised and stored per-language (normally as YAML)
• two helper methods:
  • I18n.translate for outputting text content from the locale files
  • I18n.localize for localising dates and times

Rails aliases these helpers in views and helper modules to the convenient shorthands of t and l respectively.

So the most basic usage of i18n would look something like this:

# app/config/locales/en.yml
en:
  home_title: Hello world
  home_body: How are you doing?

<!-- app/views/home/index.html.erb —>

<h1><%= t 'home_title' %></h1>
<p><%= t 'home_body' %></p>

<time datetime="<%= @published_at.iso8601 %>">
  <%= l @published_at %>
</time>

Primer over! On to the ten things…

Thing 1: You are already using it

Even if you’re not localising your app you’re almost certainly still using i18n. That’s because it’s baked into the very fabric of the framework.

For example Active Record uses i18n under the hood to generate error messages for its validations. It does so using a YAML file that ship with Active Model:

# rails/activemodel/lib/active_model/locale/en.yml
en:
  errors:
    # The default format to use in full error messages.
    format: "%{attribute} %{message}"

    messages:
      model_invalid: "Validation failed: %{errors}"
      inclusion: "is not included in the list"
      exclusion: "is reserved"
      invalid: "is invalid"
      confirmation: "doesn't match %{attribute}"
      accepted: "must be accepted"
      empty: "can't be empty"
      blank: "can't be blank"
      ...

Whenever you see something like “Title cannot be blank”, this is the i18n plumbing that was used to construct it.

Thing 2: Default error message can be overridden

Rails i18n makes it easy to override these default errors messages. Just replicate the same key structure in your application’s locale file with your chosen phrasing:

# app/config/locales/en.yml
en:
  errors:
    messages:
      # Overrides the default Rails message for presence validations
      blank: "must be provided"

Rails will check your application’s locale file first, falling back to the defaults only if the message isn’t defined there.

Model and attribute specific overrides

As well as overriding this global phrasing, it’s also possible to override the error messages for a given attribute name:

# app/config/locales/en.yml
en:
  errors:
    attributes:
       title: # phrasing will apply to any attribute called "title”
           blank: "must be provided”

Or you can get even more granular and override specific attributes on a given model. For example if we have the following model and validation:

class Album < ApplicationRecord
  validates :category, presence: true
end

It’s possible to set a bespoke error message for this specific model and attribute:

# app/config/locales/en.yml
en:
  activerecord:
    errors:
      models:
        album: # model name
          category: # attribute name
            blank: "must be selected"

Thing 3: Overriding the error message format itself

As well as overriding individual validation messages globally and for specific attributes, it’s also possible to change the actual format of error messages. By default, errors are constructed with the familiar format of "%{attribute} %{message}". This is defined in the same YAML that ships with Active Model alongside the default error messages:

# rails/activemodel/lib/active_model/locale/en.yml
en:
  errors:
    # The default format to use in full error messages.
    format: "%{attribute} %{message}"

This format works as a reasonable default to ship with the framework as it gives us mostly coherent error messages out of the box without us having to do anything, but it lacks fidelity and can result in some clunky and unfriendly sounding messages. For example, take this validation:

class Album
  validates :number_of_tracks, comparison: { greater_than: 0 }
end

If we go with the default format we end up with "Number of tracks must be greater than 0". Which is fine, but we can definitely do better.

My personal preference on the apps I work with is to remove the attribute prefix entirely:

# app/config/locales/en.yml
en:
  errors:
    # Remove the attribute name from the default error message format:
    format: "%{message}"

This frees me from having to make my error messages work with attribute’s name upfront, allowing me to write more user-friendly and coherent error messages:

# app/config/locales/en.yml
  activerecord:
    errors:
      models:
        album:
          number_of_tracks:
            greater_than: "You must upload at least one track"

Now changing the default format does come at a cost: without the attribute name prefix, the default messages that ship with Rails no longer form complete sentences. This forces me to write bespoke error messages for each validation I add to my app:

# app/config/locales/en.yml
en:
  activerecord:
    errors:
      models:
        album:
          title:
            blank: "Enter a title for the album"

Personally I’m happy to pay this cost for the benefit of improving the user experience. So whilst you may not want to do this on an existing app with thousands of error messages to backfill, it might be something you consider next time you run rails new and are starting from a clean slate.

Update: There’s actually a fairly neat way to avoid this cost, check out my follow-up post on how.

Thing 4: Using i18 for your custom validations

As well using i18n to manage and override the error messages for the built-in validations, you can also use it to manage the messages for your custom validations.

Let’s look at a custom model validation:

class DirectDebitMandate < ApplicationRecord
  validate :bank_account_must_pass_modulus_check

  private

  def bank_account_must_pass_modulus_check
    if modulus_check_fail?
      errors.add(:bank_account_number, "The account number you entered is not valid")
    end
  end
end

Here the error message is inlined as part of the validation code. Instead of doing this, we can pass in a symbol identifying the error, and define the corresponding message in the locale file using the same key structure as the built-in validations:

class DirectDebitMandate < ApplicationRecord
  validate :bank_account_must_pass_modulus_check

  private

  def bank_account_must_pass_modulus_check
    if modulus_check_fail?
      errors.add(:bank_account_number, :failed_modulus)
    end
  end
end

# app/config/locales/en.yml
en:
  activerecord:
    errors:
      models:
        direct_debit_mandate:
          bank_account_number:
            failed_modulus: "The account number you entered is not valid"

This results in more concise and compact model code, but also equally as nice is that it removes a presentational concern — the specific phrasing of the message — our the business logic.

It’s also possible to interpolate both the attribute name and the value of the attribute directly into the error message, which is useful for custom validations that are used across multiple attributes and/or models:

# app/config/locales/en.yml
en:
  activerecord:
    errors:
      messages:
        failed_modulus: "The %{attribute} '%{value}' is not valid"

Thing 5: Form labels

Another place where Rails makes use of i18n under the hood is the form helpers: specifically, the label helper methods. By default the label helper methods humanize the attribute name to arrive at the label text:

<%= form.label :address_county %> # label text will be "Address county"

One way to override the label text is to pass in a string to the helper call like so:

<%= form.label :address_county, "County (optional)" %>

I think a better way is to define label overrides in the application’s locale file:

# app/config/locales/en.yml

helpers:
  label:
    order: # the model name
      address_county: 'County (optional)'

Not only does this keep the view code more concise and less noisy, but it has the added advantage of defining this label override in one place: now any other forms that display the same label will automatically get the overridden text without you having to manually copy the literal string to every template.

And as with form errors, label overrides can be defined per-model as well as globally:

# app/config/locales/en.yml

helpers:
  label:
    address_county: 'County (optional)' # applies to all models with attributes called address_county

Thing 6: Built-in pluralisation

i18n has sophisticated support for pluralisation that we can leverage to simplify our app code. As an example, we can display the stock availability of a product by calling out to i18n like so:

<!-- app/views/albums/show.html.erb —>

<h1><%= @album.name %></h1>

<div class="availability">
  <%= t 'album.availability', count: @album.stock.count %>
</div>

And then define the phrasing based on the number of items in stock in our locale file:

# app/config/locales/en.yml
en:
  album:
    availability:
      zero: "Out of stock"
      one: "Just one album left in stock!"
      other: "%{count} albums in stock"

Much neater than a bunch of conditional logic in a helper method!

Thing 7: Action Mailer support

As well as being available in Rails controllers and views, i18n is also integrated into Action Mailer, allowing you to organise content for mailer templates in local files. Mailer get one extra feature of controllers: the ability to specify email subject lines in locale files:

# app/config/locales/en.yml

en:
  membership_mailer: # mailer name
    submitted: # mailer action
      subject: "Welcome to to the union!"
    dropped:
      subject: "Want to complete your membership application?"

Again, this enables us to move another presentational concern out of our application code and put it somewhere more appropriate.

There is also a meta benefit to defining all these bits of content (mailer subject lines, form labels, validation error messages) in an application’s locale file: it puts them all in one centralised place, making them easier to change, check for consistency and generally manage. It also makes the content more accessible to non-developers such as designers and product folks.

Thing 8: Time and date formatting

Right at the top I mentioned the I18n.localize method that is used to output localised times and dates. The way this works is by defining our own time and date formats in our locale file using the same format specification as Ruby’s strftime:

# config/locales/en.yml
en:
  date:
    # ...
    formats:
      short: "%d/%m/%Y"

Here we’ve defined a short date format that will be familiar to my UK-based audience: day/month/year. To render a date using this format we simply pass in the format identifier to I18n.localize (or its l alias):

<%= I18n.localize @album.released_on, format: short %>

or with the l shorthand:

<%= l @album.released_on, format: short %>

UK vs US dates…

Now my American readers may be confused by this short format, as across the pond the preferred way to display dates is actually month/day/year. So whilst you may not need to localise your entire application, you might want to consider localising your date rendering, especially if you have customers both sides of the Atlantic.

To do this you would define a US region specific English locale file:

# config/locales/en-US.yml
en-US:
  date:
    # ...
    formats:
      short: "%m/%d/%Y"

Then update your controller code to automatically switch locale so the user gets dates formatted appropriately for them.

There are many ways to do actually do the locale switching, but at a high-level, you identify the user’s specific locale and then make sure your controller actions are executed with that locale. Here’s an example of doing that by sniffing the user’s locale from their browser headers:

class ApplicationController < ActionController::Base
  around_action :switch_locale

  private

  def switch_locale(&action)
    browser_locale = extract_locale(request.env['HTTP_ACCEPT_LANGUAGE'])

    I18n.with_locale(browser_locale, &action)
  end
end

(The exact mechanics of the extract_locale method is left as a separate exercise, but you could do worse than use the http_accept_language library)

Thing 9: You can define locale files in Ruby

Little known fact: the i18n library supports locale files defined using Ruby! Now why would you do this, other than a burning hatred for YAML? Well one reason is that Ruby-based locale files allows you to make use of procs to dynamically generate content.

Below is an excerpt from a Ruby-based locale file from one of my applications. It uses a proc to format dates as academic years, changing the output depending on which side of September the date is:

# config/locales/en.rb

{
  en: {
    date: {
      formats: {
        academic_year: ->(date, _) {
          year = date.year
          date.month >= 9 ? "#{year}/#{year + 1}" : "#{year - 1}/#{year}"
        }
      }
    }
  }
}

Now this could just as easily be achieved using a standard Rails helper method. But for me I like the consistency and clarity of using a single mechanism (localize) for formatting dates across my application code, rather than with a mishmash of calls to bespoke helpers, DateTime#to_fs, strftime, etc.

Thing 10: …

This is the part of the lightning talk where I make a meta point about what I’ve covered. I won’t write that up here, but if you’re interested go watch the video from minute 9.

A lightning talk exploring i18n in Rails and how it can be helpful to us even when we’re not translating our applications into other languages.

I gave this talk at Brighton Ruby in 2026.

Slides

I think it’s worth taking a moment to analyse DHH’s post in more detail and make it clear exactly why it’s so problematic.

In his post, DHH complains that London is no longer a city he wants to live in because it is now only a third “native Brit”. His use of “native Brit” is as a proxy for “White British”. The implication is clear: if you are not White, you are not British.

(Update: for the avoidance of doubt that David is talking about whiteness here, when he makes his claim that London is only a third “native Brit” he references this Wikipedia post on London ethnicity statistics where the most recent census data shows 36.8% of individuals identifying as White British, excluding all those that identify as either Asian or Black British from his made up “native Brit” classification.)

In the same post he praises Tommy Robinson (actual name Stephen Christopher Yaxley-Lennon), a right-wing agitator with several convictions for violent offences and a long history of association with far-right groups such as the English Defence League and the British Nationalist Party. He then goes on to describe those that attended last weekend’s far-right rally in London as “perfectly normal, peaceful Brits” protesting against the “demographic nightmare” that has enveloped London, despite the violence and disorder they caused.

To all of that he adds a dash of Islamophobia, citing “Pakistani rape gangs” as one of the reasons for the unrest, repeating a weaponised trope borne from a long since discredited report from the Quilliam Foundation, an organisation with ties to both the the US Tea Party, and Tommy Robinson himself. A trope that exists despite the fact that the overwhelming majority of convicted child sex offenders are white men, with Asian men in fact under-represented.

According to DHH, there is nothing racist or xenophobic in saying “Britain primarily a united kingdom for the Brits”, which again let’s be absolutely clear; by “Brits” he means White people.

I’m tired

As a non-white British citizen born and raised in London, I can’t explain just how painful it is to hear this sort of toxic rhetoric being promoted by one of the most prominent and visible leaders of the Ruby community. I experienced my fair share of racism growing up in London in the eighties and nineties from the very people that DHH is lionising in his post. White men who questioned my right to exist. Places and spaces where I didn’t feel welcome or safe because of my ethnic origin. And despite the recent rise in far-right politics, for the most part things are much better today. Anyone who has spent even the smallest amount of time in the great city of London knows that it is precisely its rich multiculturalism that makes it great. The migration issue in Britain is complicated and multi-layered and DHH wades into it with all the subtlety and nuance of a bull in china shop, sharing the same old and tired xenophobic tropes.

Unfortunately it isn’t just migrants and non-white people that have come under fire in DHH’s recent writing and social media posts. There have been multiple examples of anti-trans rhetoric. There was the post where he described an ad featuring a plus-sized Black women as “grotesque” and celebrated the ads being replaced with ones featuring “blond babies” (what is it about the baby’s blondness that is relevant I wonder?). There have been the posts veering into puritanical Pronatalism, another favourite subject of the nationalist right. Whether it’s his direct intention or not, there has been a consistent theme of othering and stigmatising in his recent writing. The message is clear: if you are trans, a migrant, black, overweight, childless, have ADHD, then in DHH’s view you are somehow inferior. Instead of raising up and supporting the marginalised and vulnerable, DHH choses to exclude and punch down. This is not something the Ruby community can continue to simply ignore.

I ❤️ Ruby

I love the Ruby community dearly. It’s a community that has given me much joy, and a successful career doing work I love. It’s also a community I’ve done my fair share to help grow and prosper. Through running my local Ruby meetup, speaking at conferences, organising Rails Girls workshops, helping multiple people enter the industry through mentorship, and contributing to open source, including Rails itself. And in the over fifteen years I’ve been part of the community I’ve met many kind, thoughtful and compassionate people, few of whom I would ever imagine sharing DHH’s increasingly reactionary and toxic views. But from the safety of his blue-tick echo chamber, and despite many in the community voicing their concern and disgust, DHH is able to spin a narrative that any opposition to his views are nothing more than noise from “radical, violence-condoning nut jobs”.

Power and the complicity of silence

My sense is that since the great Basecamp implosion, rather than reflect on how he and Jason may have been at fault for what happened, DHH found solace in ideas that allowed him to frame himself as victim. DEI and “wokeness” were the actual problem, not him. The recent state of US politics has only further entrenched him in these ideas and emboldening him to fully mask-off. All that is to say I don’t imagine DHH is likely to change. Too much of his self image is now tied up in the misguided moral high ground of these right-wing ideals. So what can be done?

DHH holds an incredible amount of power. He has the trademarks to Rails, and sits as the chair of the Rails Foundation. He is also a board member at Shopify, a company that contributes a huge amount to the Ruby ecosystem, both financially, and through the incredible work of the Ruby and Rails infrastructure team. Of the twelve Rails Core members, almost half are directly employed by either Basecamp or Shopify. DHH and Basecamp drives much of the direction for new features that make it into Rails. Unfortunately DHH isn’t going anywhere.

At the same time, if we want the Ruby community to grown and thrive and for the next generation of Rubyists to feel welcome, we cannot afford to let the damage he is causing by sharing his harmful and toxic ideas go unchecked. We need many more voices to speak up and make it abundantly clear that his views are not widely held. Most importantly, DHH needs to hear directly from the other leaders in our community that his behaviour is not OK and is causing serious harm. That means the leadership at Ruby Central. That means the members of Rails Core. That means the many podcasters and conference organisers that have a platform and voice that the community listens to. That means Matz himself.

Silence and appeasement is only emboldening DHH to ratchet up the toxicity and allowing the damage to continue. Perhaps if enough of those in positions of power call him out on his bullshit he’ll at at least follow his own advice and keep his politics to himself. More importantly though, it will send a clear message to the wider Ruby community (and those who may be considering joining it) that the majority does not stand with DHH and his toxic views.

Update (9 October 2025)

Since I wrote this post others have also written on the topic. Here’s a selection in case you need further convincing:

• DHH Is Way Worse Than I Thought
• As I remember London
• On DHH’s “As I Remember London”
• Ruby deserves better leadership than DHH
• David, please stop posting

And of course if you haven’t watched it already, you really should watch Tom’s five minute talk from 2014. As relevant today as it was then.

Work

Join Together

2024 was a good year for Join Together. Our main focus was a chunky project for the National Education Union (NEU). As well as replacing their online join process, we also built them a bespoke service for upgrading existing student members to full membership after graduation.

The NEU are one of the largest (and more tech-savvy) unions in the UK, so it was a pretty big deal for us to win the project. And despite a fair amount of complexity and esoteric requirements, we totally nailed it, leaving the folks at the NEU over the moon with what we delivered.

We also shipped projects and updates of varying sizes for a good number of our existing union clients. It’s become clear that updates and ongoing development with existing clients will make up a healthy chunk of our revenue, which is helpful, as bringing new unions onboard can be a long and arduous process (it took a whole year from first contact to signed contract with NEU).

Side gig

By the tail end of the year Join Together’s projects were wrapping up and a previous client of mine got in touch asking if I’d be available for a short contract. They needed urgent help shipping some major updates in a short space of time. With no new Join Together projects starting until the new year I was able to take them up on their offer, and spent the last couple months of 2024 working on their mission. It turned into quite an intense project, working mostly solo to make significant changes to their user modelling (primarily separating admin-related code/authentication from their generic User model⁽¹⁾). In the end it all went smoothly, which was very pleasing, and I was wrapped up in time for Christmas.

I imagine this won’t be the last time I take on short contracts outside of my work on Join Together. Although we’re busy when we have projects on, the in-between time tends to be quiet as the platform mostly takes care of itself. As long as my partners in unionisation are not also on outside contracts, and can take care of urgent issues, it’ll be a good way to keep busy and the bank balance healthy.

Maintainable Ruby Podcast

This summer I made an appearance on the Maintainable Software podcast. I really enjoyed chatting with Robby and sharing my reckons about software, teams and some of the things I believe helps keep software maintainable. Some people have said nice things about the episode, so that’s good.

Conferences

I was sad to miss Haggis Ruby (I got Covid), hopefully they’ll be back in 2025. I did make it to Brighton Ruby tho (ten for ten!), which is always one of the highlights of the year. Andy puts on an incredible conference (and has great taste in cocktail flights). The talks are generally good, but for me it’s about spending time and catching up with Ruby friends, old and new, in lovely Brighton. Have you got your tickets for Brighton Ruby 2025 yet?

I did submit one talk proposal this year, to both Brighton Ruby and RailsConf, but was unsuccessful. It’s for a talk on i18n in Rails. My thesis is that everyone writing Rails code should be making more use of i18n, even when the app isn’t actually being localised into other languages, and that following the less well known Rails’ conventions for i18n can lead to simpler, clearer and more maintainable code. I’d still like to develop this talk at some point, perhaps to share with NWRUG.

Life

Welcome to the world baby Ela! That’s four nieces now, my siblings sticking to what they know.

Travels

Myself and Lauren had an Easter break in Malaga. What we hadn’t realised was that Easter is a big deal in that part of Spain. Our arrival coincided perfectly with Holly Week (Semana Santa), when various brotherhoods parade through the streets wearing unfortunately-appropriated pointy hoods and carrying massive Jesus-themed floats. It’s quite a thing to behold.

In August we went on a trip to Copenhagen by train, stopping off in Brussels and Hamburg on the way. Earlier in the year I’d managed to snag a table at Noma for their final Vegetable Season before they close as a permanent restaurant. An eye-watering amount of money for a meal, but as an experience it was one I won’t forget in a hurry and one of my main highlights of the year.

Copenhagen itself is a beautiful city and checks many of my travel boxes: great food and coffee everywhere; lots of interesting art and culture; plenty of things to do. The main highlights included: Swimming in the pristine harbour; Wandering around Freetown Christiana; The Design Musuem; and Louisiana, a beautiful modern art gallery just outside Copenhagen with notable work by Franz Gerscht, William Kentridge, Louise Bourgeois and Yayoi Kusama.

We were only there for a day, but Hamburg is also somewhere I’d love to go back to and spend more time exploring.

Walking club

Although once again we failed to make it to Snowdon (due to untimely illness), it was another solid showing for our Walking Club this year with big walks every month except August. The highlight for me was an epic hike up Scafell Pike on a glorious sunny day in June. Get out in the wilds people, it’s good for the soul.

DJing

I started DJing regularly again towards the end of the year with a bimonthly residency at Ōdiōba, a lovely little listening bar in Stockport. It’s been great fun playing records out again, and it’s got me reconnecting with my record collection. The sets will be making their way onto my Mixcloud if you’re interested (December’s should be up shortly). I also DJ’d the launch for my friend John’s new book A Time ⋅ A Place, which I would encourage you to buy as it’s great, but it’s also sold out, sorry.

Garden makeover

Almost ten years after moving in it was finally time to sort out our garden. I’m prone to procrastination, especially when it comes to major house projects like this, as personal head space tends to be in short supply. Thankfully Lauren forced matters this spring by calling various trades people for quotes on paving, kicking off a short stint as a garden designer. It’s a small garden (not much bigger than a yard really) so designing it became a careful balancing act between space for planting whilst maintaining enough room for general hanging out when the Manchester weather permits. Here’s the obligatory before and after shots for you:

I’m pretty pleased with how it turned out. It’s a much more useable space, and we were able to make the most of it this summer before winter kicked in. I’m looking forward to more planting and outdoor fun times in 2025.

Film, TV and Theatre

Three films stuck out for me this year: Perfect Days, The Zone of Interest, and Soundtrack to a Coup D’etat. Each was striking and moving in their own way, and I’d recommend all three as must-see films. Honourable mention to Civil War, Dune 2, Kneecap, Longlegs and The Substance.

On the TV front I don’t think it was a vintage year. Post strike/Covid/ZIRP slump? Shōgun, Industry and The Penguin stood out. House of the Dragon, The Bear, Slow Horses, Black Doves, English Teacher and Agatha All Along were all serviceable fun. The show I enjoyed the most however was a reality TV show on Netflix: Culinary Class Wars, a thoroughly entertaining and over the top cross between MasterChef and Squid Games.

We got to the theatre a couple of times, and the most memorable thing for me was Dark Noon, a fantastic and intense retelling of the birth of modern America by a predominantly black South African cast. If you live in Australia anywhere near Sydney (and happen to be reading this in January 2025) I would highly recommend you check it out at the Sydney Festival.

Music

I thought it might have been a quiet one for new music this year, but looking back through the releases I picked up there are some standouts for sure. Two of my favourites come from friends of mine: Private Joy’s Desire! EP and Jamie Finlay’s Sun Dogs album. Other releases that stood out were new albums from Nala Sinephro, Don Glori, Shabaka, Thandi Ntuli and Carlos Nino, Greg Foat ,Lady Wray, Sam Gandel & Sam Wilkes, Allysha Joy, Bricknasty and Hiatus Kaiyote.

Aside from new music, I’ve continued to dig out old gems in various record shops and fairs on my travels, including at Record Planet in Holland, one fo the biggest record fairs there is. Imagine two aircraft hangers full of stalls selling vinyl from all over the world. Two days digging through the crates was exhausting, but loads of fun, and I managed to come back with a modest haul of tasty records, including some I’ve been trying to hunt down for some time. It also happens to coincide with Le Guess Who?, which is a fantastic music festival.

Speaking of festivals, We Out Here was, as always, a wonderful weekend of sunshine, dancing and incredible music. I’m already looking forward to 2025.

Other live music that hit the spot this year included gigs by Shabaka, Lady Wray, Bricknasty. And not forgetting Manchester Music for Gaza, a moving day of music, talks and activities that raised over £10,000 for Medical Aid for Palestinians. Well done Ian, Yousef, Zoe, and all involved. Free Palestine! 🇵🇸

Games

I didn’t find a huge amount of time for games this year. The highlight was probably the Resident Evil 4 remake, which I thoroughly enjoyed: spooky, cinematic, and difficult in the right sort of way. Animal Well was also enjoyable, if a little baffling and opaque. I lost interest in uncovering all the eggs after getting to the end.

I also picked up Cyberpunk 2077 after the PS5 update hoping to get lost in an epic adventure. And whilst the graphics and world building are very impressive, it took too much effort (and more than a few YouTube tutorials) to wrap my head around all the systems and mechanics enough to feel vaguely competent in the game. And by the time I was in my stride it felt like I’d already seen most of the content. With that said I apparently put over 70 hours into it, so I guess it was alright?

I’ve only got a little way into Astrobot and so far it’s a joyful and fun experience that manages to capture some of Nintendo’s magic sauce. I’m looking forward to spending more time with it in 2025.

Honourable mention to Overcooked! All You Can Eat, which several years later continues to eat up countless hours as myself and my friend Lucy wring every last drop of fun out of it trying to 4-star all the levels⁽²⁾. No co-operative game has come close to its combination of chaotic fun and charm. It’s a shame Ghost Town have given up on further expansions as we’d buy them in a heart beat. Here’s hoping their recently-announced new co-operative game is able to replicate some of the same magic.

Looking ahead to 2025

After a couple years of various random issues I’m aiming to get my health and fitness back on track, and generally have a more relaxed and mindful approach to life in 2025. More time for some creative projects. And of course Snowdon must finally be conquered!

Footnotes

1: Perhaps I should blog separately about the patterns and techniques I used to do this iteratively and without any downtime?

2: Unfortunately this may actually be impossible with just the two of us as the target score on some levels appears calibrated for three or four players. Still we plough on…

Selectively stashing changes

git stash is great for temporarily stashing changes that you want to apply later, and handily it also supports selectively stashing changes with the --patch flag:

  $ git stash --patch

  diff --git a/spec/sidekiq/upload_job_spec.rb b/spec/sidekiq/upload_job_spec.rb
  index b5f1d04e..e3b6227d 100644
  --- a/spec/sidekiq/upload_job_spec.rb
  +++ b/spec/sidekiq/upload_job_spec.rb
  @@ -7,6 +7,7 @@ describe '#perform' do
       context 'the application has already been uploaded' do
         let(:membership_application) { create :forsa, :uploaded }

  +      # Here is change one of two
         it 'lets us know via Appsignal' do
           allow(Appsignal).to receive(:report_error)
           job.perform(membership_application.id)

  (1/2) Stash this hunk [y,n,q,a,d,j,J,g,/,e,p,?]?

Bonus git stash tip: you can also selectively stash entire files using -- to disambiguate the command from the paths you want stashing:

  $ git stash -- path/to/file.rb

Selective discarding changes from your current work tree

You can use the git restore command to discard local changes and restore files to their last committed state. It can also be called with the --patch flag to interactively select specific hunks to discard:

  $ git restore --patch

  diff --git a/spec/sidekiq/upload_job_spec.rb b/spec/sidekiq/upload_job_spec.rb
  index b5f1d04e..e3b6227d 100644
  --- a/spec/sidekiq/upload_job_spec.rb
  +++ b/spec/sidekiq/upload_job_spec.rb
  @@ -7,6 +7,7 @@ describe '#perform' do
       context 'the application has already been uploaded' do
         let(:membership_application) { create :forsa, :uploaded }

  +      # Here is change one of two
         it 'lets us know via Appsignal' do
           allow(Appsignal).to receive(:report_error)
           job.perform(membership_application.id)

  (1/2) Discard this hunk from worktree [y,n,q,a,d,j,J,g,/,e,p,?]?

Note the different phrasing of the prompt on the last line: Here we are choosing the changes we want to discard. Be careful, this is a destructive change, and because these are unstaged and uncommitted changes git won’t be able to help you recover the changes once they’ve been discarded!

Selectively restoring changes from another branch or commit

The git restore command also lets you restore changes from another branch or commit by specifying the --source= flag. Combine this with the --patch flag and you can interactively choose the specific changes to restore to your current work tree:

  $ git restore --source=branch-name --patch

  diff --git b/app/models/flow_definitions/neu.rb a/app/models/flow_definitions/neu.rb
  index cdabfe7a..28308244 100644
  --- b/app/models/flow_definitions/neu.rb
  +++ a/app/models/flow_definitions/neu.rb
  @@ -42,10 +42,9 @@
     step 'qualified-year' do
       radio :qualified_year do
  -      option :'2024'
  -      option :'2023'
  -      option :'2022'
  -      option :'2021'
  +      MembershipApplication::Neu::QUALIFYING_YEARS.each do |year|
  +        option :"#{year}"
  +      end

         divider

  (1/2) Apply this hunk to index and worktree [y,n,q,a,d,j,J,g,/,e,p,?]?

I wil sometimes use this when I have a spike branch that I want to pull a subset of changes from to be tidied up and committed in my current branch.

A quick note on git restore vs checkout

You may have noticed that the last two behaviours for discarding/restoring changes are also possible using git checkout. git restore was introduced to Git in version 2.23 (alongside git switch) in an effort to reduce the overloaded (and somewhat confusing) responsibilities placed on git checkout. Whilst it doesn’t look these features will be removed from git checkout any time soon, the more focused commands are easier to understand (and teach!), and arguably easier to use, at least assuming you don’t have years of muscle memory to overcome.

Thanks to Tom and Étienne for the clarifying comments.

Since then Ben Sheldon has pointed out that apps running embedded job workers (for example Sidekiq in embedded mode, GoodJob in async mode or Sucker Punch) will also be creating extra threads and therefor may need a larger thread pool.

In this follow-up post I’m going to describe the technique I used to uncover the source of the connection pool contention for the app I’m working on, and how you can do the same if your seeing mysterious ActiveRecord::ConnectionTimeoutError exceptions and don’t know where they’re coming from.

Uncovering the source of ActiveRecord connection pool contention

These connection timeouts were a long-standing mystery in our app. Although they didn’t happen with great frequency, they were happening often enough to warrant some further digging rather than letting them become another broken window.

To figure out what was going on I employed some introspection on the connection pool. Whenever a thread asks for a connection from ActiveRecord, it is assigned one from the connection pool. The connection itself stores a reference to the assigned thread as its owner. We can inspect the assigned threads in the connection pool to learn a bit more about them:

ActiveRecord::Base.connection_pool.connections.map do |connection|
  connection.owner.present? ? connection.owner.inspect : "[UNUSED]"
end.join("\n")

You can try this in the rails console, although you’ll want to fire off a quick query first, otherwise there won’t be any connections assigned!

On the actual Puma web server, if connections are only assigned to Puma threads the output will look something like this:

#<Thread:0x00007f0bbab3b740@puma srv tp 003 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep>
#<Thread:0x00007f0bbab3b3d0@puma srv tp 004 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 run>
#<Thread:0x00007f0bbab3b8f8@puma srv tp 002 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 run>
#<Thread:0x00007f0bbab3bbc8@puma srv tp 001 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>
[UNUSED]

The inspect output tells us a bit about each thread. The key part is the path that points at the line of code where the thread was started, which in this case is inside the Puma thread pool here.

To narrow down the search for the mystery threads using up our connections I wired this code into our bug tracker such that we’d log this debug output whenever an ActiveRecord::ConnectionTimeoutError was raised:

Bugsnag.configure do |config|
  config.add_on_error(proc do |event|
    if event.errors.first.error_class == "ActiveRecord::ConnectionTimeoutError"
      connection_pool_info = ActiveRecord::Base.connection_pool.connections.map do |connection|
        connection.owner.present? ? connection.owner.inspect : "[UNUSED]"
      end.join("\n")

      event.add_metadata(:app, :connection_pool_info, connection_pool_info)
    end
  end
end

It didn’t take long for the culprit to materialise:

#<Thread:0x00007f32eca13ad0@puma srv tp 002 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>
#<Thread:0x00007f32f23cd6e8 /app/.../gems/actionpack-7.0.5.1/lib/action_controller/metal/live.rb:341 sleep>
#<Thread:0x00007f32eca12a40@puma srv tp 003 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>
#<Thread:0x00007f32f2936288 /app/.../gems/actionpack-7.0.5.1/lib/action_controller/metal/live.rb:341 sleep>
#<Thread:0x00007f32eca13e40@puma srv tp 001 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 run>

We can see that two of the connections are assigned to threads spawned by ActionController::Metal::Live here.

This was the missing piece of the puzzle. It didn’t take much tracing of code to discover that it was ActiveStorage proxy requests that spin up extra threads for streaming responses, and it was these threads that were putting the extra pressure on our connection pool and causing the sporadic timeout errors.

Introspection is good actually

I wanted to write this post partly to serve as a pointer to others that might be seeing these connection timeouts, but also to make a broader point about the blurred lines between our code and the libraries our code depends on. Because when our app is running, it all effectively becomes our code. Getting comfortable diving into the source for gems you use will serve you well, and will sometimes be the only way to get to the origin of a strange bug, or unexpected behaviour.

One of my favourite ways to do this is using bundle open. It gets you to the literal gem code that your app will be using right there in your preferred editor, ready for exploring. You can even make changes to the gem code for a spot of puts debugging (although do so with care and always remember to undo your changes when you’re done!)

Aside: If you’re a VSCode user, you may want to set the BUNDLER_EDITOR environment variable to "code -n" to stop bundle open replacing your current project window.

The nuance missing from that advice is that it assumes no additional threads are ever spawned during your apps operation!

Now although you might know your code inside and out and be 100% certain that you don’t create additional threads anywhere in your code, chances are Rails is creating additional threads without you realising it…

An aside on threads and ActiveRecord’s connection pool

On a basic level, the way ActiveRecord’s connection pool works is that it assigns each thread that asks for a connection its own separate connection, which the thread then releases once it’s finished. Using a pool like this means that many threads can be querying the database at the same time, so many more requests can be processed in parallel.

If your app’s Puma config sets the maximum number of threads to 5, then you can normally expect there to be at most 5 threads asking for their own database connection, hence the advice to set the pool size to the same as size as the number of threads.

If however one of those threads creates another thread of its own, and that thread needs a database connection to hit the database, it will need its own connection. The spawned thread does not share the connection of the thread that spawned it.

So if we have threads that spawn their own threads, it’s possible to end up in a situation where there are many more threads wanting a connection then there are available connections, and you could end up seeing ActiveRecord::ConnectionTimeoutError exceptions being raised.

Where your Rails app might be spinning up additional threads without you realising it

So, back to your app. Even if you are not explicitly spawning your own threads, Rails itself could be spinning up additional threads without you realising. The main culprit here is ActiveStorage, or more specifically, ActiveStorage configured in proxy mode (commonly used if you’re serving assets via a CDN).

Threads created by ActiveStorage

ActiveStorage’s two proxy controllers return streamed responses, which (you guessed it) are processed in their own threads! So to handle a request for an ActiveStorage file via one of the proxy controllers, two threads will be called into action, both of which need their own connection from the ActiveRecord connection pool.

Normally (hopefully?) your app is processing requests fast enough for you to expect connections to be freed up from one of the other threads and become available for the streaming thread before a timeout occurs. And in this way, most of the time, the limited connections available in the pool can be shared between more threads than there are connections. The problems start if your app receives many successive/concurrent ActiveStorage proxy requests, each of which spin up an additional thread to stream a response, and they take a long time to complete their work and free up their connections; either because they’re doing expensive/slow work server-side (downloading a large file or processing an image representation for the first time), or they’re streaming the response to a slow client. At the time of writing at least, both threads hang on to their connection until the entire response is complete.

Threads created by ActiveRecord’s load_async

Another place Rails will spin up additional threads is with the new load_async method. This was introduced in Rails 7 as a way to parallelise expensive database queries. It’s less likely that these are going to cause connection timeout errors as (again, hopefully?) the expensive queries aren’t so expensive that they hang onto connections for an excessive amount of time. That said, if you’re making heavy use of load_async, you could again end up in a situation where you have much higher demand for connections than there are available connections.

So what should you do about all this?

The upshot of all this is: if you’re making use of either ActiveStorage in proxy mode, or calling load_async in our app, you probably want a connection pool size that is higher than the number of configured Puma threads. The theoretical maximum number of connections you’ll need will be whatever Puma’s configured thread count is x 2, based on the assumption that each thread can potentially spawn one additional thread, but you might be able to get away with a smaller multiple depending on the performance and load characteristics of your app.

And if you’re close to reaching or exceeding the connection limit offered by your database, consider using a separate connection pool like pgbouncer (available as a Heroku buildpack) to increase the number of connections you can configure.

Update (6 August 2023)

There’s now a follow-up post outlining one of the techniques I employed to help figure out why our connection pool was being exhausted in our Rails app.

Update (19 September 2024)

Ben Sheldon has written a great post explaining how to perfectly calculate your Rails database connection pool size. The TL;DR: just make it very big!

The git branch command accepts a --sort option which we can use to list our branches based on the last committer date:

  $ git branch --sort=-committerdate
  * main
    redact-abandoned
    log-downloads
    i18n-lint

We can also use the --format option to include the exact time and see just how fresh each branch is:

  $ git branch --sort=-committerdate --format="%(committerdate)%09%(refname:short)"
  Thu Nov 25 10:29:48 2021 +0000  main
  Fri Nov 19 16:08:49 2021 +0000  redact-abandoned
  Fri Nov 19 16:04:11 2021 +0000  log-downloads
  Thu Nov 18 21:53:37 2021 +0000  i18n-lint
  Sun Jul 18 12:49:18 2021 +0100  without-routing-key-overrides

Or for something more friendly and easy-to-parse we can ask for a relative date:

  $ git branch --sort=-committerdate --format="%(committerdate:relative)%09%(refname:short)"
  5 hours ago     main
  6 days ago      redact-abandoned
  6 days ago      log-downloads
  7 days ago      i18n-lint
  4 months ago    without-routing-key-overrides

That’s better! Let’s add this as a git recent alias to our Git config:

  $ git config --global alias.recent 'branch --sort=-committerdate --format="%(committerdate:relative)%09%(refname:short)"'

Hat tip to Tenderlove for this particularly tasty snippet.

Here are some other Git related articles you might find useful:

• An alias for cleaning up your old/redundant Git branches
• How focused commits make you a better coder
• Better Git diff output for Ruby, Python, Elixer, Go and more
• An alias for jumping from a commit SHA to the PR on GitHub
• An alias for amending your most recent commit