A lightning talk exploring i18n in Rails and how it can be helpful to us even when we’re not translating our applications into other languages.

I gave this talk at Brighton Ruby in 2026.

Slides

I think it’s worth taking a moment to analyse DHH’s post in more detail and make it clear exactly why it’s so problematic.

In his post, DHH complains that London is no longer a city he wants to live in because it is now only a third “native Brit”. His use of “native Brit” is as a proxy for “White British”. The implication is clear: if you are not White, you are not British.

(Update: for the avoidance of doubt that David is talking about whiteness here, when he makes his claim that London is only a third “native Brit” he references this Wikipedia post on London ethnicity statistics where the most recent census data shows 36.8% of individuals identifying as White British, excluding all those that identify as either Asian or Black British from his made up “native Brit” classification.)

In the same post he praises Tommy Robinson (actual name Stephen Christopher Yaxley-Lennon), a right-wing agitator with several convictions for violent offences and a long history of association with far-right groups such as the English Defence League and the British Nationalist Party. He then goes on to describe those that attended last weekend’s far-right rally in London as “perfectly normal, peaceful Brits” protesting against the “demographic nightmare” that has enveloped London, despite the violence and disorder they caused.

To all of that he adds a dash of Islamophobia, citing “Pakistani rape gangs” as one of the reasons for the unrest, repeating a weaponised trope borne from a long since discredited report from the Quilliam Foundation, an organisation with ties to both the the US Tea Party, and Tommy Robinson himself. A trope that exists despite the fact that the overwhelming majority of convicted child sex offenders are white men, with Asian men in fact under-represented.

According to DHH, there is nothing racist or xenophobic in saying “Britain primarily a united kingdom for the Brits”, which again let’s be absolutely clear; by “Brits” he means White people.

I’m tired

As a non-white British citizen born and raised in London, I can’t explain just how painful it is to hear this sort of toxic rhetoric being promoted by one of the most prominent and visible leaders of the Ruby community. I experienced my fair share of racism growing up in London in the eighties and nineties from the very people that DHH is lionising in his post. White men who questioned my right to exist. Places and spaces where I didn’t feel welcome or safe because of my ethnic origin. And despite the recent rise in far-right politics, for the most part things are much better today. Anyone who has spent even the smallest amount of time in the great city of London knows that it is precisely its rich multiculturalism that makes it great. The migration issue in Britain is complicated and multi-layered and DHH wades into it with all the subtlety and nuance of a bull in china shop, sharing the same old and tired xenophobic tropes.

Unfortunately it isn’t just migrants and non-white people that have come under fire in DHH’s recent writing and social media posts. There have been multiple examples of anti-trans rhetoric. There was the post where he described an ad featuring a plus-sized Black women as “grotesque” and celebrated the ads being replaced with ones featuring “blond babies” (what is it about the baby’s blondness that is relevant I wonder?). There have been the posts veering into puritanical Pronatalism, another favourite subject of the nationalist right. Whether it’s his direct intention or not, there has been a consistent theme of othering and stigmatising in his recent writing. The message is clear: if you are trans, a migrant, black, overweight, childless, have ADHD, then in DHH’s view you are somehow inferior. Instead of raising up and supporting the marginalised and vulnerable, DHH choses to exclude and punch down. This is not something the Ruby community can continue to simply ignore.

I ❤️ Ruby

I love the Ruby community dearly. It’s a community that has given me much joy, and a successful career doing work I love. It’s also a community I’ve done my fair share to help grow and prosper. Through running my local Ruby meetup, speaking at conferences, organising Rails Girls workshops, helping multiple people enter the industry through mentorship, and contributing to open source, including Rails itself. And in the over fifteen years I’ve been part of the community I’ve met many kind, thoughtful and compassionate people, few of whom I would ever imagine sharing DHH’s increasingly reactionary and toxic views. But from the safety of his blue-tick echo chamber, and despite many in the community voicing their concern and disgust, DHH is able to spin a narrative that any opposition to his views are nothing more than noise from “radical, violence-condoning nut jobs”.

Power and the complicity of silence

My sense is that since the great Basecamp implosion, rather than reflect on how he and Jason may have been at fault for what happened, DHH found solace in ideas that allowed him to frame himself as victim. DEI and “wokeness” were the actual problem, not him. The recent state of US politics has only further entrenched him in these ideas and emboldening him to fully mask-off. All that is to say I don’t imagine DHH is likely to change. Too much of his self image is now tied up in the misguided moral high ground of these right-wing ideals. So what can be done?

DHH holds an incredible amount of power. He has the trademarks to Rails, and sits as the chair of the Rails Foundation. He is also a board member at Shopify, a company that contributes a huge amount to the Ruby ecosystem, both financially, and through the incredible work of the Ruby and Rails infrastructure team. Of the twelve Rails Core members, almost half are directly employed by either Basecamp or Shopify. DHH and Basecamp drives much of the direction for new features that make it into Rails. Unfortunately DHH isn’t going anywhere.

At the same time, if we want the Ruby community to grown and thrive and for the next generation of Rubyists to feel welcome, we cannot afford to let the damage he is causing by sharing his harmful and toxic ideas go unchecked. We need many more voices to speak up and make it abundantly clear that his views are not widely held. Most importantly, DHH needs to hear directly from the other leaders in our community that his behaviour is not OK and is causing serious harm. That means the leadership at Ruby Central. That means the members of Rails Core. That means the many podcasters and conference organisers that have a platform and voice that the community listens to. That means Matz himself.

Silence and appeasement is only emboldening DHH to ratchet up the toxicity and allowing the damage to continue. Perhaps if enough of those in positions of power call him out on his bullshit he’ll at at least follow his own advice and keep his politics to himself. More importantly though, it will send a clear message to the wider Ruby community (and those who may be considering joining it) that the majority does not stand with DHH and his toxic views.

Update (9 October 2025)

Since I wrote this post others have also written on the topic. Here’s a selection in case you need further convincing:

• DHH Is Way Worse Than I Thought
• As I remember London
• On DHH’s “As I Remember London”
• Ruby deserves better leadership than DHH
• David, please stop posting

And of course if you haven’t watched it already, you really should watch Tom’s five minute talk from 2014. As relevant today as it was then.

Work

Join Together

2024 was a good year for Join Together. Our main focus was a chunky project for the National Education Union (NEU). As well as replacing their online join process, we also built them a bespoke service for upgrading existing student members to full membership after graduation.

The NEU are one of the largest (and more tech-savvy) unions in the UK, so it was a pretty big deal for us to win the project. And despite a fair amount of complexity and esoteric requirements, we totally nailed it, leaving the folks at the NEU over the moon with what we delivered.

We also shipped projects and updates of varying sizes for a good number of our existing union clients. It’s become clear that updates and ongoing development with existing clients will make up a healthy chunk of our revenue, which is helpful, as bringing new unions onboard can be a long and arduous process (it took a whole year from first contact to signed contract with NEU).

Side gig

By the tail end of the year Join Together’s projects were wrapping up and a previous client of mine got in touch asking if I’d be available for a short contract. They needed urgent help shipping some major updates in a short space of time. With no new Join Together projects starting until the new year I was able to take them up on their offer, and spent the last couple months of 2024 working on their mission. It turned into quite an intense project, working mostly solo to make significant changes to their user modelling (primarily separating admin-related code/authentication from their generic User model⁽¹⁾). In the end it all went smoothly, which was very pleasing, and I was wrapped up in time for Christmas.

I imagine this won’t be the last time I take on short contracts outside of my work on Join Together. Although we’re busy when we have projects on, the in-between time tends to be quiet as the platform mostly takes care of itself. As long as my partners in unionisation are not also on outside contracts, and can take care of urgent issues, it’ll be a good way to keep busy and the bank balance healthy.

Maintainable Ruby Podcast

This summer I made an appearance on the Maintainable Software podcast. I really enjoyed chatting with Robby and sharing my reckons about software, teams and some of the things I believe helps keep software maintainable. Some people have said nice things about the episode, so that’s good.

Conferences

I was sad to miss Haggis Ruby (I got Covid), hopefully they’ll be back in 2025. I did make it to Brighton Ruby tho (ten for ten!), which is always one of the highlights of the year. Andy puts on an incredible conference (and has great taste in cocktail flights). The talks are generally good, but for me it’s about spending time and catching up with Ruby friends, old and new, in lovely Brighton. Have you got your tickets for Brighton Ruby 2025 yet?

I did submit one talk proposal this year, to both Brighton Ruby and RailsConf, but was unsuccessful. It’s for a talk on i18n in Rails. My thesis is that everyone writing Rails code should be making more use of i18n, even when the app isn’t actually being localised into other languages, and that following the less well known Rails’ conventions for i18n can lead to simpler, clearer and more maintainable code. I’d still like to develop this talk at some point, perhaps to share with NWRUG.

Life

Welcome to the world baby Ela! That’s four nieces now, my siblings sticking to what they know.

Travels

Myself and Lauren had an Easter break in Malaga. What we hadn’t realised was that Easter is a big deal in that part of Spain. Our arrival coincided perfectly with Holly Week (Semana Santa), when various brotherhoods parade through the streets wearing unfortunately-appropriated pointy hoods and carrying massive Jesus-themed floats. It’s quite a thing to behold.

In August we went on a trip to Copenhagen by train, stopping off in Brussels and Hamburg on the way. Earlier in the year I’d managed to snag a table at Noma for their final Vegetable Season before they close as a permanent restaurant. An eye-watering amount of money for a meal, but as an experience it was one I won’t forget in a hurry and one of my main highlights of the year.

Copenhagen itself is a beautiful city and checks many of my travel boxes: great food and coffee everywhere; lots of interesting art and culture; plenty of things to do. The main highlights included: Swimming in the pristine harbour; Wandering around Freetown Christiana; The Design Musuem; and Louisiana, a beautiful modern art gallery just outside Copenhagen with notable work by Franz Gerscht, William Kentridge, Louise Bourgeois and Yayoi Kusama.

We were only there for a day, but Hamburg is also somewhere I’d love to go back to and spend more time exploring.

Walking club

Although once again we failed to make it to Snowdon (due to untimely illness), it was another solid showing for our Walking Club this year with big walks every month except August. The highlight for me was an epic hike up Scafell Pike on a glorious sunny day in June. Get out in the wilds people, it’s good for the soul.

DJing

I started DJing regularly again towards the end of the year with a bimonthly residency at Ōdiōba, a lovely little listening bar in Stockport. It’s been great fun playing records out again, and it’s got me reconnecting with my record collection. The sets will be making their way onto my Mixcloud if you’re interested (December’s should be up shortly). I also DJ’d the launch for my friend John’s new book A Time ⋅ A Place, which I would encourage you to buy as it’s great, but it’s also sold out, sorry.

Garden makeover

Almost ten years after moving in it was finally time to sort out our garden. I’m prone to procrastination, especially when it comes to major house projects like this, as personal head space tends to be in short supply. Thankfully Lauren forced matters this spring by calling various trades people for quotes on paving, kicking off a short stint as a garden designer. It’s a small garden (not much bigger than a yard really) so designing it became a careful balancing act between space for planting whilst maintaining enough room for general hanging out when the Manchester weather permits. Here’s the obligatory before and after shots for you:

I’m pretty pleased with how it turned out. It’s a much more useable space, and we were able to make the most of it this summer before winter kicked in. I’m looking forward to more planting and outdoor fun times in 2025.

Film, TV and Theatre

Three films stuck out for me this year: Perfect Days, The Zone of Interest, and Soundtrack to a Coup D’etat. Each was striking and moving in their own way, and I’d recommend all three as must-see films. Honourable mention to Civil War, Dune 2, Kneecap, Longlegs and The Substance.

On the TV front I don’t think it was a vintage year. Post strike/Covid/ZIRP slump? Shōgun, Industry and The Penguin stood out. House of the Dragon, The Bear, Slow Horses, Black Doves, English Teacher and Agatha All Along were all serviceable fun. The show I enjoyed the most however was a reality TV show on Netflix: Culinary Class Wars, a thoroughly entertaining and over the top cross between MasterChef and Squid Games.

We got to the theatre a couple of times, and the most memorable thing for me was Dark Noon, a fantastic and intense retelling of the birth of modern America by a predominantly black South African cast. If you live in Australia anywhere near Sydney (and happen to be reading this in January 2025) I would highly recommend you check it out at the Sydney Festival.

Music

I thought it might have been a quiet one for new music this year, but looking back through the releases I picked up there are some standouts for sure. Two of my favourites come from friends of mine: Private Joy’s Desire! EP and Jamie Finlay’s Sun Dogs album. Other releases that stood out were new albums from Nala Sinephro, Don Glori, Shabaka, Thandi Ntuli and Carlos Nino, Greg Foat ,Lady Wray, Sam Gandel & Sam Wilkes, Allysha Joy, Bricknasty and Hiatus Kaiyote.

Aside from new music, I’ve continued to dig out old gems in various record shops and fairs on my travels, including at Record Planet in Holland, one fo the biggest record fairs there is. Imagine two aircraft hangers full of stalls selling vinyl from all over the world. Two days digging through the crates was exhausting, but loads of fun, and I managed to come back with a modest haul of tasty records, including some I’ve been trying to hunt down for some time. It also happens to coincide with Le Guess Who?, which is a fantastic music festival.

Speaking of festivals, We Out Here was, as always, a wonderful weekend of sunshine, dancing and incredible music. I’m already looking forward to 2025.

Other live music that hit the spot this year included gigs by Shabaka, Lady Wray, Bricknasty. And not forgetting Manchester Music for Gaza, a moving day of music, talks and activities that raised over £10,000 for Medical Aid for Palestinians. Well done Ian, Yousef, Zoe, and all involved. Free Palestine! 🇵🇸

Games

I didn’t find a huge amount of time for games this year. The highlight was probably the Resident Evil 4 remake, which I thoroughly enjoyed: spooky, cinematic, and difficult in the right sort of way. Animal Well was also enjoyable, if a little baffling and opaque. I lost interest in uncovering all the eggs after getting to the end.

I also picked up Cyberpunk 2077 after the PS5 update hoping to get lost in an epic adventure. And whilst the graphics and world building are very impressive, it took too much effort (and more than a few YouTube tutorials) to wrap my head around all the systems and mechanics enough to feel vaguely competent in the game. And by the time I was in my stride it felt like I’d already seen most of the content. With that said I apparently put over 70 hours into it, so I guess it was alright?

I’ve only got a little way into Astrobot and so far it’s a joyful and fun experience that manages to capture some of Nintendo’s magic sauce. I’m looking forward to spending more time with it in 2025.

Honourable mention to Overcooked! All You Can Eat, which several years later continues to eat up countless hours as myself and my friend Lucy wring every last drop of fun out of it trying to 4-star all the levels⁽²⁾. No co-operative game has come close to its combination of chaotic fun and charm. It’s a shame Ghost Town have given up on further expansions as we’d buy them in a heart beat. Here’s hoping their recently-announced new co-operative game is able to replicate some of the same magic.

Looking ahead to 2025

After a couple years of various random issues I’m aiming to get my health and fitness back on track, and generally have a more relaxed and mindful approach to life in 2025. More time for some creative projects. And of course Snowdon must finally be conquered!

Footnotes

1: Perhaps I should blog separately about the patterns and techniques I used to do this iteratively and without any downtime?

2: Unfortunately this may actually be impossible with just the two of us as the target score on some levels appears calibrated for three or four players. Still we plough on…

Selectively stashing changes

git stash is great for temporarily stashing changes that you want to apply later, and handily it also supports selectively stashing changes with the --patch flag:

  $ git stash --patch

  diff --git a/spec/sidekiq/upload_job_spec.rb b/spec/sidekiq/upload_job_spec.rb
  index b5f1d04e..e3b6227d 100644
  --- a/spec/sidekiq/upload_job_spec.rb
  +++ b/spec/sidekiq/upload_job_spec.rb
  @@ -7,6 +7,7 @@ describe '#perform' do
       context 'the application has already been uploaded' do
         let(:membership_application) { create :forsa, :uploaded }

  +      # Here is change one of two
         it 'lets us know via Appsignal' do
           allow(Appsignal).to receive(:report_error)
           job.perform(membership_application.id)

  (1/2) Stash this hunk [y,n,q,a,d,j,J,g,/,e,p,?]?

Bonus git stash tip: you can also selectively stash entire files using -- to disambiguate the command from the paths you want stashing:

  $ git stash -- path/to/file.rb

Selective discarding changes from your current work tree

You can use the git restore command to discard local changes and restore files to their last committed state. It can also be called with the --patch flag to interactively select specific hunks to discard:

  $ git restore --patch

  diff --git a/spec/sidekiq/upload_job_spec.rb b/spec/sidekiq/upload_job_spec.rb
  index b5f1d04e..e3b6227d 100644
  --- a/spec/sidekiq/upload_job_spec.rb
  +++ b/spec/sidekiq/upload_job_spec.rb
  @@ -7,6 +7,7 @@ describe '#perform' do
       context 'the application has already been uploaded' do
         let(:membership_application) { create :forsa, :uploaded }

  +      # Here is change one of two
         it 'lets us know via Appsignal' do
           allow(Appsignal).to receive(:report_error)
           job.perform(membership_application.id)

  (1/2) Discard this hunk from worktree [y,n,q,a,d,j,J,g,/,e,p,?]?

Note the different phrasing of the prompt on the last line: Here we are choosing the changes we want to discard. Be careful, this is a destructive change, and because these are unstaged and uncommitted changes git won’t be able to help you recover the changes once they’ve been discarded!

Selectively restoring changes from another branch or commit

The git restore command also lets you restore changes from another branch or commit by specifying the --source= flag. Combine this with the --patch flag and you can interactively choose the specific changes to restore to your current work tree:

  $ git restore --source=branch-name --patch

  diff --git b/app/models/flow_definitions/neu.rb a/app/models/flow_definitions/neu.rb
  index cdabfe7a..28308244 100644
  --- b/app/models/flow_definitions/neu.rb
  +++ a/app/models/flow_definitions/neu.rb
  @@ -42,10 +42,9 @@
     step 'qualified-year' do
       radio :qualified_year do
  -      option :'2024'
  -      option :'2023'
  -      option :'2022'
  -      option :'2021'
  +      MembershipApplication::Neu::QUALIFYING_YEARS.each do |year|
  +        option :"#{year}"
  +      end

         divider

  (1/2) Apply this hunk to index and worktree [y,n,q,a,d,j,J,g,/,e,p,?]?

I wil sometimes use this when I have a spike branch that I want to pull a subset of changes from to be tidied up and committed in my current branch.

A quick note on git restore vs checkout

You may have noticed that the last two behaviours for discarding/restoring changes are also possible using git checkout. git restore was introduced to Git in version 2.23 (alongside git switch) in an effort to reduce the overloaded (and somewhat confusing) responsibilities placed on git checkout. Whilst it doesn’t look these features will be removed from git checkout any time soon, the more focused commands are easier to understand (and teach!), and arguably easier to use, at least assuming you don’t have years of muscle memory to overcome.

Thanks to Tom and Étienne for the clarifying comments.

Since then Ben Sheldon has pointed out that apps running embedded job workers (for example Sidekiq in embedded mode, GoodJob in async mode or Sucker Punch) will also be creating extra threads and therefor may need a larger thread pool.

In this follow-up post I’m going to describe the technique I used to uncover the source of the connection pool contention for the app I’m working on, and how you can do the same if your seeing mysterious ActiveRecord::ConnectionTimeoutError exceptions and don’t know where they’re coming from.

Uncovering the source of ActiveRecord connection pool contention

These connection timeouts were a long-standing mystery in our app. Although they didn’t happen with great frequency, they were happening often enough to warrant some further digging rather than letting them become another broken window.

To figure out what was going on I employed some introspection on the connection pool. Whenever a thread asks for a connection from ActiveRecord, it is assigned one from the connection pool. The connection itself stores a reference to the assigned thread as its owner. We can inspect the assigned threads in the connection pool to learn a bit more about them:

ActiveRecord::Base.connection_pool.connections.map do |connection|
  connection.owner.present? ? connection.owner.inspect : "[UNUSED]"
end.join("\n")

You can try this in the rails console, although you’ll want to fire off a quick query first, otherwise there won’t be any connections assigned!

On the actual Puma web server, if connections are only assigned to Puma threads the output will look something like this:

#<Thread:0x00007f0bbab3b740@puma srv tp 003 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep>
#<Thread:0x00007f0bbab3b3d0@puma srv tp 004 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 run>
#<Thread:0x00007f0bbab3b8f8@puma srv tp 002 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 run>
#<Thread:0x00007f0bbab3bbc8@puma srv tp 001 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>
[UNUSED]

The inspect output tells us a bit about each thread. The key part is the path that points at the line of code where the thread was started, which in this case is inside the Puma thread pool here.

To narrow down the search for the mystery threads using up our connections I wired this code into our bug tracker such that we’d log this debug output whenever an ActiveRecord::ConnectionTimeoutError was raised:

Bugsnag.configure do |config|
  config.add_on_error(proc do |event|
    if event.errors.first.error_class == "ActiveRecord::ConnectionTimeoutError"
      connection_pool_info = ActiveRecord::Base.connection_pool.connections.map do |connection|
        connection.owner.present? ? connection.owner.inspect : "[UNUSED]"
      end.join("\n")

      event.add_metadata(:app, :connection_pool_info, connection_pool_info)
    end
  end
end

It didn’t take long for the culprit to materialise:

#<Thread:0x00007f32eca13ad0@puma srv tp 002 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>
#<Thread:0x00007f32f23cd6e8 /app/.../gems/actionpack-7.0.5.1/lib/action_controller/metal/live.rb:341 sleep>
#<Thread:0x00007f32eca12a40@puma srv tp 003 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 sleep_forever>
#<Thread:0x00007f32f2936288 /app/.../gems/actionpack-7.0.5.1/lib/action_controller/metal/live.rb:341 sleep>
#<Thread:0x00007f32eca13e40@puma srv tp 001 /app/.../gems/puma-6.2.2/lib/puma/thread_pool.rb:106 run>

We can see that two of the connections are assigned to threads spawned by ActionController::Metal::Live here.

This was the missing piece of the puzzle. It didn’t take much tracing of code to discover that it was ActiveStorage proxy requests that spin up extra threads for streaming responses, and it was these threads that were putting the extra pressure on our connection pool and causing the sporadic timeout errors.

Introspection is good actually

I wanted to write this post partly to serve as a pointer to others that might be seeing these connection timeouts, but also to make a broader point about the blurred lines between our code and the libraries our code depends on. Because when our app is running, it all effectively becomes our code. Getting comfortable diving into the source for gems you use will serve you well, and will sometimes be the only way to get to the origin of a strange bug, or unexpected behaviour.

One of my favourite ways to do this is using bundle open. It gets you to the literal gem code that your app will be using right there in your preferred editor, ready for exploring. You can even make changes to the gem code for a spot of puts debugging (although do so with care and always remember to undo your changes when you’re done!)

Aside: If you’re a VSCode user, you may want to set the BUNDLER_EDITOR environment variable to "code -n" to stop bundle open replacing your current project window.

The nuance missing from that advice is that it assumes no additional threads are ever spawned during your apps operation!

Now although you might know your code inside and out and be 100% certain that you don’t create additional threads anywhere in your code, chances are Rails is creating additional threads without you realising it…

An aside on threads and ActiveRecord’s connection pool

On a basic level, the way ActiveRecord’s connection pool works is that it assigns each thread that asks for a connection its own separate connection, which the thread then releases once it’s finished. Using a pool like this means that many threads can be querying the database at the same time, so many more requests can be processed in parallel.

If your app’s Puma config sets the maximum number of threads to 5, then you can normally expect there to be at most 5 threads asking for their own database connection, hence the advice to set the pool size to the same as size as the number of threads.

If however one of those threads creates another thread of its own, and that thread needs a database connection to hit the database, it will need its own connection. The spawned thread does not share the connection of the thread that spawned it.

So if we have threads that spawn their own threads, it’s possible to end up in a situation where there are many more threads wanting a connection then there are available connections, and you could end up seeing ActiveRecord::ConnectionTimeoutError exceptions being raised.

Where your Rails app might be spinning up additional threads without you realising it

So, back to your app. Even if you are not explicitly spawning your own threads, Rails itself could be spinning up additional threads without you realising. The main culprit here is ActiveStorage, or more specifically, ActiveStorage configured in proxy mode (commonly used if you’re serving assets via a CDN).

Threads created by ActiveStorage

ActiveStorage’s two proxy controllers return streamed responses, which (you guessed it) are processed in their own threads! So to handle a request for an ActiveStorage file via one of the proxy controllers, two threads will be called into action, both of which need their own connection from the ActiveRecord connection pool.

Normally (hopefully?) your app is processing requests fast enough for you to expect connections to be freed up from one of the other threads and become available for the streaming thread before a timeout occurs. And in this way, most of the time, the limited connections available in the pool can be shared between more threads than there are connections. The problems start if your app receives many successive/concurrent ActiveStorage proxy requests, each of which spin up an additional thread to stream a response, and they take a long time to complete their work and free up their connections; either because they’re doing expensive/slow work server-side (downloading a large file or processing an image representation for the first time), or they’re streaming the response to a slow client. At the time of writing at least, both threads hang on to their connection until the entire response is complete.

Threads created by ActiveRecord’s load_async

Another place Rails will spin up additional threads is with the new load_async method. This was introduced in Rails 7 as a way to parallelise expensive database queries. It’s less likely that these are going to cause connection timeout errors as (again, hopefully?) the expensive queries aren’t so expensive that they hang onto connections for an excessive amount of time. That said, if you’re making heavy use of load_async, you could again end up in a situation where you have much higher demand for connections than there are available connections.

So what should you do about all this?

The upshot of all this is: if you’re making use of either ActiveStorage in proxy mode, or calling load_async in our app, you probably want a connection pool size that is higher than the number of configured Puma threads. The theoretical maximum number of connections you’ll need will be whatever Puma’s configured thread count is x 2, based on the assumption that each thread can potentially spawn one additional thread, but you might be able to get away with a smaller multiple depending on the performance and load characteristics of your app.

And if you’re close to reaching or exceeding the connection limit offered by your database, consider using a separate connection pool like pgbouncer (available as a Heroku buildpack) to increase the number of connections you can configure.

Update (6 August 2023)

There’s now a follow-up post outlining one of the techniques I employed to help figure out why our connection pool was being exhausted in our Rails app.

Update (19 September 2024)

Ben Sheldon has written a great post explaining how to perfectly calculate your Rails database connection pool size. The TL;DR: just make it very big!

The git branch command accepts a --sort option which we can use to list our branches based on the last committer date:

  $ git branch --sort=-committerdate
  * main
    redact-abandoned
    log-downloads
    i18n-lint

We can also use the --format option to include the exact time and see just how fresh each branch is:

  $ git branch --sort=-committerdate --format="%(committerdate)%09%(refname:short)"
  Thu Nov 25 10:29:48 2021 +0000  main
  Fri Nov 19 16:08:49 2021 +0000  redact-abandoned
  Fri Nov 19 16:04:11 2021 +0000  log-downloads
  Thu Nov 18 21:53:37 2021 +0000  i18n-lint
  Sun Jul 18 12:49:18 2021 +0100  without-routing-key-overrides

Or for something more friendly and easy-to-parse we can ask for a relative date:

  $ git branch --sort=-committerdate --format="%(committerdate:relative)%09%(refname:short)"
  5 hours ago     main
  6 days ago      redact-abandoned
  6 days ago      log-downloads
  7 days ago      i18n-lint
  4 months ago    without-routing-key-overrides

That’s better! Let’s add this as a git recent alias to our Git config:

  $ git config --global alias.recent 'branch --sort=-committerdate --format="%(committerdate:relative)%09%(refname:short)"'

Hat tip to Tenderlove for this particularly tasty snippet.

Here are some other Git related articles you might find useful:

• An alias for cleaning up your old/redundant Git branches
• How focused commits make you a better coder
• Better Git diff output for Ruby, Python, Elixer, Go and more
• An alias for jumping from a commit SHA to the PR on GitHub
• An alias for amending your most recent commit

First things first, let’s acknowledge the fact that doing this well does take a certain level of tooling knowledge and practice. That means getting comfortable selectively staging changes (git add --patch) and revising your working history as you go (git rebase --interactive). But once you get the hang of it, it becomes second nature and not doing it will start to feel a bit icky, like skipping test coverage for a new piece of code, or not brushing your teeth before bed.

1. Slowing down

As well as tooling and practice, it also take a level of discipline. That’s because to understand how a piece of work might be delivered as small focused changes, you have to slow down and think about how it could be sliced up and delivered iteratively. This is actually the first benefit in disguise! Figuring out a plan for the work that breaks it down in terms of small, iterative steps is a useful strategy for making a large or difficult problem manageable.

I personally do this with a checklist of tasks that I write down before starting a piece of work. I then update and adjust the list as I progress through the work and my understanding grows and changes. The tasks don’t always map one-to-one to commits, but they often do.

@tomstuart talks more about this and other strategies for breaking down large and difficult problems in his talk Get Off the Tightrope.

2. Taking small steps

Whilst figuring out a series of small, digestible steps to tackle a problem can help get over the anxiety of a large or complex task, it’s not always possible to figure out the whole plan before you start; Sometimes you just need to take that first step. You might not be able to see the whole picture yet, but you see one small change that will get you a little closer. A small, focused commit is the perfect way to “bank” some progress, capturing any insight in the commit message, before continuing to chip away at the problem. Countless times I’ve found myself solving a seemingly complicated and difficult task this way, with only a rough idea of where I’m going, but edging ever closer to the solution one small commit at a time until poof!, the problem is solved.

3. Banking your progress

I love the idea of banking progress in neatly packaged focused commits. It’s a bit like reaching a save point in a video game. You feel a sense of progress. It gives you a perfect reset point if you happen to get lost (git reset --hard HEAD). It also allows you to unload what you’ve learnt (in the commit message) and reset your working memory, freeing up brain cycles to tackle the next part of the challenge.

4. Spotting problems early

Making small, focused changes that you then wrap up in a commit also helps you get ahead of bugs and unintended side-effects. A banked commit is a perfect opportunity to trigger a build and make sure the change hasn’t broken something elsewhere, rather than waiting until all the work is finished and potentially face an avalanche of unforeseen failures that could have stemmed from any number of changes. Spotting and resolving a bug or issue is much easier when the surface area of the change is small and focused vs a 1000-line Hail Mary commit of unrelated changes. Which leads nicely into…

5. Easier debugging

With small focused commits you unlock the full power of Git’s secret bug-smashing tool: git bisect. I’ve not had to use it very often, but when I have it’s felt like a super-power for finding the source of regressions and bugs. bisect will still work without small focused commits, just not quite as effectively.

I’ve successfully taught another person `git bisect` and used it to find an obscure bug in minutes that surely would’ve taken hours/days to find otherwise. If you find yourself saying “this used to work…” in your code, use `git bisect`.

— Mitchell Hashimoto (@mitchellh) January 20, 2021

More about git bisect over in the docs.

6. Less painful merge conflicts

This one only dawned on me recently: resolving merge conflicts becomes more straightforward when commits are small and focused. Perhaps I assumed I’d just got better at doing it (which is still partly true; the more you do it, the better you get at doing it) but on reflection I’d now attribute most of it to those small, focused commits. Because each commit is small and doing one thing, it’s more staightforward figuring out what the delta between the upstream change and the one being applied in the commit should be.

7. Writing more disposable code

This one sounds weird, but changes that are shaped into focused atomic commits tend to be easier to revert cleanly, making code written that way easier to delete! You might need to do this whilst your still working on the feature, perhaps if you decide to change direction, or even much later in the future as the software evolves. As @tef_ebooks puts it: Write code that is easy to delete. Shaping changes into small and focused commits is one thing that will help you do that.

8. Bringing forward changes

Related to that, focused commits are also easier to pull out and use elsewhere. Maybe you’ve fixed a bug or performed a refactoring whilst still working on a larger piece of work. You can cherry-pick and ship those commits early to get the benefit now whilst also shrinking the size of your work-in-progress.

Spent a couple days working methodically through a Rails 6 upgrade, banking small, focused commits as we went. Made for calm, steady progress, with most of the commits shippable against Rails 5, shrinking the size and risk of the actual upgrade. Small, focused commits, always.

— Tekin Süleyman (@tekin) January 21, 2021

9. Better code review

Saving one of my favourites for (almost) last, small focused commits lead to an infinitely better code review process, both for you and the reviewer. The reviewer gets the option to tackle the review in smaller commit-sized chunks, making it easier to review and less likely bugs will slip through.

For an example of what I mean, imagine trying to understand all the changes in this pull request in one go versus reviewing the changes commit by commit, with the commit messages as guiding commentary.

The reviewer’s feedback can also be more targeted, and because the changes are in focused commits, chances are you’ll have a simpler time incorporating any feedback into the existing commits (for those of you that aren’t afraid of or against rewriting the history of an open PR to avoid those “PR feedback” commits that add noise and distractions to the history).

10. A searchable history for future code archaeologists

I won’t go into this one too much as I cover it extensively in A Branch in time (a story about revision histories), but small focused commits with well-written commit messages make the software more maintainable long-term as they provide a commentary to the code’s evolution that can be searched by future developers to better understand the software and the way it has changed over time.

Further exploration

I’ll be writing more on the specific techniques that you can use to shape focused commits in the future. In the meantime if you want to explore these ideas further I’d recommend you check out the following talks:

• A Branch in time (a story about revision histories) a talk I’ve done illustrating some of these ideas
• Simplify writing code with deliberate commits by @joelchippindale
• Getting more from Git by @alicebartlett

Git blame to the rescue?

When this happens you might instinctively reach for git blame to help you figure out what’s going on. After all, git blame gives you the commit that most recently touched the line, and often that’s enough to point you in the right direction. But often it isn’t and you’re none the wiser. That’s because:

• git blame is too coarse: it reports against the whole line. If the most recent change isn’t related to the part of the line you’re interested, you’re out of luck.
• git blame is too shallow: it only reports a single change; the most recent one. The story of the particular piece of code you’re interested in may have evolved over several commits.
• git blame is too narrow: it only considers the file you are running blame against. The code you are interested in may also appear in other files, but to get the relevant commits on those you’ll need to run blame several times.

If you only use git blame you’re limiting yourself to a one-dimensional perspective of the code you’re trying to understand. Wouldn’t it be better to view things in 3D!?

A more powerful way to trace the history of WTF code

Thankfully Git has some pretty powerful search tools built right in. Let’s take a closer look at some of the tools at our disposal.

Find all commits containing a particular piece code

If git blame is entry-level history search, git log -S (also known as “the pickaxe”) is how you take things to the next level. It lets you search for all commits that contain a given string:

$ git log -S "method_name"

This will return all the commits that contain method_name across the entire repository, giving you a bespoke history of a piece of code across both multiple commits and files. You can use the pickaxe to examine the entire history of a particular snippet: all the places it’s been used; how it’s usage has changed; if it’s moved file; when it was first introduced, and more.

See the diffs alongside the commit messages

If you include the -p option (short for --patch) you get the full diff alongside the commit messages, giving you even more context and making it easier to spot the relevant changes:

$ git log -S "method_name" -p

Find the commit that first added some code

Say you want to find the first commit that introduced a particular class, method or snippet of code. You can use the pickaxe combined with the --reverse option to get the commits in reverse-chronological order so the commit where the code first appears is listed at the top:

$ git log -S "method_name" -p --reverse

Find when a piece of code was deleted

Because the pickaxe search finds both additions and deletions, you can even use it to find code that has since been deleted:

$ git log -S "deleted code" -p

Assuming the snippet no longer exists in the codebase the first commit returned will be the one where it was removed.

Limit the scope of the search

Because we’re using git log you can also provide a path to focus the search to a given file or folder:

$ git log -S "some code" -p app/models/user.rb

Search deeper with a regular expression

I mostly stick to the pickaxe because I find working with regular expressions cumbersome, but if you want to perform a more advanced search using a regular expression you can do so with -G instead:

$ git log -G "REGEX HERE"

It’s also worth noting that searching with -S has a subtle limitation that -G does not: -S only shows commits that either added or removed the search string. If the string you are searching for moved within the same file but was otherwise unchanged, -S won’t include that commit, whereas -G will.

Search the commit messages themselves

Sometimes searching for literal code is too narrow, or the information you’re after isn’t actually in the code but in the commit messages themselves. In which case you can use --grep to search commit messages instead:

$ git log --grep "commit message search"

In summary

Next time some code has you puzzled and you want to understand more about it, look beyond git blame and dig deeper into the history using the pickaxe and friends:

• Find the entire history of a snippet of code with git log -S
• Include -p to see the diff as well as the commit messages
• Include --reverse to see the commit that introduced the code listed first
• Scope search to specific folders or files by including a path
• Search with a regular expression using git log -G
• Search commit messages using git log --grep

Epilogue

Of course searching a codebase’s history will only be fruitful if the history itself is helpful. If the history hasn’t been constructed with atomic commits that have useful commit messages, searching it will likely be a frustrating and unhelpful experience.

To learn more about the how and why of putting together more useful histories, check out A Branch in Time (a talk about revision histories).