12 Sep 2014

Performance testing Rails against real data

Have you ever had painfully slow requests in your Rails application? Tools like NewRelic and rack-mini-profiler are great for identifying the cause of poor performance. You can then use performance tests to write repeatable benchmarks and use them to measure and evaluate your optimisations. But out of the box, these tests won’t always give you the entire picture: often requests will only perform poorly when made against a full production database. In such cases, standard Rails performance tests give you a false measure, making it harder to test the benefit of your optimisations.

The good news is that it’s fairly straightforward to run your performance tests against real-world data. Here’s how:

1. Configure a new environment for your “real-world” benchmarks in database.yml

Here I’m pointing the config at my development database because it already has a close approximation of production data in it. Feel free to set up an entirely separate database for your benchmarks.

2. Add a config/environments/benchmark.rb file:

Instead of using the existing test.rb configuration, we create a new one that more closely mirrors production.rb. This allows us to do things like turn caching on. This is particularly useful when your optimisations rely on using caching. Note that this should be tuned to the specifics of you application.

3. Create a test_benchmark_helper.rb file to setup your benchmark tests:

You can add any other dependencies your benchmark tests rely on in here, as well as configure the default profiling options as you see fit.

4. Write a performance test:

This requires benchmark_helper.rb so that it runs against the correct environment. These tests look like standard rails integration tests. Note that you’ll have access to a full database, so you can load any specific instances you need right there in the tests; no need for fixtures or factories.

5. Add a rake task for running your real-world benchmarks:

This rake task will run all the tests in test/performance against your benchmark database. Note that the main difference between this and the default test:benchmark rake task is that it skips the test:prepare step, which would rebuild your database, clearing it of all data. That would be bad.

6. Run your benchmark tests with the newly created rake task:

    $ bundle exec rake test:real_world_benchmark

This generates outputs for each test in the tmp/performance directory. You can now run these tests both before and after your optimisations to see exactly how much of an improvement you’ve made. Job done!

Note: These instructions are for Rails 3 applications. The performance test functionality was extracted out of Rails 4 to a separate gem. It should be possible to get real-world benchmarks working on a Rails 4 application by including this gem in your Gemfile and making any tweaks as necessary.