Performance testing Rails against real data

September 12, 2014

Have you ever had painfully slow requests in your Rails application? Tools like NewRelic and rack-mini-profiler are great for identifying the cause of poor performance. You can then use performance tests to write repeatable benchmarks and use them to measure and evaluate your optimisations. But out of the box, these tests won’t always give you the entire picture: often requests will only perform poorly when made against a full production database. In such cases, standard Rails performance tests give you a false measure, making it harder to test the benefit of your optimisations.

The good news is that it’s fairly straightforward to run your performance tests against real-world data. Here’s how:

1. Configure a new environment for your “real-world” benchmarks in database.yml


benchmark:
  encoding: utf8
  adapter:  mysql2
  database: app_development
  username: root
  password:

Here I’m pointing the config at my development database because it already has a close approximation of production data in it. Feel free to set up an entirely separate database for your benchmarks.

2. Add a config/environments/benchmark.rb file:

Instead of using the existing test.rb configuration, we create a new one that more closely mirrors production.rb. This allows us to do things like turn caching on. This is particularly useful when your optimisations rely on using caching. Note that this should be tuned to the specifics of you application:


# config/environments/benchmark.rb
Rails.application.configure do
  # Settings specified here will take precedence over those in config/application.rb.

  # Code is not reloaded between requests.
  config.cache_classes = true

  # Eager load code on boot. This eager loads most of Rails and
  # your application in memory, allowing both threaded web servers
  # and those relying on copy on write to perform better.
  # Rake tasks automatically ignore this option for performance.
  config.eager_load = true

  # Full error reports are disabled and caching is turned on.
  config.consider_all_requests_local       = false
  config.action_controller.perform_caching = true

  # Disable Rails's static asset server (Apache or nginx will already do this).
  config.serve_static_assets = false

  # Compress JavaScripts and CSS.
  config.assets.js_compressor = :uglifier
  # config.assets.css_compressor = :sass

  # Do not fallback to assets pipeline if a precompiled asset is missed.
  config.assets.compile = false

  # Generate digests for assets URLs.
  config.assets.digest = true

  # Set to :debug to see everything in the log.
  config.log_level = :info
  # Use a different cache store in production.
  config.cache_store = :mem_cache_store

  # Enable locale fallbacks for I18n (makes lookups for any locale fall back to
  # the I18n.default_locale when a translation cannot be found).
  config.i18n.fallbacks = true

  # Send deprecation notices to registered listeners.
  config.active_support.deprecation = :notify

  # Disable automatic flushing of the log to improve performance.
  # config.autoflush_log = false

  # Use default logging formatter so that PID and timestamp are not suppressed.
  config.log_formatter = ::Logger::Formatter.new

  # Do not dump schema after migrations.
  config.active_record.dump_schema_after_migration = false
end

3. Create a test_benchmark_helper.rb file to setup your benchmark tests:


# test/benchmark_helper.rb
#
# Use this helper to write performance tests that benchmark requests in a more
# "production"-like context. This makes it easier to accurately measure the
# real-world gains of any performance optimisations.
#
# Performance tests run with this helper will run the tests in the "benchmark"
# environment, which differs from the "test" environment in the following ways:
#
#   * It behaves more like "production" (i.e. caching is enabled)
#   * It runs against the "development" database (i.e. a full data dump)
#   * The database is *not* rebuilt before the tests are run
#
# By enabling caching and running against the "development" database, we get a
# more accurate measure of how requests will actually perform in production
# with a full database is present.
#
# A rake task exists that will run these performance tests:
#
#  rake test:real_world_benchmarks
#
ENV["RAILS_ENV"] = "benchmark"
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
require 'rails/performance_test_help'

class ActionDispatch::PerformanceTest
  self.profile_options = { :runs => 5, :metrics => [:wall_time],
                           :output => 'tmp/performance', :formats => [:flat] }
end

You can add any other dependencies your benchmark tests rely on in here, as well as configure the default profiling options as you see fit.

4. Write a performance test:


# test/performance/benchmark_against_real_world_data_test.rb
require 'benchmark_helper'

class BenchmarkAgainstRealWorldDataTest < ActionDispatch::PerformanceTest

  setup do
    # login, etc
  end

  test ‘new publication page’ do
    get '/publications/new'
  end

  test ‘editing publication page` do
    publication = Publication.last
    get "/publications/#{publication.id}/edit"
  end
end

This requires benchmark_helper.rb so that it runs against the correct environment. These tests look like standard rails integration tests. Note that you’ll have access to a full database, so you can load any specific instances you need right there in the tests; no need for fixtures or factories.

5. Add a rake task for running your real-world benchmarks:


# lib/tasks/test_benchmark.rb
#
# Use this rake task to run performance tests against the "benchmark"
# environment. See test/benchmark_helper.rb for more details.
namespace :test do
  Rake::TestTask.new(:real_world_benchmark => ['test:benchmark_mode']) do |t|
    t.libs << 'test'
    t.pattern = 'test/performance/**/*_test.rb'
  end
end

This rake task will run all the tests in test/performance against your benchmark database. Note that the main difference between this and the default test:benchmark rake task is that it skips the test:prepare step, which would rebuild your database, clearing it of all data. That would be bad.

6. Run your benchmark tests with the newly created rake task:

  $ bundle exec rake test:real_world_benchmark

This generates outputs for each test in the tmp/performance directory. You can now run these tests both before and after your optimisations to see exactly how much of an improvement you’ve made. Job done!

Note: These instructions are for Rails 3 applications. The performance test functionality was extracted out of Rails 4 to a separate gem. It should be possible to get real-world benchmarks working on a Rails 4 application by including this gem in your Gemfile and making any tweaks as necessary.

tekin.co.uk