At Profitably, we’ve been using Resque for a few months now, and we’re loving it. Resque’s web interface is a great way to get a high-level understanding of your background workers, and its pluggable design has made it easy for others to contribute a number of really useful plugins. For Profitably, getting control of our background jobs is critical. Our software doesn’t have a ton of public-facing controllers & actions yet, but under the water line it’s already a large system built around mapping and aggregating financial data. (If you’re a Rubyist, and this sounds interesting to you, you should check out our jobs page.)
From a code perspective, it’s not that hard to move to Resque—you write Job classes and you just run them. But I found that I had a few speedbumps in setting it up for production, not least because configuration is not my strong suit. So I decided to describe how we configured Resque to run in production. There are a lot of duct-tapey parts to our setup, and it’s quite likely you’ll make some different choices, but what I describe below has worked well for us so far.
Build a resque-web directory in your Rails app
A lot of the work from setting up Resque is from setting up resque-web, the Sinatra app that helps you monitor Resque.
If you suck at sysadmin like I do, you will be tempted to just run Resque without resque-web. Don’t do that. The easy visibility you get from resque-web can be a lifesaver.
I committed a self-contained resque-web directory in the main Rails app, and then pointed Passenger to it. The resque-web directory looks like this:
It’s weird to put a 2nd web app in the config directory of your Rails app, but it seemed less weird than any other place. If you wanted to be super-tidy about it, you could maybe make the case for a separate project with its own deploy file, but it’s only a few files. Also, keeping it with the Rails app makes it easier to keep it in sync with the Redis config, which I ended up sharing across the Rails app and the resque-web app. More on that below.
For config.ru, copy over the basic config.ru that comes packaged with Resque. I made a few modifications: These are described below. The other two files, config/resque-web/public/placeholder.txt and config/resque-web/tmp/placeholder.txt, are just empty files that will force Git to keep their directories, which are both used by Passenger.
Modify config.ru
You’ll need to modify config.ru from what’s packaged with Resque. In particular, there’s a decent amount of work involved in giving resque-web access to configuration information in the Rails app, since it’s not a Rails app itself. All of the modifications described below should happen before the last two lines, which you should leave alone:
use Rack::ShowExceptions
run Resque::Server.new
Require Resque in config.ru
config.ru has this line for adding Resque to your load path:
You should replace it with lines that find the lib directory wherever you’ve got the Resque gem. In Profitably’s case, we’re not on Bundler yet (I know, embarrassing), and we’ve got our gems in vendor/gems. So our load path config looks like:
The first and second are easy because they’ll both get the Rails environment. But the third is a little trickier because it won’t be loading your Rails app. So we need to put the Redis config somewhere that’s not tightly bound to Rails, and include it from both your Rails environment and from the resque-web application.
So the steps I took are:
Put the Redis config in config/initializers/redis.rb, where Rails will load it automatically
Require that redis.rb file in config.ru, so resque-web will get it too.
# Figure out the Rails environment, defaulting to production
ENV['RAILS_ENV'] ||= 'production'
# Require the redis config
require ::File.expand_path(::File.dirname(__FILE__) + "/../initializers/redis")
It’s a bit messy, because you’ve got a non-Rails-app loading ENV['RAILS_ENV'], but c’est la vie.
Add an HTTP Basic authentication password to config.ru (optional)
These lines add a quick-and-dirty password in case somebody finds out what subdomain we’re using for resque-web:
AUTH_PASSWORD = "top-secret-password"
if AUTH_PASSWORD
Resque::Server.use Rack::Auth::Basic do |username, password|
password == AUTH_PASSWORD
end
end
Set up a Passenger vhost to start resque-web
Wherever Apache looks for virtual host config files, put a config file for resque-web—for us that’s /etc/apache2/sites-enabled/resque-web, which looks like this:
Passenger knows to look for config.ru in /u/apps/myapp/current/config/resque-web, which it will use to start resque-web.
Running resque-web in environments other than production (optional)
You may want to run resque-web in other deployed environments, such as staging. If you want to pass a different environment to resque-web, you can do so by hard-coding it in your vhost file:
This is visible in config.ru as ENV['RACK_ENV'], so you can copy that into the Rails environment setting by adding to the lines that I described above:
... and then that gets used by config/initializers/redis.rb.
Setup up your Cap file to restart resque-web when you deploy
Not that resque-web’s going to change that often, but you might as well:
namespace :deploy do
task :restart_resque_web, :roles => :app, :except => { :no_release => true } do
run "#{try_sudo} touch #{File.join(
current_path,'config','resque-web','tmp','restart.txt'
)}"
end
after 'deploy:restart', 'deploy:restart_resque_web'
end
Loading the Rails environment into your Resque workers
Now that you’re done configuring resque-web, it’s time to set up Resque workers. When you’re starting out you can test them out by running them on the server directly. However, if you just run
RAILS_ENV=production QUEUE=* rake resque:work
Rake and Resque will load, try to run your jobs without your Rails environment, and then spit out a ton of errors. Load the environment into the rake task like so:
The workers will take jobs off the queues in order, starting with queue1. In practice, we put most of our jobs into the middle queue, with a few exceptions. It’s also nice to have the option of re-prioritizing everything in production by hand if we have to, by moving it up or down one queue.
Setup a Cap task to restart resque workers on deploy
This step isn’t much different from setting up worker processes for anything else—if you did this for Delayed::Job or another background processing library, this’ll be pretty familiar to you.
A lot of people will do this with something like God, but we did it with PID files. Remembering to load the Rails environment, and using multiple queues, our Cap task looks like this:
namespace :deploy do
task :restart_resque_workers, :roles => :resque_worker, :except => { :no_release => true } do
pid_file = "#{current_path}/tmp/pids/resque_worker.pid"
run "test -f #{pid_file} && cd #{current_path} && kill -s QUIT `cat #{pid_file}` || rm -f #{pid_file}"
queues = (1..5).to_a.map { |i| "queue#{i}" }.join(',')
run "cd #{current_path}; RAILS_ENV=#{rails_env} QUEUE=#{queues} VERBOSE=1 nohup rake environment resque:work& > #{current_path}/log/resque_worker.log && echo $! > #{pid_file}"
end
after 'deploy:restart', 'deploy:restart_resque_workers'
end
Go to lunch, or have a drink (depending on the time of day)
So that’s about it. It was a decent amount of work getting it set up, but now we have a simple queueing system that doesn’t hit our SQL database, a great top-level interface into the whole system, and we can make use of all those great Resque plugins.
I hope this write-up is helpful, and as always, suggestions are very welcome.
Previous Topic
