The following are some notes I put together on getting started with Vanity in a Rails 3 application. Vanity is an A/B Testing framework. I covered A/Bingo previously but we moved to Vanity since we were already using Redis, and had some existing organizational knowledge about Vanity internals with other team members. The documentation is pretty good but I thought this post could help fill in some details for newcomers. Read the labnotes blog documentation and 2-minute demo instructions first. We have used Vanity with Redis, which is the single data store it originally supported, but has since been expanded to support ActiveRecord (RDBMS) and MongoDB. Keeping experiment and participant tracking in a separate data store like Redis was superior to the database for speed, and has allowed us to develop some patches to disable Vanity if Redis goes down without taking the site being tested down as well.

Getting started

(1) Create the required directories to store your experiments, a generator would be useful here, but one does not currently exist. This example creates an experiment that presents multiple prices for a commerce website. In vanity each experiment requires a Ruby source file in the experiments directory. Run the following commands from Rails.root.

mkdir -p experiments/metrics          
touch experiments/price_options.rb

Add the following to your price_options.rb Ruby file.

ab_test "Price options" do
  description "Which price is the best price?"
  alternatives 19, 29, 39
end

Now the experiment can be referenced by its name, by using the ab_test method and passing the name of the experiment as a symbol (it is downcased and underscorized for you). You can call ab_test from views, controllers, helpers, and even model files, they all use the same method signature.

(2) Create a metric. When a website visitor of yours is participating in an experiment, they will see one of the alternatives defined by that experiment. An experiment should have some goal associated with it, in this example we want the site visitor to purchase something. Vanity will handle showing the alternatives randomly, tracking which participant saw which alternative, and ensuring participants that are identified continue to see the same alternative on multiple site visits. We need to instrument the application code though at specific places to indicate when a goal has been met though. Create a metric file from Rails.root.

touch experiments/metrics/signup.rb

Add the following to your metric file. You could have a "database-backed" model by using a model class from the Rails application where Vanity is used, but this example will be a simple metric.

metric "Purchase" do
  description "Tracks the performance of various prices in terms of how many purchases their participants produce"
end

Site visitors need to be identified in some way for Vanity to work properly. This is a bit easier with logged-in users, because they probably have a database record that identifies their account uniquely, which could be used with Vanity to identify them. Without a database record, you can create a session to identify each user, or use their IP address, or some other way that makes sense for your application. In the project I'm working on, we have visitors that do not have accounts yet. I use the use_vanity method from Vanity without any arguments, which creates a vanity_id for that visitor. Their vanity_id is available across requests as a cookie.

When some participants have viewed alternatives and you're ready to look at report output, you can generate results. As of this writing, this report may not work. You will need to have your data store running (Redis in our case) since it will use the datastore to pull participant data. Typically we just use the dashboard to view participation counts, and how alternatives are doing relative to one another.

vanity report --output vanity.html

Vanity reports are available with some controller and view code provided by the gem. The documentation has you create a vanity controller, but you can also move these controller actions and view code around in your application, if there is an existing dashboard for example. If you follow the instructions, the gem has actions that make assumptions about the controller being there and being named a certain way, and a route being available to route to the vanity controller. The following syntax is for Rails 3.

match '/vanity(/:action(/:id(.:format)))', :controller=>:vanity 

Create the vanity controller to report results:

touch app/controllers/vanity_controller.rb

Inside your controller (to get the actions) make sure to include Vanity::Rails::Dashboard. We ended up using the vanity template Erb files directly by copying them into the main application code so we could edit them and have more control over the presentation.

Viewing the vanity controller dashboard won't work if the experiment object has a nil value for created_at. By default in development Vanity won't record participants for the experiment, so without recording turned on, the experiment doesn't set created_at at load time, which breaks the dashboard when you try and load it. To enable tracking in the development environment, I set:

Vanity.playground.collecting = true

Now when the server starts up, it connects to Redis to set the experiment created_at first, this is basically the record that backs the experiment model file. You should see something in the Redis log like the following:

+1297968800.845316 "setnx" "vanity:experiments:price_options:created_at" "1297968800"

I'm planning to write more about Vanity as we have some ideas for additional tooling, and have some commits that help make it more robust that we can hopefully push up to the project. I have had a couple minor patches merged in to the main Vanity source, so I know the maintainer is open to adding new features and bug fixes. Are you doing A/B testing? Are you using Vanity or something else?