whiskey_disk

Whiskey Disk -- embarrassingly fast deployments.

A very opinionated deployment tool, designed to be as fast as technologically possible. (For more background, read the WHY.txt file) Should work with any project which is git hosted, not just Ruby / Ruby on Rails projects. Allows for local deploys as well as remote.

You can right-arrow through a talk on the design process of whiskey_disk, given at the 2011 Madison Ruby Conference (as well as the Ruby Hoedown), entitled "Free Whiskey" by going to http://madisonruby.rickbradley.com (slide source available here).

Or... right-arrow through a short whiskey_disk presentation at http://wd2010.rickbradley.com/ (slide source available here.), covering the 0.2.*-era functionality.

You can also right-arrow through a shorter but more up-to-date whiskey_disk "lightning talk" presentation (from the 2010 Ruby Hoedown) at http://wdlightning.rickbradley.com/ (slide source available here.), covering the 0.4.*-era functionality.

tl;dr

First:

% gem install whiskey_disk

Then make a deploy.yml file (in config/ if you're doing a Rails project):

staging:
  domain: "[email protected]"
  deploy_to: "/path/to/where/i/deploy/staging.mydomain.com"
  repository: "https://github.com/username/project.git"
  branch: "staging"
  rake_env:
    RAILS_ENV: 'production'

then:

% wd setup --to=staging

then:

% wd deploy --to=staging

Selling points

• If you share the same opinions as we do there's almost no code involved, almost no dependencies, and it uses stock *nix tools (ssh, bash, rsync) to get everything done.

• Written completely spec-first for 100% coverage. We even did that for the rake tasks, the init.rb and the plugin install.rb (if you swing that way).

• 1 ssh connection per run -- so everything needed to do a full setup is done in one shot. Everything needed to do a full deployment is done in one shot. (Having 8 minute deploys failing because I was on CDMA wireless on a train in India where the connection won't stay up for more than 2-3 minutes is not where I want to be any more.)

• Deployment configuration is specified as YAML data, not as code. Operations to perform after setup or deployment are specified as rake tasks.

• You can do local deployments, by this I mean you can use whiskey_disk to deploy fully running instances of your application to the same machine you're developing on. This turns out to be surprisingly handy (well, I was surprised). NOTE: be sure to set your deploy_to to a place other than the current local checkout.

• You can do multi-project deployments, specifying deployment data in a single deploy.yml config file, or keep an entire directory of project deployment config files.

• You can separate per-deployment application configuration information (e.g., passwords, database configs, hoptoad/AWS/email config data, etc.) in separate repositories from the application, and whiskey_disk will merge the correct data onto the deployed application at deployment time. You can even share sets of configuration files among deployment targets that behave alike.

• You can have per-developer configurations for targets (especially useful for "local" or "development" targets). Use .gitignore, or specify a config_branch and everyone can have their own local setup that just works.

• There's no before_after_before_after hooks. You can use a well-defined rake hook and/or a bash script to run additional tasks.

• You can enable "staleness checks" so that deployments only happen if either the main repo, or the config repo (if you're using one) has changes that are newer than what is currently deployed.

• Put whiskey_disk in a cron, with staleness checks enabled, and you can do hands-free automated deployments whenever code is pushed to your deployment branch of choice!

• You can deploy to multiple remote targets at once. Currently this is limited to one-after-the-other synchronous deployments, but we're thinking about doing them in parallel once we're happy with the stability of this feature.

• Assign hosts to roles (e.g., "web", "db", "app") and vary the shell or rake post-setup/post-deploy actions you run based on those roles.

• Limit the actions you run after deployment based on whether files of interest actually changed.

Assumptions

• your project is managed via git
• you are deploying over ssh, or deploying locally and have a bash-compatible shell
• you are comfortable defining (optional) post-setup and post-deployment actions with rake
• you have an optional second git repository for per-application/per-target configuration files
• you have an optional Rakefile in the top directory of your project's checkout

Dependencies

On the server from which the whiskey_disk process will be kicked off:

• ruby
• rake
• whiskey_disk
• ssh (if doing a remote deployment).

On the deployment target server (which may be the same as the first server):

• a bash-compatible shell
• rsync (only if using a configuration repository)
• ruby, rake, whiskey_disk (only if running post_setup or post_deploy hooks)

If you're running on OS X or Linux you probably have all of these installed already. Note that the deployment target system doesn't even have to have ruby installed unless post_* rake hooks are being run.

Installation

As a gem:

% gem install whiskey_disk

As a rails plugin:

% script/plugin install git://github.com/flogic/whiskey_disk.git

Configuration

• look in the examples/ directory for sample configuration files
• main configuration is in <app_root>/config/deploy.yml
• config files are YAML, with a section for each target.

Known config file settings (if you're familiar with capistrano and vlad these should seem eerily familiar):

domain:              host or list of hosts on which to deploy (these are ssh connect strings)
                     can also optionally include role information for each host
deploy_to:           path to which to deploy main application
repository:          git repo path for main application
branch:              git branch to deploy from main application git repo (default: master)
deploy_config_to:    where to deploy the configuration repository
config_repository:   git repository for configuration files
config_branch:       git branch to deploy from configuration git repo (default: master)
config_target:       configuration repository target path to use
project:             project name (used to compute path in configuration checkout)
post_deploy_script:  path to a shell script to run after deployment
post_setup_script:   path to a shell script to run after setup
rake_env:            hash of environment variables to set when running post_setup and post_deploy rake tasks

A simple config/deploy.yml might look like:

qa:
  domain: "[email protected]"
  deploy_to: "/var/www/www.ogtastic.com"
  repository: "[email protected]:www.ogtastic.com.git"
  branch: "stable"
  rake_env:
    RAILS_ENV: 'production'

• defining a deploy:<target>:post_setup rake task (e.g., in lib/tasks/ or in your project's Rakefile) will cause that task to be run at the end of deploy:setup

• defining a deploy:<target>:post_deploy rake task (e.g., in lib/tasks/ or in your project's Rakefile) will cause that task to be run at the end of deploy:now

It's easy to specify a local deployment. The simplest way is to just not specify a "domain":

local:
  deploy_to: "/var/www/www.ogtastic.com"
  repository: "[email protected]:www.ogtastic.com.git"
  branch: "stable"
  rake_env:
    RAILS_ENV: 'production'

Or, just specify the string 'local' as the domain:

local:
  domain: "local"
  deploy_to: "/var/www/www.ogtastic.com"
  repository: "[email protected]:www.ogtastic.com.git"
  branch: "stable"
  rake_env:
    RAILS_ENV: 'production'

For deploying to multiple hosts, the config/deploy.yml might look like:

qa:
  domain:
  - "[email protected]"
  - "[email protected]""
  deploy_to: "/var/www/www.ogtastic.com"
  repository: "[email protected]:www.ogtastic.com.git"
  branch: "stable"
  rake_env:
    RAILS_ENV: 'production'

You can even include a local deployment along with remote deployments, simply use the 'local' name:

qa:
  domain:
  - "local"
  - "[email protected]""
  deploy_to: "/var/www/www.ogtastic.com"
  repository: "[email protected]:www.ogtastic.com.git"
  branch: "stable"
  rake_env:
    RAILS_ENV: 'production'

If you need special flags passed to ssh for a given domain, specify an array of ssh_options flags:

qa:
  domain:
  - name: "[email protected]"
    ssh_options:
    - "-t"
    - "-vv"
    - "-p 443"
  deploy_to: "/var/www/www.ogtastic.com"
  repository: "[email protected]:www.ogtastic.com.git"

Specifying domains, with or without roles

There are a number of ways to specify domains (the ssh connection strings denoting the hosts where your code will be deployed). Here are just a few examples:

Just a single domain:

staging:
  domain: "[email protected]"

Just a single domain, but specified as a one-item list:

qa:
  domain:
  - "[email protected]"

A list of multiple domains:

production:
  domain:
  - "[email protected]"
  - "[email protected]"
  - "[email protected]"

Using the "name" label for the domain names (if using roles, as described below, the "name" label is required, otherwise it's optional and superfluous):

ci:
  domain:
  - name: "[email protected]"

It's also possible to assign various "roles" to the domains to which you deploy. Some common usages would be "www", which might need a post_deploy task which notifies some web server software (apache, nginx, passenger, unicorn, etc.) that it should refresh the contents being served; or perhaps "db", which might need some set of post-deployment database migrations run (and which shouldn't be run from multiple servers).

The role names are simply strings and you can create whichever roles you wish. See the section below entitled "Taking actions based on roles" to see how to use roles to control actions when setting up or deploying to a target.

Roles are described in the domain: section of the configuration file. There are, of course, a few different valid ways to specify roles. Note that the domain name must now be labeled when roles are being specified for the domain.

A single role for a domain can be specified inline:

production:
  domain:
  - name: "[email protected]"
    roles: "web"

While multiple roles for a domain must be specified as a list:

production:
  domain:
  - name: "[email protected]"
    roles:
    - "web"
    - "app"
    - "db"

But domains with roles can be specified alongside simple domains as well:

production:
  domain:
  - name: "[email protected]"
  - "[email protected]"
  - name: "[email protected]"
    roles:
    - "web"
    - "app"
    - "db"

And, if you need to assign roles for a local deployment, you can do that as well:

local:
  domain:
  - name: "local"
    roles:
    - "web"
    - "app"
    - "db"

All that s