shipit-engine

Shipit - Documentation

Shipit is a deployment tool that makes shipping code better for everyone. It's especially great for large teams of developers and designers who work together to build and deploy GitHub repos. You can use it to:

• add new applications to your deployment environment without having to change core configuration files repeatedly — shipit.yml is basically plug and play
• control the pace of development by pushing, locking, and rolling back deploys from within Shipit
• enforce checklists and provide monitoring right at the point of deployment.

Shipit is compatible with just about anything that you can deploy using a script. It natively detects stacks using bundler and Capistrano, and it has tools that make it easy to deploy to Heroku or RubyGems. At Shopify, we've used Shipit to synchronize and deploy hundreds of projects across dozens of teams, using Python, Rails, RubyGems, Java, and Go.

This guide aims to help you set up, use, and understand Shipit.

Shipit requires a database (MySQL, PostgreSQL or SQLite3), redis, and Ruby 2.6 or superior.

Table of contents

I. INSTALLATION & SETUP

• Installation
• Updating an existing installation

II. USING SHIPIT

• Adding stacks
• Working on stacks
• Configuring stacks

III. REFERENCE

• Format and content of shipit.yml
• Script parameters
• Configuring providers
• Free samples

IV. INTEGRATING

• Registering webhooks

V. CONTRIBUTING

• Instructions
• Local development

I. INSTALLATION & SETUP

Installation

To create a new Shipit installation you can follow the setup guide.

Updating an existing installation

1. If you locked the gem to a specific version in your Gemfile, update it there.
2. Update the shipit-engine gem with bundle update shipit-engine.
3. Install new migrations with rake shipit:install:migrations db:migrate.

Specific updates requiring more steps

If you are upgrading from 0.21 or older, you will have to update the configuration. Please follow the dedicated upgrade guide

II. USING SHIPIT

The main workflows in Shipit are adding stacks, working on stacks, and configuring stacks.

A stack is composed of a GitHub repository, a branch, and a deployment environment. Shipit tracks the commits made to the branch, and then displays them in the stack overview. From there, you can deploy the branch to whatever environment you've chosen (some typical environments include production, staging, performance, etc.).

Add a new stack

1. From the main page in Shipit, click Add a stack.
2. On the Create a stack page, enter the required information:
   • Repo
   • Branch
   • Environment
   • Deploy URL
3. When you're finished, click Create stack.

Work on an existing stack

1. If you want to browse the list of available stacks, click Show all stacks on the main page in Shipit. If you know the name of the stack you're looking for, enter it in the search field.
2. Click the name of the stack you want to open.
3. From a stack's overview page, you can:
   • review previous deploys
   • deploy any undeployed commits by clicking Deploy
   • rollback to an earlier build by clicking Rollback to this deploy
   • adjust the stack's settings by clicking the gear icon in the page header
   • perform any custom tasks that are defined in the shipit.yml file
4. When you're ready to deploy an undeployed commit, click the relevant Deploy button on the stack's overview page.
5. From the Deploy page, complete the checklist, then click Create deploy.

Edit stack settings

To edit a stack's settings, open the stack in Shipit, then click the gear icon in the page header.

From a stack's Settings page, you can:

• change the deploy URL
• enable and disable continuous deployment
• lock and unlock deploys through Shipit
• resynchronize the stack with GitHub
• delete the stack from Shipit

III. REFERENCE

Configuring shipit.yml

The settings in the shipit.yml file relate to the different things you can do with Shipit:

• Installing Dependencies (dependencies)
• Deployment (deploy, rollback, fetch)
• Environment (machine.environment, machine.directory, machine.cleanup)
• CI (ci.require, ci.hide, ci.allow_failures)
• Merge Queue (merge.revalidate_after, merge.require, merge.ignore, merge.max_divergence)
• Custom Tasks (tasks)
• Custom links (links)
• Review Process (review.checklist, review.monitoring, review.checks)
• Inherit From(inherit_from)

All the settings in shipit.yml are optional. Most applications can be deployed from Shipit without any configuration.

Also, if your repository is deployed different ways depending on the environment, you can have an alternative shipit.yml by including the environment name.

For example for a stack like: my-org/my-repo/staging, shipit.staging.yml will have priority over shipit.yml.

In order to reduce duplication across different environment specific files, you can specify an inherit_from key in your relevant shipit.yml file. This key expects a string of the file name to inherit from. If this key is specified, a deep-merge will be performed on the file therein, overwriting any duplicated values from the parent. See Inherit From(inherit_From) for example.

Lastly, if you override the app_name configuration in your Shipit deployment, yourapp.yml and yourapp.staging.yml will work.

Respecting bare shipit.yml files

Shipit will, by default, respect the "bare" shipit.yml file as a fallback option if no more specifically-named file exists (such as shipit.staging.yml).

You can configure this behavior via the attribute Shipit.respect_bare_shipit_file.

• The value false will disable this behavior and instead cause Shipit to emit an error upon deploy if Shipit cannot find a more specifically-named file.
• Setting this attribute to any other value (including nil), or not setting this attribute, will cause Shipit to use the default behavior of respecting bare shipit.yml files.

You can determine if Shipit is configured to respect bare files using Shipit.respect_bare_shipit_file?.

Installing dependencies

The dependencies step allows you to install all the packages your deploy script needs.

Bundler

If your application uses Bundler, Shipit will detect it automatically and take care of the bundle install and prefix your commands with bundle exec.

By default, the following gem groups will be ignored:

• default
• production
• development
• test
• staging
• benchmark
• debug

The gems you need in order to deploy should be in a different group, such as deploy.

For example:

dependencies:
  bundler:
    without:
      - development
      - test
      - debug

Other dependencies

If your deploy script uses another tool to install dependencies, you can install them manually via dependencies.override:

dependencies:
  override:
    - npm install

dependencies.pre If you wish to execute commands before Shipit installs the dependencies, you can specify them here.

For example:

dependencies:
  pre:
    - mkdir tmp/
    - cp -R /var/cache/ tmp/cache

dependencies.post If you wish to execute commands after Shipit installed the dependencies, you can specify them here:

For example:

dependencies:
  post:
    - cp -R tmp/cache /var/cache/

Deployment

The deploy and rollback sections are the core of Shipit:

deploy.override contains an array of the shell commands required to deploy the application. Shipit will try to infer it from the repository structure, but you can change the default inference.

For example:

deploy:
  override:
    - ./script/deploy

deploy.pre If you wish to execute commands before Shipit executes your deploy script, you can specify them here.

For example:

deploy:
  pre:
    - ./script/notify_deploy_start

deploy.post If you wish to execute commands after Shipit executed your deploy script, you can specify them here.

For example:

deploy:
  post:
    - ./script/notify_deploy_end

You can also accept custom environment variables defined by the user that triggers the deploy:

deploy.variables contains an array of variable definitions.

For example:

deploy:
  variables:
    -
      name: RUN_MIGRATIONS
      title: Run database migrations on deploy
      default: 1

deploy.variables.select will turn the input into a <select> of values.

For example:

deploy:
  variables:
    -
      name: REGION
      title: Run a deploy in a given region
      select:
        - east
        - west
        - north

deploy.max_commits defines the maximum number of commits that should be shipped per deploy. Defaults to 8 if no value is provided.

To disable this limit, you can use use an explicit null value: max_commits: null. Continuous Delivery will then deploy any number of commits.

Human users will be warned that they are not respecting the recommendation, but allowed to continue. However continuous delivery will respect this limit. If there is no deployable commits in this range, a human intervention will be required.

For example:

deploy:
  max_commits: 5

deploy.interval defines the interval between the end of a deploy and the next deploy, when continuous delivery is enabled. You can use s, m, h, d as units for seconds, minutes, hours, and days. Defaults to 0, which means a new deploy will start as soon as the current one finishes.

For example, this will wait 5 minutes after the end of a deploy before starting a new one:

deploy:
  interval: 5m

deploy.retries enables retries for a stack, and defines the maximum amount of times that Shipit will retry a deploy that finished with a failed, error or timedout status.

For example, this will retry a deploy twice if it fails.

deploy:
  retries: 2

rollback.override contains an array of the shell commands required to rollback the application to a previous state. Shipit will try to infer it from the repository structure, but you can change the default inference. This key defaults to disabled unless Capistrano is detected.

For example:

rollback:
  override:
    - ./script/rollback

rollback.pre If you wish to execute commands before Shipit executes your rollback script, you can specify them here:

For e